- Problem
- Aymara has only a few hundred thousand digital words compared to billions for Spanish, leaving it outside most language tools built today.
- Solution
- A measured question-answering and retrieval system over Aymara Wikipedia, with a public benchmark, documented failed experiments and a validation pipeline that requires human review.
- My role
- Research lead and engineer
- Result
- From 1/10 to 7/10 retrieval accuracy in six documented iterations, with a public site and reproducible pipeline.
Overview
YAIS Aymara Research documents the attempt to build a question-answering and retrieval system for Aymara, measuring each step and publishing the failures alongside the gains.
Problem
Low-resource languages are left out of most NLP systems because the data is scarce. For Aymara, the risk is not just poor performance but hallucinated answers presented as fact.
My role
Research lead and engineer. I designed the benchmark, ran the experiments, built the public site and wrote the reproducible pipeline.
Solution
A retrieval pipeline over 5,469 Aymara Wikipedia articles with a custom benchmark of 30 questions — 10 answerable and 20 designed to test abstention — plus a public research site exported from an Obsidian vault.
Technical approach
Python pipelines with sentence-transformers, LaBSE, NLLB-200, FAISS and BM25; fine-tuning on Kaggle's free GPU; 63 automated tests; and a TanStack Start site with Recharts for the published notes.
Challenges
The real bottleneck is not compute but validators: native speakers who can review machine-generated data before it enters the training set.
Results
Retrieval accuracy moved from 1/10 to 7/10 on the custom benchmark. The key lesson: training with the wrong objective improved the model's own metric by 37% while destroying real task performance.
What I learned
That alignment between training objective and real task matters more than hyperparameters, and that negative results should be published if the field is to move forward responsibly.
Technologies
- Python
- sentence-transformers
- LaBSE
- NLLB-200
- FAISS
- BM25
- TanStack Start
- Recharts