The analyzer disagreed with the generator, and it was right

Field notes from building speech AI for a low-resource language, where the most useful component is the one that refuses to agree with the others.

I build language technology for an Australian Aboriginal language I do not speak. That means I cannot proofread the output. My defence has been to build components that check each other: a rule-based generator that only synthesises from a curated verb list, and a morphological analyzer, validated against a published grammar, that parses any sentence back into glossed morphemes.

Earlier this week they disagreed. The lesson generator produced a sentence meant to say that elders teach children. I put the sentence into the analyzer, and it read the verb back as say. Not teach. The generated line actually said something like “elders said children.”

My first instinct was that the analyzer had a bug. It did not. The generator’s verb table mapped teach to the root for say, so the generator had written say, fluently and confidently, and the analyzer had simply read what was there. The two components were not in conflict. One of them was wrong and the other one caught it.

Auditing the whole table

If one mapping was wrong, others would be. I checked every verb the lesson generator was allowed to use, 58 of them, against every source I hold that deserves the word gold: the two published grammars of the language, with their verb root lists and conjugation classes, and a few thousand sentence-level pairs from the primary grammar.

The matching needed care. Grammars write morpheme boundaries into roots, so yirr+ma and yirrma are the same root, and two orthography conventions differ on three graphemes. My first comparison pass flagged the system’s most-used roots as unattested, and every one of those flags was my own string matching failing on exactly these conventions. Normalise first, then judge.

After normalising, the results sorted into four piles.

The core held. The motion root the system uses for go, come and walk appears in 22 canonical sentence pairs with exactly those meanings. The root for say appears directly. See, sit, sing and swim all check out. Whatever else was wrong, the foundation was real.

Six verbs sat on a root that means something else. Teach on say. Read and show on see. Dance on sing. These produce fluent output that is simply about a different action.

Six more were contradicted by the grammars. Both grammars gloss eat to a root that is not the one my table used, and the sentence pairs back them: zero examples support my root as eat. Same for run, think, float and crawl. My table was not simplifying these. It was wrong about them.

Two were worse than wrong. My root for fly matches a gold root that means cover. My root for bite matched nothing at all; an earlier check had called it attested because the string happened to occur inside an unrelated longer root, which is what you get for grepping a language.

Fifty-eight became eleven

The verb list the generator may draw from is now: come, go, walk, say, talk, speak, see, look, sing, sit, swim. Every one has its root attested with that meaning in a published grammar, most with sentence-level examples.

Eleven verbs is not many. A lesson generator with eleven verbs writes narrower lessons than one with fifty-eight. But the fifty-eight were an illusion: a third of them generated a different verb than the lesson claimed, which in language teaching materials is not a rough edge. A child drilled on a wrong verb has learned a wrong verb.

The removals are not deletions. Each blocked verb carries a note recording why it was removed and, where a grammar supplies the correct root and its conjugation class, the exact path to restoring it: add the attested root, verify it round-trips, move it back. Several verbs will return within days. They will return attested.

And the two lists can no longer drift apart. An assertion at server start fails the whole process if a blocked verb reappears on the allowed list. The invariant that used to live in my attention now lives in the code, which is the only place invariants survive.

What I would take from this

Build components that can contradict each other. A generator alone produces fluent output and fluency reads as correctness. The analyzer had independent grounding, a published grammar it was validated against, and that independence is what made the disagreement informative. Two components that share their assumptions cannot catch each other.

Fluent and wrong is the dangerous quadrant. Broken output gets caught by anyone. Correct output needs no catching. Output that is grammatical, natural, and about the wrong thing sails through every review that does not involve a speaker or a grammar. For languages with few speakers and little documentation, that quadrant is where the real damage lives, because wrong material can outlive the ability to correct it.

An extraction-based “unattested” is a claim about your extraction. My first pass declared several verbs absent from the grammars because they were absent from my extracted root list. Going back to the actual PDFs found most of them: one as a causative derivation, one as a loan verb, one under a root my list had missed. The grammars were fine. My pipeline’s picture of them was lossy, and I nearly published the gap as fact.

Shrinking a capability can be the most honest work in the project. Nothing I have shipped recently improved the system as much as cutting its verb list by four fifths. The number that matters was never how many verbs the generator could use. It was how many it could use and be right.

Postscript, two days later

The verbs came back, and then some. Going back to the two grammars properly, full text rather than my extracted lists, recovered four hundred and fifty roots my pipeline had dropped, including several verbs I had confidently declared unattested. Each restoration went through the same gate: attested root, verified round-trip, then back on the list. The generator now draws from just over a hundred verbs, roughly double where it started, except now every one can show the page it came from.

The verb that started all this came back too. Both grammars derive teach as a causative of know, “cause to know,” and the generator now produces exactly that form. The sentence that was fluently wrong two weeks ago is fluently right today, and the difference between those two states is the entire point of building the checks.