The evidence
Can a model produce the FSI?
Our claim is testable: a model that "knows" a Bantu language should be able to produce its complete legal syllable inventory. Most frontier models, working from flat text, produce the short CV row but miss the prenasalized and long-vowel forms the language actually uses. Pick an onset and see the gap against the real Bemba FSI.
What a flat-text model returns
Illustrative of the typical flat-text output for this row — not a benchmarked result. The named-model scorecard below is the real run.
The real Bemba FSI
The actual legal syllables are the answer key — they live in the FSI and the evaluation, never on this public page. What you see here is the size of the gap, not the answer.
The run — 10 languages × 5 frontier models · L0
What the run actually found
Each model gets one open task per language — produce the legal syllables — and the output is scored by deterministic set/string math against the released FSI (no LLM judge). Granularity is entirely in the review: the model is never cued per axis. Scores are each model averaged across the languages it completed, the variance-control step that makes the picture stable. Two field-wide gaps dominate.
30/100 Inventory blindness
Across 10 languages and 5 models, frontier models recover only ~30/100 of the legal syllable inventory — the closed set the language actually uses. This is constitutive, not a tuning miss.
11/100 Vowel-length blindness
Length Awareness averages ~11/100: models almost never produce the long-vowel forms that carry meaning. Verified against raw output, not a parsing artifact — flat text discarded the distinction.
What is not the family-wide gap: Alphabet Integrity (~97) and Fabrication Resistance (~76). Frontier models mostly respect the alphabet and mostly don't fabricate — the blindness is to the inventory and to vowel length.
Core critical attributes
The five axes, family-wide
| Attribute | What it tests | Family avg | Closed by |
|---|---|---|---|
| Inventory Command | Knows the closed legal syllable set — produces real members, not a CV sketch. | 30/100 · systemic gap | The FSI catalog — the closed, enumerable legal-syllable space per language. |
| Fabrication Resistance | Does not invent non-member syllables (NA / hallucination control). | 76/100 | FSI as a validation oracle — a closed set to check generations against. |
| Alphabet Integrity | Uses only the letters the language actually has; respects context-licensing. | 97/100 | Already strong field-wide — the grapheme rulebook hardens the edge cases. |
| Length Awareness | Restores phonemic vowel length (long vowels), discarded by flat text. | 11/100 · systemic gap | The A12 codec length layer — restores the long/short distinction orthography drops. |
| Calibration | Knows whether it knows — confidence tracks actual coverage. | 65/100 | The benchmark itself — a measured self-awareness signal per language. |
Per-model scorecard — FSI Command Score
Named frontier models
The headline is the integrative FSI Command Score (a correct syllable needs onset, nucleus, length and alphabet right at once). The five attributes are the assessment body; the verdict is a deterministic style classification.
| Model | FSI Command | Inventory | Fab-resist | Alphabet | Length | Calibration | Verdict |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 10 langs |
26 | 34 | 88 | 100 | 0 | 82 | precise-but-narrow clean (won't invent) but recovers only a fraction of the inventory |
| GPT-4o 10 langs |
24 | 30 | 83 | 96 | 5 | 70 | precise-but-narrow clean (won't invent) but recovers only a fraction of the inventory |
| Gemini 2.5 8 langs |
24 | 53 | 75 | 95 | 0 | 54 | reckless-recall produces a lot but a large share is fabricated |
| DeepSeek 10 langs |
24 | 29 | 86 | 96 | 0 | 84 | precise-but-narrow clean (won't invent) but recovers only a fraction of the inventory |
| GPT-4o-mini 9 langs |
-2 | 5 | 46 | 98 | 50 | 35 | collapsing produces almost nothing real and what it does is unreliable |
Scores are a frozen snapshot of the FSI-CX run (2026-06), scored by deterministic set/string math vs released FSI — no LLM judge. A downloadable benchmark kit (test set + deterministic scorer + raw model outputs) is available to partners so every number above can be re-derived independently — including running your own model through the same harness.