The alphabet test every frontier model fails.

Sixteen models, one basic question — list a language's complete alphabet. Every one aces English and collapses on the languages it barely saw. See who fails, and run your own model →

Today's models write essays, draft code, and answer legal questions in dozens of languages. But being fluent isn't the same as knowing the basics. We asked 16 of them one simple thing: list every building block a language is made from — its complete alphabet. Every model recites English perfectly. Not one can do it for a language whose alphabet it has barely seen.

Every language has one

An operating alphabet is the complete set of building blocks a written language is made from — and every language has one. English has its 26 letters. Japanese has its kana plus a bounded set of kanji. Korean has Hangul. Arabic and Hebrew have their abjad. Hindi, Tamil, Amharic and Khmer have their letter-and-matrix systems. And each Bantu language has its Full Syllable Inventory — Kinyarwanda 490, Bemba 530, Luvale 245. Different shapes, same job: the foundation every word is built from. This test covers three of them — English letters, Mandarin syllables, and Bantu syllables — but the idea is universal.

One score, out of 26

We report a single number, read as "X out of 26." 26 is the size of the English alphabet — a ruler everyone knows — so every language's result lands on the same scale. 26/26 is perfect. 13/26 means the model produced only half the alphabet. A model loses points two ways: for missing real units, and for inventing ones that aren't real. So a model can't win by padding its answer with guesses.

These alphabets matter as much as English's

We use 26 not to crown English, but because everyone instantly knows what missing a letter of A–Z means. So instead of an unfamiliar number for an unfamiliar alphabet, we translate the failure into terms no one can dismiss: a model's collapse on Kinyarwanda becomes "like missing six letters of A–Z." The scale doesn't shrink other languages down to English — it carries their failures across so an English speaker can feel the size of them.

And that is the real point. A language's alphabet is foundational infrastructure — the thing that makes the language teachable, testable, searchable, and fixable by a machine. If we'd call a model broken for failing English's A–Z, then failing the equivalent alphabet of Kinyarwanda or Luvale is the same kind of broken. No system has truly mastered language while it treats English's alphabet as essential and everyone else's as optional. That is what L26 measures — not whether a model is clever, but whether it holds every language's foundation to the same standard.

The pattern: models know the alphabets they've seen a million times

The failures aren't random. They come down to one thing — how often the model has seen the alphabet. English's A–Z appears in billions of pages: books, classrooms, charts, songs, primers. Mandarin's syllables are printed in tables and textbooks. But a Bantu language's complete set of syllables has almost never been written out anywhere as one finished list — so the models have hardly seen it, and never learned it.

The drop isn't gradual. It falls off a cliff exactly where the alphabets stop being common. Size doesn't explain it — Luvale is the smallest set (245) and one of the hardest.

The leaderboard — blind: the model gets nothing but the question

Here the model is handed no hints at all — no rules, no list — and simply asked to produce the alphabet, the way you'd ask a child for the ABCs. Every score is out of 26.

16 models · 682 scored answers · 5 runs each · out of 26 · ground truth from the calibrated abs_syllables inventories · deterministic scoring, no AI judge. The best any model manages on the Bantu alphabets is Gemini 3 at 13.9/26 — barely half.

It's the same for everyone — open or closed, US or not

No group has cracked it, because it's not about who built the model or how it was released. It's about how much the alphabet appears in the text models learn from.

The good news: hand them the list, and they get it right

Give a model the actual set of building blocks — the calibrated Full Syllable Inventory — and the same model that just failed springs back toward a perfect 26. It could always do the work; it was only ever missing the list. That's the whole point: the alphabet, written out, is the thing that fixes this. Watch each model's blind→with-the-list recovery on the live board.

How the test works

Deterministic scoring, no AI judge: the score is recall × precision × 26. The bar: a perfect pass needs every real unit and zero invented ones. No peeking: the real answer is never shown to the model. Ground truth: the calibrated abs_syllables Full Syllable Inventories — Kinyarwanda 490, Bemba 530, Luvale 245. Each model is run 5 times. This is Iteration 2: the "blind" question was rebuilt to give the model nothing up front (an earlier version accidentally described the language's structure, which flattered the scores), and every answer is graded against the canonical inventory. The corrected numbers are lower — and truer.

The thing that closes the gap already exists

The benchmark is the measurement. The Full Syllable Inventory is the fix. It gives a language a proper, written-out alphabet — one clear, complete list that tells models, speech tools, dictionaries and evaluators exactly which units are real. English has had this forever, as A–Z. For Bantu languages, BantuNomics has built the same thing at the syllable level: native-checked, standardized, versioned inventories. Without them, models guess. With them, models can be measured, taught and corrected against a standard.