Alphabet Test · Phased

Cold, then assisted — measure the tool-lift

Run the same model twice on one language: a cold pass with no tools, then an assisted pass where it may search and use tools. We grade both together and show the tool-lift = assisted − cold. If tools barely move the score, that's the proof: the Full Syllable Inventory isn't sitting on the open web to be scraped or RAG'd — the pilot is the only source.

No grade is shown between the two passes. Enter the cold attempt, reveal the assisted step, enter that, then grade both at once. "No tools" is honor-based — the same blind/scaffolded protocol the L26 benchmark uses.

Language

Model name (optional, self-reported)

Pass A — Cold no tools

Ask the model to produce the complete syllable set for Bemba (bem) from its own knowledge — no search, no tools. Paste its list.

Cold attempt — syllables (comma or space separated)

Pass B — Assisted tools allowed

Now let the model search the web and use any tools for the same task, same language. Paste the assisted list. (Give it a genuine chance — the point is to see how far tools get it.)

Assisted attempt — syllables

Have your agent do both passes in one call

The same phased grader is one HTTP POST — your agent produces both attempts and gets the tool-lift back:

POST https://fsi.bantunomics.com/api/v1/alphabet-test-phased
{ "language": "bem", "cold": [ … ], "assisted": [ … ], "model_name": "your-model", "duration_ms": 0 }

Also available as the MCP tool fsi_phased_test. Connect your agent from your workspace after signing in.