Measured

Measured IndexLive

Measured Index, issue 1

A comparison of language models on Hassan's own test sets: how often each one gets the answer right, what one full pass costs, and how long a call takes. Every number links to the raw run that produced it, and nothing is shown before that run exists.

Results, issue 1

Every number below links to the raw run file it came from: runs/index-1-2026-11-20-run2.txt

Ties on score

  • Claude Sonnet 5.5, Claude Opus 5.5 and Claude Fable 5.2 have the same score.
  • Claude Haiku 4.5 and Claude Opus 5 have the same score.

Within each group the order is the harness's tie-break, not a difference in score: at equal score, index_run.py orders models by cost per run, lowest first, then by median call time, lowest first, and a model whose cost or time could not be measured goes after the others.

Measured Index results: rank, model, score, cost per run, median call and failures. Each number links to the raw run file.
RankModelScoreCost per runMedian callFailures
1Claude Haiku 5.5claude-haiku-5-5The highest score of the models that finished their pass.34of 35USD 0.00330.65 s0
2Claude Sonnet 5.5claude-sonnet-5-5Fewer points than Claude Haiku 5.5; most of the gap is on “Route a ticket, typed output”. Same score as Claude Opus 5.5 and Claude Fable 5.2, and first of them by the tie-break, with the lowest cost per run.33of 35USD 0.0911.13 s0
3Claude Opus 5.5claude-opus-5-5Same score as Claude Sonnet 5.5; after it by the tie-break, at a higher cost per run.33of 35USD 0.1741.20 s0
4Claude Fable 5.2claude-fable-5-2Same score as Claude Opus 5.5; after it by the tie-break, at a higher cost per run.33of 35USD 0.3281.07 s0
5Claude Haiku 4.5claude-haiku-4-5-20251001Fewer points than Claude Fable 5.2; most of the gap is on “Route a ticket, three calls”. Same score as Claude Opus 5, and first of them by the tie-break, with the lowest cost per run.32of 35USD 0.0270.67 s0
6Claude Opus 5claude-opus-5Same score as Claude Haiku 4.5; after it by the tie-break, at a higher cost per run.32of 35USD 0.3911.33 s0
7Claude Opus 4.6claude-opus-4-6Fewer points than Claude Opus 5; most of the gap is on “Route a ticket, three calls”.28of 35USD 0.1351.63 s0

Side by side, test by test

Each row is one test, each column one model in rank order. The bar is the share of that test's items answered right, so a short bar in one row and a long one in another shows where a model's points come from.

Points per test for each measured model, in rank order, with cost per run, median call and failures.
Claude Haiku 5.5Claude Sonnet 5.5Claude Opus 5.5Claude Fable 5.2Claude Haiku 4.5Claude Opus 5Claude Opus 4.6
Route a ticket, one call10of 1010of 1010of 1010of 1010of 1010of 1010of 10
Route a ticket, three calls9of 109of 109of 109of 108of 109of 104of 10
Route a ticket, typed output10of 109of 109of 109of 109of 109of 109of 10
Answer with tools5of 55of 55of 55of 55of 54of 55of 5
Cost per runUSD 0.0033USD 0.091USD 0.174USD 0.328USD 0.027USD 0.391USD 0.135
Median call0.65 s1.13 s1.20 s1.07 s0.67 s1.33 s1.63 s
Failures0000000

Why each model ranks where it does

  1. 1Claude Haiku 5.5: The highest score of the models that finished their pass.
  2. 2Claude Sonnet 5.5: Fewer points than Claude Haiku 5.5; most of the gap is on “Route a ticket, typed output”. Same score as Claude Opus 5.5 and Claude Fable 5.2, and first of them by the tie-break, with the lowest cost per run.
  3. 3Claude Opus 5.5: Same score as Claude Sonnet 5.5; after it by the tie-break, at a higher cost per run.
  4. 4Claude Fable 5.2: Same score as Claude Opus 5.5; after it by the tie-break, at a higher cost per run.
  5. 5Claude Haiku 4.5: Fewer points than Claude Fable 5.2; most of the gap is on “Route a ticket, three calls”. Same score as Claude Opus 5, and first of them by the tie-break, with the lowest cost per run.
  6. 6Claude Opus 5: Same score as Claude Haiku 4.5; after it by the tie-break, at a higher cost per run.
  7. 7Claude Opus 4.6: Fewer points than Claude Opus 5; most of the gap is on “Route a ticket, three calls”.

Ranked by score, highest first. Equal scores are ordered by cost per run, lowest first, then by median call. A model whose pass did not finish is shown apart, without a rank.

The run's files

The book's repository is private, so the run's files are copied here unchanged. Each one is the book repository's file at commit 46c3ac3; the release check hashes every copy again before the site is built.

  • index-1-2026-11-20-run2.txtBook repository path: runs/index-1-2026-11-20-run2.txt · SHA-256: 6b5d7b38bebeba49bc6e45eee4868b7b41be94db073f5a7b0bed19e2b81acebb
  • index-1-2026-11-20-run2.jsonlBook repository path: runs/index-1-2026-11-20-run2.jsonl · SHA-256: 518045967028ca33e5bd771674c67ebce9a860f09f91039e5dab8220ad594e8f
  • index-1-2026-11-20-run2-results.jsonBook repository path: runs/index-1-2026-11-20-run2-results.json · SHA-256: 52d545f9584ea4412fd8c43954aecface1946603928fb3ad426ab26a34dfcdc4
  • index-1-2026-11-20-run2-env.txtBook repository path: runs/index-1-2026-11-20-run2-env.txt · SHA-256: 1bb95c86363eb3d9af9bca9475d69cd948cd9272b38bcb3f607d58a269066d6f

What this run cost in all, from its own cost lines: USD 1.15.

Run with scripts/index_run.py at commit 378fee4.

An earlier partial run the same day

Before this run, a first run of the same harness on the same tests stopped when the API credit ran out. It is kept here unchanged, because it shows something one run alone cannot: how much a score moves from one run to the next. Nothing from it is ranked or priced in the tables above.

Claude Opus 5, Claude Opus 4.6 and Claude Haiku 4.5 did not finish it.

Claude Fable 5.2, Claude Opus 5.5, Claude Sonnet 5.5 and Claude Haiku 5.5 finished it, each with a score of 33 of 35.

Claude Haiku 5.5 scored 33 there and 34 in this run. So a gap of one point can be chance: the same model moved by one point between two runs on the same day.

Claude Sonnet 5.5, Claude Opus 5.5 and Claude Fable 5.2 scored the same in both runs.

What the earlier run cost in all, from its own cost lines: USD 0.817.

Its files

  • index-1-2026-11-20.txtBook repository path: runs/index-1-2026-11-20.txt · SHA-256: 538b0d21a0f856e0edfafa68e5531e54531b35cbedf07a32049610b9adfedede
  • index-1-2026-11-20.jsonlBook repository path: runs/index-1-2026-11-20.jsonl · SHA-256: aadf35e34d15d56148f77a8b9f2b05bea6ee4e329c872b45e276693db6d679bf
  • index-1-2026-11-20-results.jsonBook repository path: runs/index-1-2026-11-20-results.json · SHA-256: c5ae953756f06200c25a2aa3e00e784daabd64372f2f71c0ebb9b9877909a9cf
  • index-1-2026-11-20-env.txtBook repository path: runs/index-1-2026-11-20-env.txt · SHA-256: 4ab1cc3c7b46c7fb1b80e1552cd876a39feced188d2594d00348020766b51cdb

Not measured

There is no API key for these providers in the harness, so nothing was run for them and nothing is estimated in their place.

  • OpenAI · Not measured: no key
  • Google · Not measured: no key
  • Meta (Llama, open weights) · Not measured: no key
  • Mistral AI · Not measured: no key
  • DeepSeek · Not measured: no key

What each test measures

Four tests, 35 scored items per model. The three routing tests use the same 10 tickets and the same answer key, so the only thing that changes between them is how the model is asked.

Route a ticket, one call

10 scored items

What it measures
Can the model put a support ticket in the right one of 4 queues when asked directly, and answer with the queue's name only?
How it is scored
One point when the reply, trimmed and in lower case, is exactly the queue in the answer key. 10 tickets, one call each.
Where points are lost
A model loses the point by being wrong, or by being right in the wrong form: a sentence, a queue in quotes or in bold. A strict reader counts both the same, as a program would.
How to do better
Ask for one word and name the words allowed. Better still, use a typed output, as in the third test, so the form cannot drift.
Test set and key:
scripts/ch2_course_collapse.py

Route a ticket, three calls

10 scored items

What it measures
The same tickets, routed by a chain: summarize the ticket, list the queues that could own it, then pick one. It measures what the extra steps add or lose.
How it is scored
One point when the last call's reply is exactly the key's queue. Each ticket takes three calls, so cost and time add up three times.
Where points are lost
Each step sees only the step before it. A summary can drop the detail that decided the queue, and the last step can answer with more than one word.
How to do better
Keep a chain only where a step adds something a single call cannot. Compare the two on the same answer key before keeping it.
Test set and key:
scripts/ch2_course_collapse.py

Route a ticket, typed output

10 scored items

What it measures
The same tickets, with the answer parsed into a type that accepts only the 4 queue names. It measures whether the model fills the type, and with the right queue.
How it is scored
One point when the typed queue matches the key. A call that raises an error, or returns nothing, is a failure and scores zero.
Where points are lost
The type rules out wrong forms. What is left is the decision itself, plus any call whose output could not fill the type.
How to do better
Read the failed calls in the raw run: they show where an output stopped or drifted, which a single score hides.
Test set and key:
scripts/ch3_course_route_typed.py

Answer with tools

5 scored items

What it measures
A small inventory agent with two tools, a stock lookup and a calculator, answers 6 questions in up to 5 steps each. 5 of them have one right value; the open-ended one is asked but not scored.
How it is scored
One point when the final answer holds the right value as a number of its own, not inside a longer number, and, for the comparison question, names the right item as the larger. Hitting the step limit, or an answer cut off by the token limit, is a failure.
Where points are lost
A model loses points by doing the arithmetic in its head and getting it wrong, by looking up the wrong item, or by running out of steps or tokens before it answers.
How to do better
Describe each tool by when to use it, keep a step limit that stops the loop, and grade the trace as well as the answer.
Test set and key:
scripts/ch4_course_agent.py

One ticket, asked three ways

The exact text each routing test sends for the same ticket, side by side. The answer key for this ticket: billing

Route a ticket, one call
Route this support ticket to exactly one queue.

billing  - charges, invoices, refunds, plan renewals, anything about money
auth     - sign-in, passwords, sessions, lockouts, account access
bug      - the software is behaving incorrectly: crashes, errors, broken features
general  - anything else, including permissions, sharing, and "who do I ask"

Reply with one word: billing, auth, bug, or general.

Ticket: I was charged twice for the same month.
Route a ticket, three calls
  1. Step 1
    Summarize this support ticket in one sentence:
    
    I was charged twice for the same month.
  2. Step 2
    Which of these queues could own this? billing, auth, bug, general
    
    <the summary>
  3. Step 3
    Pick exactly one queue from this list. Reply with one word.
    
    <the candidates>
Route a ticket, typed output
Route this support ticket to exactly one of: billing, auth, bug, general.

I was charged twice for the same month.

The answer must fit

queue: Literal["billing", "auth", "bug", "general"]

And one agent question

What is the total value of the webcams on hand?

Tools: lookup_stock, calculate. The answer key's value: 186.

Reading a row

Score
Points out of 35 scored items, one pass per item in each run. A point is one answer, so a gap of one point can be chance: the earlier partial run shows the same model moving between two runs on the same day.
Cost per run
What this model's whole pass cost: the input and output tokens the API reported, times the price per token Anthropic publishes. Thinking is billed as output, so a model that thinks more pays more for the same answer.
Median call
The median time one call took, in seconds, timed on the machine that ran the test. It includes the network and any thinking, and it comes from one machine on one day, so read it as a comparison, not a promise.
Failures
Items with no usable answer: an API error, an empty answer, an answer cut off by the token limit, a typed call that failed, or an agent that ran out of steps. A failure scores zero and is also counted on its own, so a wrong answer and no answer are not confused.

How the order is decided

Ranked by score, highest first. Equal scores are ordered by cost per run, lowest first, then by median call. A model whose pass did not finish is shown apart, without a rank.

Sponsored, unknown, not measured

Sponsored
Nothing. No model maker paid for this issue or chose what it tests.
Settings
Each model is called as a developer would call it by default: no system prompt, no temperature or thinking setting, and the same token limits for all.
Test sets
The tests are the book's own exercises and answer keys, written before this Index existed. No test was written or changed for it.
Not measured
Safety is not in this issue: no safety test set is part of the harness yet. Models from other providers are not measured, because there is no key for them, and nothing is estimated in their place.
Unknown
Each item is asked once per run. Two runs on the same day give a first look at how much a score moves between runs, not a measure of it. Prices are the ones Anthropic publishes; they change, so the cost of a pass holds for the day of the run.

Run it yourself

The harness, the test sets and the raw output are kept with the book's measurement scripts. One command runs the whole Index without calling any model, so every test can be read before a single call is paid for.

  1. Read the method: python3 scripts/index_run.py describe
  2. Run it with no call: python3 scripts/index_run.py run --name index-1-YYYY-MM-DD --dry-run
  3. Estimate the cost of a paid run: python3 scripts/index_run.py estimate

Harness: scripts/index_run.py · Test set and key: scripts/ch2_course_collapse.py, scripts/ch3_course_route_typed.py, scripts/ch4_course_agent.py

See alsoThe MethodBuild log

Waitlist

Waitlist opens soon

There is no form here yet, so this page collects nothing about you.