Measured Index #1: run index-1-2026-11-20-run2 ========================================================================== started (UTC) : 2026-11-20T19:20:33+00:00 harness : scripts/index_run.py anthropic : 1.2.0 pydantic: 2.13.5 python : 3.12.16 on Linux-7.0.0-1020-azure-x86_64-with-glibc2.43 book default : claude-opus-5 (BOOK.md, Course Module 1 harness row) prices : https://platform.claude.com/docs/en/about-claude/pricing (read 2026-11-20), USD per million tokens limits/model : 80 calls, 120000 tokens; max_tokens single/chain 1024, typed 1000, agent 2048; agent steps 5 suites : route-single (10), route-chain (10), route-typed (10), agent (5) models the key lists (15): claude-fable-5, claude-fable-5-1, claude-fable-5-2, claude-haiku-4-5-20251001, claude-haiku-5-5, claude-opus-4-5-20251101, claude-opus-4-6, claude-opus-4-7, claude-opus-4-8, claude-opus-5, claude-opus-5-5, claude-sonnet-4-5-20250929, claude-sonnet-4-6, claude-sonnet-5, claude-sonnet-5-5 claude-fable-5-2 callable claude-opus-5-5 callable claude-sonnet-5-5 callable claude-haiku-5-5 callable claude-opus-5 callable claude-opus-4-6 callable claude-haiku-4-5-20251001 callable listed, not in Issue 1's plan: claude-fable-5, claude-fable-5-1, claude-opus-4-5-20251101, claude-opus-4-7, claude-opus-4-8, claude-sonnet-4-5-20250929, claude-sonnet-4-6, claude-sonnet-5 -------------------------------------------------------------------------- model claude-fable-5-2 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. OK got 'general' route-chain (3 calls per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' route-typed (1 call per ticket, messages.parse) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' agent (up to 5 steps per question) 1. OK got 'There are **7 headsets** in stock (SKU HS-03), priced at $89' 2. OK got '12 × 45 = **540**' 3. OK got 'The total value of webcams on hand is **$186.00** (3 units o' 4. OK got 'You have more keyboards: 12 on hand (KB-01) versus 7 headset' 5. OK got 'Bringing mice (SKU MS-02) up to 10 units would cost **$195.0' 6. -- got 'For slow-moving items, the usual high-volume methods (EOQ, n' -------------------------------------------------------------------------- model claude-opus-5-5 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. OK got 'general' route-chain (3 calls per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' route-typed (1 call per ticket, messages.parse) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' agent (up to 5 steps per question) 1. OK got 'There are 7 headsets in stock (SKU HS-03). Each one costs $8' 2. OK got '12 × 45 = **540**' 3. OK got 'The webcams on hand are worth **$186.00** in total. There ar' 4. OK got 'We have more keyboards: 12 in stock, compared with 7 headset' 5. OK got 'It would cost **$195.00** to bring mice up to 10 units. - *' 6. -- got 'For slow-moving items, the goal is to keep enough stock to c' -------------------------------------------------------------------------- model claude-sonnet-5-5 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. OK got 'general' route-chain (3 calls per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' route-typed (1 call per ticket, messages.parse) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' agent (up to 5 steps per question) 1. OK got 'There are 7 headsets in stock (SKU HS-03, priced at $89.00 e' 2. OK got '12 × 45 = 540.' 3. OK got 'The webcams on hand (SKU WC-04) are worth a total of **$186.' 4. OK got 'You have more keyboards than headsets. There are 12 keyboard' 5. OK got 'Bringing mice (SKU MS-02) up to 10 units would cost **$195.0' 6. -- got '# Reorder policy for slow-moving items The aim is to avoid ' -------------------------------------------------------------------------- model claude-haiku-5-5 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. OK got 'general' route-chain (3 calls per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' route-typed (1 call per ticket, messages.parse) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. OK got 'general' agent (up to 5 steps per question) 1. OK got 'There are 7 headsets in stock (SKU HS-03).' 2. OK got '12 × 45 = 540.' 3. OK got 'The total value of webcams on hand is **$186.00**. The stock' 4. OK got 'We have more keyboards than headsets: 12 keyboards (SKU KB-0' 5. OK got 'Bringing mice up to 10 units would cost **$195.00**. The mo' 6. -- got "There's no single right answer, but a sound policy for slow-" -------------------------------------------------------------------------- model claude-opus-5 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. OK got 'general' route-chain (3 calls per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' route-typed (1 call per ticket, messages.parse) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'bug' agent (up to 5 steps per question) 1. OK got 'There are **7 headsets** in stock (SKU: HS-03, $89.00 each).' 2. OK got '12 × 45 = **540**' 3. OK got 'The total value of webcams on hand is **$186.00**. - SKU: W' 4. MISS got '**Keyboards** — you have more of those. | Product | SKU | O' 5. OK got '**Cost to bring mice up to 10 units: $195.00** | Field | Va' 6. -- got 'For slow movers, the usual EOQ/reorder-point math gets shaky' -------------------------------------------------------------------------- model claude-opus-4-6 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. OK got 'general' route-chain (3 calls per ticket) 1. MISS got '**billing**' 2. OK got 'billing' 3. MISS got '**auth**' 4. OK got 'bug' 5. MISS got '**auth**' 6. MISS got '**general**' 7. MISS got '**billing**' 8. OK got 'bug' 9. MISS got '**billing**' 10. OK got 'general' route-typed (1 call per ticket, messages.parse) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' agent (up to 5 steps per question) 1. OK got 'There are currently **7 headsets** (SKU: HS-03) in stock, pr' 2. OK got '**12 × 45 = 540**' 3. OK got 'The total value of webcams on hand is **$186.00** (3 units ×' 4. OK got "We have more **keyboards** in stock! Here's the breakdown: " 5. OK got "Here's the summary: | Product | SKU | On Hand | Target | Un" 6. -- got 'Great question! A good reorder policy for slow-moving items ' -------------------------------------------------------------------------- model claude-haiku-4-5-20251001 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. OK got 'general' route-chain (3 calls per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. MISS got 'general' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' route-typed (1 call per ticket, messages.parse) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' agent (up to 5 steps per question) 1. OK got 'There are **7 headsets** in stock. The product is listed as ' 2. OK got '12 * 45 = **540**' 3. OK got 'The total value of the webcams on hand is **$186.00**. Here' 4. OK got "We have **more keyboards than headsets**. Here's the breakdo" 5. OK got 'It would cost **$195.00** to bring mice up to 10 units (you ' 6. -- got "I'd be happy to help you think through a reorder policy for " ========================================================================== results ========================================================================== # model score single chain typed agent cost/run median s fail 1 claude-haiku-5-5 34/35 10/10 9/10 10/10 5/5 0.00327 0.648 0 2 claude-sonnet-5-5 33/35 10/10 9/10 9/10 5/5 0.09058 1.134 0 3 claude-opus-5-5 33/35 10/10 9/10 9/10 5/5 0.17366 1.196 0 4 claude-fable-5-2 33/35 10/10 9/10 9/10 5/5 0.32771 1.073 0 5 claude-haiku-4-5-20251001 32/35 10/10 8/10 9/10 5/5 0.02718 0.668 0 6 claude-opus-5 32/35 10/10 9/10 9/10 4/5 0.39091 1.331 0 7 claude-opus-4-6 28/35 10/10 4/10 9/10 5/5 0.13550 1.632 0 - OpenAI UNMEASURED: no API key in this repository; not run and not estimated - Google UNMEASURED: no API key in this repository; not run and not estimated - Meta (Llama, open weights) UNMEASURED: no endpoint or key in this repository; not run and not estimated - Mistral AI UNMEASURED: no API key in this repository; not run and not estimated - DeepSeek UNMEASURED: no API key in this repository; not run and not estimated score: points out of the scored items (route-single 10, route-chain 10, route-typed 10, agent 5). cost/run: USD for this model's whole pass, logged tokens times the published price. median s: median wall time of one model call, from this runner. fail: items with no usable answer (API error, empty, max_tokens, raise or None, step cap). tokens, claude-haiku-5-5: in 14568, out 3629 (thinking 1021), 63 calls, stop reasons end_turn 56, tool_use 7 tokens, claude-sonnet-5-5: in 15800, out 5898 (thinking 1421), 63 calls, stop reasons end_turn 56, tool_use 7 tokens, claude-opus-5-5: in 14730, out 5737 (thinking 1803), 62 calls, stop reasons end_turn 56, tool_use 6 tokens, claude-fable-5-2: in 14209, out 5351 (thinking 2131), 62 calls, stop reasons end_turn 56, tool_use 6 tokens, claude-haiku-4-5-20251001: in 15670, out 2302 (thinking not reported), 64 calls, stop reasons end_turn 56, tool_use 8 tokens, claude-opus-5: in 18937, out 11849 (thinking 3699), 62 calls, stop reasons end_turn 56, tool_use 6 tokens, claude-opus-4-6: in 15484, out 2323 (thinking not reported), 64 calls, stop reasons end_turn 56, tool_use 8 finished (UTC) : 2026-11-20T19:33:00+00:00 wrote : /home/runner/work/_temp/out/runs/index-1-2026-11-20-run2.jsonl wrote : /home/runner/work/_temp/out/runs/index-1-2026-11-20-run2-results.json exit 0