AI, tested and proven.
Three questions. Guess a number, then watch it get measured.
Question 1 of 3From chapter 1 of the book
One prompt, sent to the same AI model 20 times at its default settings. How many distinct answers came back?
The three measurements
Each number below comes from a run in the book's own repository.
Question 1 of 3From chapter 1 of the book
One prompt, sent to the same AI model 20 times at its default settings. How many distinct answers came back?
Measured: 10 distinct answers out of 20.
A scale from 1 to 20. The measured value, marked with a gold tick, is 10 distinct answers out of 20.
- What happened
- 10 distinct answers in 20 calls. The most common one came back 8 times.
- Why the gap
- A model picks its next words by sampling, so by default the same prompt does not give the same text twice. All 20 answers agreed in meaning: only the wording moved.
- How it was measured
- 20 calls with one prompt to claude-opus-4-6. Identical replies were grouped and the groups counted.
- In the book
- Chapter 1 of the book runs the same 20 calls under a second sampling setting and counts again.
- Run file:
- runs/ch1-sampling-2026-09-07.txt
- Script:
- scripts/ch1_sampling.py
Question 2 of 3From chapter 8 of the book
An AI assistant with three tools gets one short question. The instructions for its tools travel with the question. What share of that first message was instructions?
Measured: 98.0 % of the first message.
A scale from 0 to 100. The measured value, marked with a gold tick, is 98.0 % of the first message.
- What happened
- The first message was 748 tokens long (a token is a piece of a word). 733 of them were instructions for the tools. The question itself was 8.
- Why the gap
- The instructions for every tool are sent in full with each message, whether or not a tool gets used. A short question does not make them shorter.
- How it was measured
- Counted with the model provider's own token counter on claude-opus-4-6, by comparing the message with and without each part.
- In the book
- Chapter 8 of the book measures what happens to this cost when one assistant is split into several.
- Run file:
- runs/ch5-context-composition.txt
- Script:
- scripts/ch5_context_composition.py
Question 3 of 3From chapter 14 of the book
An AI assistant handles 12 conversations, one after the other. Then it handles the same 12 with 8 going at once. How many times faster did it finish?
Measured: 5.75 times faster.
A scale from 1 to 12. The measured value, marked with a gold tick, is 5.75 times faster.
- What happened
- One after the other, the 12 conversations took 123.4 seconds. With 8 going at once, they took 21.5 seconds: 5.75 times faster, not 8.
- Why the gap
- With 12 conversations and 8 going at once, there are two rounds, and the second is only half full. Each conversation also took about as long as before: the waiting overlaps, it does not shrink.
- How it was measured
- The same 12 conversations were run on claude-opus-4-6 with 1, 2, 4 and 8 going at once, and the total time on the clock was recorded for each.
- In the book
- Chapter 14 of the book looks at what running conversations side by side costs, and when not to do it.
- Run file:
- runs/ch14-async-2026-10-28.txt
- Script:
- scripts/ch14_async_run.py
What Measured is
Most AI content shows a demo. Measured shows the number: what was tested, what happened, why, and how it was measured.
The book
AI Engineering, Measured
by Hassan Mansour
The book. Every number in it came from a run. Launching soon.
One step leads to the next
Free
The three questions on this page. Free tools will be added here as each one is built.
Courses
Measured Academy, with a test in every module. No course is open yet.
For teams
A hands-on workshop, 2 hours or 3.5 hours, quoted for your team after a free 20-minute fit call.
The Measured family
- Launching soon
Measured Academy
by Hassan Mansour
Courses with a test in every module. For learners aged 18 and over.
- Live
AI at Work, Measured
A hands-on team workshop, 2 hours or 3.5 hours. The first pilot has not run yet.
- Live
Proofline
An open-source multi-agent pipeline with evals and a human approval gate, built in public.
- Live
Measured Lab
Check AI work against rules you can read, in your browser: Evaluate, Observe and Secure, with result links anyone can run again. Open now.
- Live
Measured Index
by Hassan Mansour
A model comparison on Hassan's own test sets: score, cost per run and speed, each number linked to its raw run file. Issue 1 is out.
Who measures
Hassan Mansour
Senior Applied AI Developer
Works on agentic AI and RAG in a regulated enterprise.
- 6+ years at Bell Canada building production systems, about the last 3 in AI
- Bell Hackathon winner; two-time recipient of the Bell Canada Excellence and Innovation Award
- NSERC and FRQNT research scholarship recipient
- BSc Computer Science, Concordia; MSc candidate, Multilingual Digital Communication, McGill
- Works in English and French; native Arabic
Waitlist
Waitlist opens soon
There is no form here yet, so this page collects nothing about you.