By Model VS · Guide text updated . A high exam score does not measure how the same configuration feels in an interactive session. This page defines the clocks — time-to-first-token, throughput, and thinking-token time — and how to record them without mixing them into the quality cell. It contains no provider latency figures. Write to us if a file link on this page 404s.
Why can a high benchmark score still feel slow?
A published quality score measures whether a completed attempt met an exam’s rubric. It does not measure how long you waited for the first visible token, how fast the rest of the answer arrived, or how many thinking tokens a reasoning setting spent before anything useful appeared. A high LiveBench category or a high SWE-bench resolved percent can belong to a configuration that feels worse, in an interactive session, than a lower-scoring id you can actually sit with. The score can still be true. The feeling is a different fact, and it has to be timed on your own calls.
Interactive work cares about three waits that a completed-exam score is silent on: the pause before anything appears, the smoothness of the stream once it starts, and the total time to an answer you can use. A batch coding agent that runs overnight can tolerate a long total time. A support reply that a person is watching cannot. Writing one latency budget for both jobs is how a high-scoring thinking configuration gets dropped into a live chat and then blamed for being “bad,” when the exam never claimed to be a clock.
Thinking and reasoning settings make the gap more visible. Source ids often encode a thinking budget or an effort suffix. The quality number, if you cite one, belongs to that suffix. The hidden trace may be long, billed, and invisible in the user-facing answer. A person on the other side of the screen experiences that trace as delay. This page will not invent a millisecond figure for any named model, and it will not say that one vendor is a fixed amount faster than another. Model VS snapshots do not store latency. The saved Arena, LiveBench, and specialist files have no latency, time-to-first-token, or throughput key. Absence is not a zero-millisecond result. It means the cell stays blank until you fill it from calls you ran.
What should you measure as time-to-first-token?
Time-to-first-token is a wall-clock you define: from the moment the request is sent to the moment the first token you chose to count arrives. Write down which token that is. For a thinking configuration, the first reasoning token (if the API exposes it) and the first token of the user-visible answer are different events. The person using the product usually sees the second. If you only record the first, you will understate the wait they felt. If the stream includes a role header or a punctuation token before the answer, say whether you counted it.
Measure this on your own calls, on the full id you will actually use, in the region you will actually hit. Freeze the prompt length, the supplied context, and whether a cached prefix was warm. A short greeting and a long pasted transcript are not the same test. Run the measurement more than once. Report a distribution of your runs — a median, and a tail you chose in advance, such as the longest run or a high percentile — with the count of runs and the date. This page will not tell you what a good median is, and it will not supply a typical provider figure for you to copy. A number remembered from a slide, with no call and no date, is not a measurement.
End-to-end time-to-first-token bundles the network, any queue, a possible cold start, and the model’s own first-token work. For a product decision, end-to-end is what the user felt. If you can separate those pieces, label each one. If you cannot, label the figure as end-to-end and do not pretend you isolated the model. Discarding a warmup run is allowed when you write that you discarded it. Hiding the slow first call because it was inconvenient is not a measurement.
How is throughput different from a thinking-token bill?
Throughput is how fast tokens arrive after generation is underway: output tokens per unit of generation time, or a total visible-token count divided by the time from the first of those tokens to the last. It is not time-to-first-token. A run can clear a time-to-first-token budget and then trickle, or it can start slowly and then stream quickly. Interactive feel is both clocks, plus the total time to a usable answer. Write them in separate cells. Do not add them into a single invented “snappiness” score and put our name on it.
Thinking-token cost is a third fact. Tokens spent in a hidden reasoning trace may be billed, they take time, and they may not appear in the visible answer. If the API reports a reasoning or thinking token count, write it next to the visible output count. If it does not, write that the thinking spend is unknown, and still measure the wait until the first user-visible answer token. Those thinking tokens can inflate both latency and the bill without changing the length of what the user reads. A high exam score on a high-effort thinking id includes that budget. If you will call a cheaper or faster setting, you do not own that score.
The bill those tokens produce is not a Model VS price. Cost-per-quality triage is the sister sheet for placing a public list price you copy, with a URL and a date, next to one saved quality field. That page already treats latency as a separate concern and refuses to fold it into a site-made ratio. This essay is that concern: time, and the thinking tokens that consume it, measured on your calls. You may care about both columns in the same meeting. They are still two columns. A cheaper id that misses the latency budget is not a bargain for an interactive job, and a fast id that fails the quality cell is not a better interactive model. It is a failed fast answer.
| Clock | What you measure | What it is not |
|---|---|---|
| Time-to-first-token | Wall-clock from sending the request to the first token you chose to count, with that choice written down. | Not throughput, and not a quality score. |
| Throughput | Visible output tokens per unit time after generation is underway, or total visible tokens divided by that generation window. | Not the wait before the first token. |
| Thinking-token time | Time, and token count when the API reports it, spent before the user-visible answer. | Not a license to keep a high-effort exam score on a faster setting. |
| Time to a usable answer | Wall-clock until the output meets the acceptance rule you already wrote, including retries the policy allows. | Not a substitute for pass or fail on that rule. |
How do you time interactivity without confounding quality?
Time and quality are separate columns on the same frozen setup. Do not use “faster” as a proxy for “better answers,” and do not use a higher exam score as a proxy for a tolerable wait. Run the clocks on the same prompts, the same ids, and the same acceptance rule you would use for a quality check. A faster id that fails the rubric is not more interactive in a useful sense. A slower id that passes is not automatically too slow; that depends on the budget you wrote, not on a preference for speed in the abstract.
Do not shorten the prompt, drop tools, or trim the context on the slow id to make the clock look kinder. Do not compare a streaming user interface to a buffered one and attribute the difference to the model. Do not turn a thinking budget off for the timing pass and on for the quality pass. If you change the thinking budget, the suffix, or the endpoint, you have a new configuration. Time it again, and re-score quality on the setting you will actually call. The old exam number stays with the old id.
Say what you did with warmup, concurrency, and retries. A first call may be slower than the next ones; write whether you discarded it. A single-user timing is not a multi-user service level. If you need a figure under load, measure under load, and do not present the quiet run as the loaded one. Retries belong on the clock that the user would experience: if the policy would retry a timeout in production, the time to a usable answer includes that retry. If it would not, do not hide a timeout by retrying off the record.
The decision record already has a place for elapsed time next to pass or fail. Fill that place with the clocks defined here, not with a marketing adjective. This site does not run your timing harness, does not store figures you might jot down, and does not publish a latency leaderboard. The numbers, if any, are yours, dated, and tied to the id and the prompt length you actually used.
When does a latency budget remove a high-scoring id?
A latency budget is a hard constraint, in the same family as a region you cannot use or a license you cannot accept. Methodology already keeps price, latency, and data-handling terms separate from a public measurement. How we rate lists a latency budget beside your data and your deployment constraints, and it does not define the clocks. This page does. Write the budget in the units you will measure: a median time-to-first-token, a tail you are willing to live with, a throughput floor if the stream has to keep up, a time to a usable answer, and the prompt lengths those figures were written for. This page will not fill those thresholds in for you.
If a high-scoring suffix cannot meet that budget on your calls, it leaves the shortlist for this job. The exam score remains a true fact about that exam. It is not a ticket into an interactive product. A lower-scoring id that meets the quality rule you wrote and stays inside the budget can be the right pick for this job. That is not a cost-per-quality ratio, and it is not a Model VS ranking. It is a constraint doing the work a constraint is for.
Do not reuse an interactive budget for a batch agent without saying so, and do not reuse a batch budget for a live reply. If you relax the budget, write the new number and the reason. If you meet the budget by switching to a faster configuration, the quality cell on the old configuration is retired. Run the quality check again. When both columns are filled for the id you will actually call — one quality result on your cases or a cited public field you have not pretended is a clock, and one set of timings from your calls — you can decide. You are still not owed a winner. A high exam score that misses the budget is a documented miss, not a dilemma this site will arithmetic away.
Questions about latency and benchmark quality
Does a high LiveBench or SWE-bench score include latency?
No. Those scores describe completed attempts on their exams. They do not record time-to-first-token or throughput on your calls.
Is time-to-first-token the same as tokens per second?
No. Time-to-first-token is the wait before the first token you are measuring. Throughput is how fast tokens arrive after generation is underway. A run can be acceptable on one clock and fail the other.
Do thinking tokens belong in a latency measurement?
If a thinking setting spends tokens before the user-visible answer, that time is part of the wait the user feels. Record thinking tokens separately when the API reports them, and keep them off the quality cell. They can also change the bill; that bill is not a Model VS price.
Can a faster configuration keep the score of a slower one?
No. A suffix or thinking budget is part of the id the score belongs to. If you turn thinking down to meet a latency budget, you need a new quality check on the setting you will actually call.