By Model VS · Guide text updated . This page starts after a public shortlist already has names. The 30-minute shortlist ends with at most three ids and the cases you will run next. The bake-off is those cases, and whatever larger sample the decision is worth. It is not a new leaderboard, and it is not a Model VS score. Write to us if a file link on this page 404s.
What is a private bake-off allowed to decide?
A private bake-off is a comparison of already-shortlisted model ids on work you own and are allowed to test. Public tables can support the shortlist. They cannot accept a candidate for your job. The bake-off’s job is narrower than a company-wide evaluation and narrower than a new ranking: of these ids, on these cases, with this frozen setup, which ids meet the acceptance rule often enough to adopt, which failure modes remain, and whether the sample is too thin to say.
That is all it is allowed to decide. It is not allowed to invent a Model VS composite, to wash a public rating with a private pass rate, or to stand in for a population-wide success figure. Three tickets that go well are a successful smoke test. They are not a rate. A rate would need a sample you could defend to the person who inherits the decision, including the cases that failed.
The names on the sheet should already be full source identifiers, including the effort or thinking suffix you will actually call. If the meeting that produced the names is still arguing about Arena intervals, it is not yet a bake-off. Close that argument with the interval essay, then come back. If the sheet still has a long list of ids, it is still a shortlist meeting. The bake-off is expensive enough, in reviewer time and in API spend, that it should not be used to rediscover a hard constraint you could have applied from a license paragraph or a region you cannot use.
How we rate already says a leaderboard position is evidence for shortlisting, not a guarantee on your data, and it already allows an inconclusive private check. This essay is the protocol for that check, starting after the names exist.
What has to be frozen before the first call?
Freeze the comparison before anyone generates an answer. The record has to be complete enough that a later reviewer can rerun the same call: the exact model identifier and the date, the prompt template and system message, the supplied context, the tool names and versions, the sampling settings, the reasoning or thinking budget, the retry policy, the timeout, and the endpoint or region you will hit. If an API version string is visible, keep it. If a temperature is left at a product default, write that it was the default, and write which product you opened.
Each candidate gets the same permitted inputs. If one system will not run without a different scaffold, a different tool, or a longer context window, you are comparing configured systems, not isolated model ids. That comparison can be the right one. Label it. Account for the extra work and the extra cost. Do not silently give one id a retriever, a longer transcript, or a second attempt and then treat the pass rates as a model result.
Write the acceptance rule before the first completion arrives. For code, that is the test command and what you will inspect in the patch besides a green run. For grounded answers, it is the supplied material and the rule for unsupported claims. For writing, it is a rubric with enough grain that two reviewers can disagree in words rather than in taste. An expected output, or an explicit rubric, has to exist while the cell is still empty. A rubric invented after you have seen which vendor stumbled is not a rubric. It is a post-hoc defense.
If you change the prompt, the tools, the thinking budget, or the retry policy after looking at a result, that run is over. Save it as a development note if you want. Start a new bake-off for the new setup, on cases that were not used to tune the change. Mid-flight edits are how a “winner” appears that only exists on the prompt you wrote to save it.
How many of your own tasks is enough?
Use work you have permission to test. Strip account numbers, customer names, and anything else you are not allowed to send to a third-party endpoint. The half-hour sheet already asks for three instances. Keep those three as a smoke test if they are representative. Do not stop there and call the job finished, unless the only decision you needed was “does this id fail in an obvious way on the first things we tried.”
A planned sample has a shape, not just a count. Include ordinary cases, ambiguous inputs, and known failure cases, in proportions you write down before the run. If you add hard cases only after your preferred id has a comfortable lead, the lead is no longer a measurement of the job. If you drop a case because every id failed it, you have changed the job. A case that is too hard for every candidate still belongs in the record. It tells you the acceptance rule may be wrong, or that none of the ids is ready.
Keep a holdout set you do not use to tune prompts. The development cases — the ones you looked at while writing the rubric or adjusting the system message — are useful, and they are contaminated. Score the holdout once, at the end, with the frozen setup. If the holdout reverses the development ranking, the development ranking is the one you do not ship. You may not have a large holdout. Write the size you actually had. A holdout of zero is a risk you record, not a hidden pass.
“Enough” is not a number this page will invent. Enough is when the person who will live with a wrong pick can say that another batch of similar cases is unlikely to flip the decision, or that the remaining uncertainty is cheaper than a further delay. If one noisy case would change the order, you do not have enough. Repeated generations on the same prompt are not independent new tasks. They can tell you a single prompt is unstable. They do not grow the sample. Stratify from the real workload — a week’s tickets, a set of closed bugs, a folder of documents you are allowed to use — rather than inventing a neat dozen that all look like a demo.
How do you keep identities and training data from leaking into the score?
Where the score depends on a person’s judgment, hide the model identities from the reviewer. Shuffle the order. Strip vendor names, model ids, and header fingerprints from the text you hand over. A reviewer who already prefers a vendor will still have instincts; blinding removes the easy version of that bias, not every version. If the stakes are high, use a second reviewer, and write in advance what you will do when they disagree. Do not let the model grade its own work as the official score. A self-grade can be a note. It is not a pass.
Leakage is the other way a score becomes fiction. Prompts copied from a public benchmark, a vendor blog, or a social post may already be familiar to the model. They are a weak test of your job, and they are a contaminated test of “our work.” Confidential cases you have permission to use, that have not been published, are a better exam if the endpoint’s data-handling terms allow them. If you cannot send those cases, you cannot run this bake-off on them. Use weaker, permitted cases and write that limitation down. Do not paste a secret into a consumer chatbot to get a cleaner sample.
Prompt-tuning leakage is quieter and more common. If the team spent an afternoon improving the prompt on ten tickets, those ten tickets are a training split. They can stay in a development log. They should not be the only evidence in the decision record. The holdout is what you quote. If you have no holdout, say that the figure is a development figure.
Do not give one id extra context that the others are denied, then call the comparison blinded. A longer context window is a system difference. If you use it, every candidate that is allowed that window gets the same documents. If one id cannot accept the packet, that is a constraint result, not a quality insult. Record it as a constraint.
How should failures be recorded, and what does the eval itself cost?
Failed, refused, and timed-out attempts stay in the denominator. A timeout is a result. A refusal is a result. A patch that turns the tests green and also deletes a guard you needed is a failure, which is why code review is not only the test command. For grounded answers, check each factual claim against the supplied material and record unsupported claims, even when the prose is fluent. For each fail, write a reason short enough to group later: wrong fact, missed instruction, unsafe action, empty reply, tool error, budget exhausted, wall-clock exceeded. A bare “fail” column will not help the person who has to design a fallback.
Retries are not free evidence. If the policy allows one retry, every id gets that policy, and both attempts stay in the log. The accepted result, if any, is the one the policy would ship. Counting only the lucky retry is how a fragile id looks stable. The same is true of “best of n” sampling. If you will not pay for n completions in production, do not score the best of n in the bake-off and then forget to write that down.
The evaluation has a cost of its own, distinct from a production unit cost and distinct from a public list price you might write beside a quality field. The eval cost is the API spend for the bake-off, including thinking tokens and retries, plus the reviewer hours, plus the time to write the rubric and to inspect failures. Budget it before the first call. A thinking configuration on a long transcript can make a modest sample expensive. If you shrink the sample to save that money, you may be buying a false winner. If you cannot afford a fair sample on the configuration you will actually call, say so. Run the smoke test, report that the full bake-off is blocked on budget, and do not crown an id.
Record eval spend separately from any later estimate of production cost per accepted result. The two figures answer different questions. Mixing them makes a cheap eval look like a cheap product, or an expensive eval look like an expensive product. This page will not produce a currency figure. It will not divide a pass count by a dollar amount and call the ratio ours.
| Cell | What belongs there |
|---|---|
| Frozen setup | Full id, date, prompt, context, tools, sampling, thinking budget, retries, timeout, and endpoint. |
| Planned sample | Ordinary, ambiguous, and known-hard cases, with proportions written before the run. |
| Holdout | Cases not used to tune the prompt, scored once at the end. A holdout of zero is a recorded risk. |
| Outcome | Pass or fail with a reason, including refusals and timeouts, left in the denominator. |
| Eval cost | API spend for this run and reviewer hours, kept apart from any production unit cost. |
| Decision | Adopt, reject for a hard rule, or inconclusive, with a fallback for the failures you saw. |
When do you stop with an inconclusive result?
You are not owed a winner. The honest outputs of a bake-off are: adopt this id for this job, reject this id for a hard rule, or inconclusive. Inconclusive is the right output when the remaining ids pass and fail almost the same cases, when a single noisy case would flip the order, when the holdout contradicts the development set, or when the eval budget is exhausted before the sample can separate them. Write the uncertainty in that language. Collect more representative cases if the decision is worth the further cost. Do not break the tie by averaging in a public rating, a LiveBench category, or a specialist percent. Those cells belong to a different exam. Leave them unaveraged.
Reject a candidate that fails a required privacy or correctness rule even if its average on the rest of the sample looks better. Averages do not outvote a hard stop. If every remaining id fails the hard rule, the bake-off is empty, and empty is information. It means the shortlist and the job do not yet intersect. Do not borrow a nearby model name to avoid leaving with zero.
Stop, also, when the setup has moved: a new model version, a new prompt, a new tool, a new thinking budget, or a new workload. The previous record is still the record of that setup. It is not a license to keep the old pass rate. Recheck. Keep a fallback for the failure types you already observed, including the id you did not pick, because an inconclusive or a narrow win is not a reason to have no plan for the cases that failed.
When the sheet has a decision a later reader can audit — the frozen setup, the planned sample, the holdout, the failures and their reasons, the eval cost as you measured it, and either a choice or a written inconclusive — stop. Do not add cases until a favored id wins. Do not open a fourth model to escape an inconclusive pair. A two-id inconclusive, dated, with failures attached, is a cleaner handoff than a late winner that only exists because the exam moved.
Questions about a private bake-off
Is a three-case check from the half-hour sheet a bake-off?
It is a starting set. Three cases can expose an obvious failure. They do not establish a success rate, and they are not a Model VS result.
Can a public leaderboard score break a tie in a private bake-off?
No. The bake-off is scored on your cases with a frozen setup. A public rating or task score belongs to a different exam. Leave the cells unaveraged.
What counts as a failure you have to keep?
A failed, refused, or timed-out attempt stays in the denominator. Dropping it makes the pass rate look better than the run you actually had.
When is an inconclusive bake-off the right output?
When the remaining ids look similar on a sample that is too small for the cost of being wrong, or when the eval budget is exhausted before the sample can separate them. Record the uncertainty and keep a fallback. You are not owed a winner.