InsightsHalf-hour sheet

Three names, then stop.

In thirty minutes, write one acceptance sentence, drop ids that fail a hard constraint, and leave with at most three candidates.

ClockThen stop
05Acceptance sentence and hard stops
15One preference pass, one category
25Specialist row only if the job is that exam
30Three names and three cases

By Model VS · Guide text updated . The clock below is a procedure for a small team. The filled column in the sheet is an example about a fictional product. It is not a recommendation and it contains no benchmark result. Blank cells can be typed in the browser; nothing is saved or sent. Write to us if a file link on this page 404s.

What do you freeze in the first five minutes?

Set a timer. The point of the timer is to stop a shortlist from becoming a literature review. In the first five minutes, two people, or one person talking out loud, write a single acceptance sentence and a list of hard stops. The acceptance sentence names the output and the failure. “Draft a reply to a billing question using only the help-center article in the prompt. An answer that states a refund amount not present in that article is unacceptable.” That sentence is specific enough to choose a LiveBench category later. “We need the best model” is not a sentence this procedure can use. If you cannot write the failure mode, you are not ready to open a leaderboard. Spend the five minutes on the sentence instead.

Hard stops are constraints that end a candidacy without a score. Examples that belong in this class:

  • The data must stay in a region your vendor actually offers
  • The license must allow your use
  • The context must hold the transcript you already have
  • The API must expose the tool you already call
  • The price ceiling is a number finance already signed

Write them as yes or no tests. A vibe such as “not too expensive” is not a hard stop. A number is. If you do not know the number, the shortlist’s output is “blocked on a price ceiling,” which is a legitimate way to end the half hour. Methodology is explicit that public measurements are not a substitute for those constraints.

Also freeze the configuration words you are willing to pay for. If the team will not run a maximum thinking budget on every ticket, say so now. Source ids often encode that budget with a suffix. An id you cannot afford is not a candidate, even if a public table likes it. Write the allowed suffixes next to the acceptance sentence. You will copy full ids later, and this note stops you from dropping the suffix because it is inconvenient to type.

Where do the public numbers come from?

Open three files at most, and only the passes the acceptance sentence requires. The table lists the timestamps stored on the current snapshots so the sheet can cite a date without inventing one. Copy the timestamp you mean. “Arena publish date” and “file saved” are different clocks. LiveBench has a release string plus a save time. The specialist file has a save time and, for SWE-bench Verified, a dataset timestamp. Quote the one you opened.

Files this half hour is allowed to open. Timestamps are the ones stored on the snapshots. They are not a claim that the upstream site stood still after the save.
PassFileTimestamp on this fileCopy onto the sheetLeave off the sheet
Preference pass/data/arena-latest.jsonLeaderboard publish date 2026-10-02; saved October 6, 2026 (2026-10-06T09:00:53.013Z).Full source id, raw rank, published interval, vote countA winner declared because the rank numeral is smaller
Task pass/data/livebench-latest.jsonRelease 2026-06-25; saved October 6, 2026 (2026-10-06T09:00:12.143Z).Full source id and the one category named by the acceptance sentenceOverall, unless the acceptance sentence is itself the unweighted mean
Submission pass/data/specialists-latest.jsonSaved October 6, 2026 (2026-10-06T09:00:11.687Z); SWE-bench Verified dataset timestamp August 16, 2026 (2026-08-16T04:23:43.000Z).Model id, metric, verification flag, and the card link — and only when the job is that benchmarkA percent copied without the flag or the link

The preference pass is the Arena overall table at /#leaderboard. Copy the full source id, the raw rank, the published interval, and the vote count for any id that survives the hard stops. If two survivors have overlapping intervals, keep both. The interval essay is the reading rule: overlap is a reason to refuse a one-name conclusion. Do not spend this pass hunting a two-point lead. You are collecting candidates, not delivering a verdict.

The task pass is one LiveBench category. Choose it from the acceptance sentence, using the category guide. A billing reply that must follow a pasted article is an instruction-following problem before it is a coding problem. Overall is the wrong column unless your sentence truly is “the unweighted mean of every category.” Copy the full id and that category’s average. Leave the other categories in the file. They will tempt you to widen the meeting, and the timer is there to prevent that.

The submission pass stays closed unless the acceptance sentence is about the benchmark itself. A support bot is not a SWE-bench run. A coding agent that must land a patch might be. If you open the specialist table, copy the model id, the metric, the verification flag, and the card link, and read them with the card essay. A percent without the flag is an incomplete cell. Leave it incomplete rather than filling it from memory.

If the preference pass and the task pass order the same ids differently, stop and label the conflict. Do not average the columns. The disagreement essay shows how this site prints an exact-id reversal, and it explains why a family nickname is not a pair. On your sheet, two rows are enough: the id you are keeping for the vote, and the id you are keeping for the category. They may be the same string. They may not.

When do you close the leaderboard?

Close it when you have three candidates or when the timer hits twenty-five minutes, whichever comes first. A candidate is a full source id that survived the hard stops and has a reason tied to the acceptance sentence. “Ranked well” is not a reason. “Interval overlaps the neighbor, and the instruction-following category is the one our sentence needs” is a reason. “Removed because the license fails the hard stop” is also a reason, and rejected ids belong on the sheet so the next meeting does not revive them.

The last five minutes are for the three cases, not for a fourth model. Pick three instances from work you are allowed to use. For the fictional support bot in the example column, those instances are old billing tickets with the account numbers removed. For your team, they are whatever the acceptance sentence describes. Write the prompt you will freeze, the suffix you will call, and where the fail log will go. How we rate is the protocol for that check: same setup, failures kept, inconclusive allowed. You will not finish the check inside the thirty minutes unless the cases are already in your clipboard. Finishing the list of cases is the goal. Running them is the next block of work, on a different clock.

Stop early if a hard stop empties the list. An empty shortlist is information. It means the constraint and the public tables do not intersect, or you have not been given a price ceiling, or the id you need is not in the file. Record which of those it is. Do not borrow a nearby model name to avoid leaving with zero. Zero is a cleaner input to the next meeting than a name nobody can defend.

Also stop if you notice the meeting has started comparing products, retention tools, or agent frameworks rather than model ids. Those can be the right meeting. They are not this half hour. The rankings on this site do not price a seat or review a vendor’s security questionnaire. Put that work on a different agenda so the model sheet stays about the sentence you wrote at minute zero.

What leaves the room on the handoff sheet?

The sheet has one job: a person who missed the meeting can see the sentence, the rejects, the candidates, and the three cases without reopening the leaderboard. Copy the table. The example column is fiction about an internal support bot that does not exist. It shows the grain of a filled cell. It does not name a model you should buy, and it does not report a score. The last column starts blank. You can type in it. The page does not store what you type. Paste the result into your own document before you close the tab.

Example column: a fictional internal support bot, invented to show the sheet. Not a recommendation, not a ranking, and not a result from any Model VS snapshot. Your notes are local to this tab.
Blank to fillWhat belongs thereExample — fictional support botYour notes
Acceptance sentence The output, and what makes it unacceptable. Example. Draft a reply to a billing question using only the help-center article pasted into the prompt. A refund amount that is not in the article is unacceptable.
Hard stops Yes or no constraints: region, license, context, tools, price. Example. Must use the vendor’s US endpoint. Must accept a 30 thousand token transcript. Must stay inside the team’s current chat budget. No other region is in scope.
Allowed configuration The suffixes or effort settings you will actually call. Example. Standard chat setting only. A maximum thinking budget is out of budget for this bot, so those ids are out.
Preference pass Full Arena ids, intervals, votes, and the snapshot date you copied. Example. Copy two ids whose intervals overlap. Keep both. Date comes from the file table above, not from memory.
Task pass One LiveBench category, the full ids, the release, and the save time. Example. The sentence is about following a pasted article, so the category is IF, which the rankings table labels Instruction following. Overall stays closed.
Submission pass Specialist id, metric, flag, and link — or an explicit “not this job.” Example. Not this job. The bot does not resolve repository issues. The cell stays closed on purpose.
Candidates At most three full ids, each with a reason tied to the sentence. Example. One id kept for the preference overlap. One id kept for the instruction category. A third slot left empty rather than filled with a nickname.
Rejected Ids removed, and the hard stop or mismatch that removed them. Example. Any id whose suffix is a maximum thinking budget. Any id whose license the team cannot accept. Reasons are constraints, not tastes.
Next three cases Real items you may use, the frozen prompt, and where failures will be kept. Example. Three redacted billing tickets from last month. Same prompt. Failures stay in the denominator. The run happens after this meeting.

Hand the sheet to the person who will run the cases. They should be able to call the endpoint without inventing a suffix, and they should know which result would kill a candidate. If the sheet does not say that, the half hour is not finished. Add the missing line. Do not add a fourth candidate to make the page look complete.

A week later, the files may have moved. The candidates remain the ids you wrote, tied to the timestamps you copied. If you refresh the shortlist, start a new sheet. Do not edit the old reasons to match a new rank. The October changelog is the demonstration that a list moves because the file changed. Your sheet should be just as explicit about its own date.

Questions about the half hour

Can a small team finish model selection in thirty minutes?

You can finish a shortlist: one acceptance sentence, the ids removed for a hard constraint, at most three candidates, and the three cases you will run next. A company-wide evaluation does not fit in that window.

Which Model VS surfaces should the half hour open?

Open the Arena table, the one LiveBench category named by the acceptance sentence, and the specialist table only when the job is that benchmark. Then use How we rate for the three-case check.

Is the filled column on this page a recommendation?

No. That column is an example about a fictional internal support bot. It shows the shape of the sheet. It is not a ranking and it does not name a model you should buy.

What if Arena and LiveBench disagree during the session?

Write both ids on the sheet and leave the cells unaveraged. The disagreement essay shows exact-id reversals from the current files. The three cases you listed are what decide this job.