InsightsSubmission evidence

SWE-bench “Verified” is a subset. The flag is a separate fact.

A benchmark name can describe how the tasks were filtered. A row on our specialist table still needs its own submission flag, its own scaffold, and its own date.

A field of teal tiles with a brighter cluster and a separate gold flag on one isolated item.
Verified names a subset of tasks. A submission flag is not the same claim.
Flag readingTwo meanings
01Name the subset
02Read the submission flag
03Open the linked card

By Model VS · Guide text updated . Counts and the first rows are filled from the saved specialist snapshot at build time. Model VS does not re-run SWE-bench. A linked model card is the evidence the source row points at, not a reproduction by us. Report a correction if a flag or link does not match the file.

The subset

What does the Verified subset describe?

SWE-bench Verified is a human-filtered subset of 500 instances from SWE-bench, created in collaboration with OpenAI. Human annotators reviewed each instance to ensure the problem descriptions are clear, the test patches are correct, and the tasks are solvable given the available information.

SWE-bench project, SWE-bench Verified. Excerpt used to identify the subset. The page also separates a “bash only” model comparison from a leaderboard of fuller systems.

That is a statement about the questions, not about a vendor’s press release. The subset was filtered so the issue text and the tests were considered usable by human reviewers. OpenAI’s introduction of the subset describes the same idea: annotators screened samples for underspecified issues and tests that reject valid fixes, and the released slice has 500 samples. Read that note for the annotation scale and the limits they already named. Static datasets go stale. OpenAI has since argued, in a later note, that contamination and remaining test flaws are reasons they stopped treating this subset as a frontier score. Whether you agree with that recommendation or not, it is a reason to cite the date and the harness, not a naked percentage.

Other slices exist. SWE-bench Lite is a different, smaller set. A full-system leaderboard and a bash-only leaderboard are different comparisons even when both say Verified. If two blog posts quote “SWE-bench” without the subset and the harness, they may not be comparable. Our table is labeled SWE-bench Verified because that is the benchmark id in the file. The label does not upgrade every row into a checked submission.

The flag

What does the submission flag add?

Each specialist row carries a boolean verified field from the leaderboard record we sync, plus a link that is supposed to be the submission evidence. How we rate already says a benchmark name is not a submission certificate. This page is the worked case. “Verified” in the title means the task subset. “Source marks this submission verified” would mean the row’s own flag is true. “Community submission — not marked verified” means the flag is false. Those sentences are not synonyms.

A false flag does not mean the number is fabricated, and a true flag would not mean Model VS reproduced the run. It means the upstream record did or did not mark the submission as verified. You still have to open the link and look for the model id, the scaffold, the test split, the attempt count, and the inference budget. If the link is a generic model card with a single percentage and no harness, write “unknown” next to those fields. Unknown is a result. It is how you avoid promoting a screenshot.

The file also has a dataset timestamp that can be older than the day we copied the leaderboard. Both times are printed in the generated section. Quote the one you mean. “As of our specialist sync” and “as of the dataset card” can be different days. Neither is “this morning” unless the timestamp says so.

Saved rows

Why count flags before quoting the top row?

The build reports how many saved rows are marked verified, how many are not, and how many have no flag. It then prints the first five rows in file order so you can see the flag sitting next to a high reported score. A high score with an unverified flag is still a high reported score. It is not a certificate. Do not drop the flag when you move the number into a slide.

Snapshot saved October 6, 2026 (2026-10-06T09:00:11.687Z). SWE-bench Verified dataset timestamp on that file: August 16, 2026.

  • Rows in the saved file: 67
  • Marked verified: 0
  • Not marked verified: 67
  • Verification field absent: 0
First five rows in the saved SWE-bench Verified list, in source order. The benchmark name contains “Verified”; the submission flag is the separate column.
RankModel idReported scoreSubmission flag in our snapshotLinked evidence
1ornith-ai/Ornith-1.5-397B86.00 % resolvedCommunity submission — not marked verifiedOrnith-1.5-397B model card
2mindlab-research/Macaron-V1-Venti85.60 % resolvedCommunity submission — not marked verifiedModel Card
3mindlab-research/Macaron-V1-Coding-Venti85.60 % resolvedCommunity submission — not marked verifiedModel Card
4ornith-ai/Ornith-1.0-397B82.40 % resolvedCommunity submission — not marked verifiedOrnith-1.0-397B model card
5deepseek-ai/DeepSeek-V4-Pro80.60 % resolvedCommunity submission — not marked verifiedModel Card

The same specialist file's GPQA table, dataset timestamp September 21, 2026, has 100 rows: 0 marked verified, 100 not marked verified, 0 with no flag. The flag question is not unique to the coding benchmark.

The same file’s GPQA table is included in that count when it exists, because the habit is not limited to coding. A reasoning accuracy with an unverified flag has the same citation problem. GPQA and SWE-bench still must not be added together. They do not share a unit.

System, not a bare model

Does the reported score include a scaffold?

SWE-bench scores are produced by a system that edits a repository and runs tests. That system can include a scaffold, a retry loop, a test viewer, retrieval, or a specific agent harness. The SWE-bench site itself distinguishes a minimal bash setup from arbitrary systems. A percentage copied from a full-system board is not the percentage you will get from a chat box with no tools, and it is not the percentage you will get from your IDE unless your IDE is that system.

Ask five questions before the number enters a procurement note:

A small glowing core inside a large architectural scaffold that dominates the frame.
The scaffold around the model is part of the system being measured, not a detail you can ignore.
  1. Which subset: Verified, Lite, or another split, and how many instances does that page claim?
  2. Which harness: bash-only, a named agent, or an unnamed internal runner?
  3. Which model id, including size and reasoning setting, not the product family?
  4. Is the submission flag true, false, or missing in the snapshot you are citing, and what does the linked page actually show?
  5. What is the date of the source page and the date of our file?

If any answer is missing, the honest shortlist line is “reported percent, harness unknown,” not “best coding model.” Our specialist view says the same thing in shorter form: agent and coding results may include a scaffold. This essay is the longer version of that sentence.

A different coding exam

Why will LiveBench coding not confirm this table?

LiveBench’s Coding category, on the release in our other file, is an average of its own task columns. It does not resolve GitHub issues in the SWE-bench repositories, and it does not use the Verified subset. A model can sit high on one and off the saved list of the other. That is not a sync error. It is two exams. Use the category guide when you want those columns, and use this page when you want the submission list. A field-by-field reading of the saved row and its card link is in Reading specialist and HF cards.

Arena preference is a third exam. People voting on chat replies are not grading patches. Do not break a tie on SWE-bench by looking at who has the friendlier Arena rank. If you need both a conversational assistant and a coding agent, you may end up with two systems. That is allowed. The decision protocol tells you to label a comparison of configured systems as a comparison of systems.

Citation

What has to sit beside the percentage?

A citation that survives a week later has the model id, the subset, the harness if known, the reported metric, the submission flag, the link, and both timestamps. Example shape, with the numbers left for the table above to supply: “On the Model VS specialist snapshot saved on the printed sync time, the first SWE-bench Verified row reported the printed percent resolved. The submission flag in that file was the printed flag. Evidence link: the printed URL. Dataset timestamp: the printed dataset time. Model VS did not reproduce the run.”

Then run a tiny version of the job on a repository you are allowed to touch. One issue with a real test is more informative than moving a public percentage into a roadmap. Keep the patch, the command, and the fail log. If the public row’s harness is not your harness, say that the pilot is a new experiment. The public number remains context. It does not become your pass rate.

What we will not add

We will not mark a row verified because the benchmark title contains the word. The flag in the file is the flag we show. If the upstream record changes, the next validated snapshot can change the count. Until that file changes, the count on this page is the count in the file.

FAQ

Questions about SWE-bench flags

What does Verified mean on this page?

SWE-bench Verified is a human-filtered subset of tasks. Each specialist row also carries a boolean verified field from the leaderboard record we sync. "Verified" in the title means the task subset. "Source marks this submission verified" would mean the row's own flag is true. Those sentences are not synonyms. A benchmark name is not a submission certificate.

What does a false verification flag mean?

A false flag does not mean the number is fabricated, and a true flag would not mean Model VS reproduced the run. It means the upstream record did not mark the submission as verified. A high score with an unverified flag is still a high reported score. It is not a certificate.

Does a specialist result endorse the base model alone?

No. SWE-bench scores are produced by a system that edits a repository and runs tests. That system can include a scaffold, a retry loop, a test viewer, retrieval, or a specific agent harness. A percentage copied from a full-system board is not the percentage you will get from a chat box with no tools. If the harness is unknown, the honest shortlist line is "reported percent, harness unknown," not "best coding model."

Will this page mark a row verified because the title says Verified?

No. We will not mark a row verified because the benchmark title contains the word. The flag in the file is the flag we show. Model VS does not re-run SWE-bench. A linked model card is the evidence the source row points at, not a reproduction by us.