By Model VS · Guide text updated . Field tables are copied from the saved specialist snapshot at build time. We do not fetch the model card body, and we do not re-run the benchmark. Send a correction if a link or flag does not match the file.
Which fields did the file keep?
The specialist file names its source as the Hugging Face official leaderboard API. It also stores a guide URL. That URL is the Hub’s leaderboard data guide, and it is the right next page if you want Hugging Face’s own account of how leaderboard records are shaped. Model VS is not affiliated with Hugging Face or with the labs that published the cards. Affiliation is not implied by a link, a rank, or a parameter count.
Each benchmark object in the file carries an id, a label, a domain word, a dataset id, a source URL, a metric name, a dataset timestamp, and a list of rows. Each row carries a rank, a reported value, a verification flag, a model id, a source object, a lower-is-better flag, and a parameter count. The source object carries a URL, a link name, and an external bit. That is the inventory. There is no field for “Model VS reproduced this,” no field for “we recommend this,” no Arena rating, and no LiveBench category. If a slide needs one of those, it has to come from somewhere else, with its own date.
The first table is the first SWE-bench Verified row in source order. The second is the first later row in that same list whose card URL does not contain the model id string. If the file has no such row, the build says so and uses the first GPQA row instead. Both tables print every stored field listed above, plus one derived check: whether the card URL contains the full model id. The derived check is ours. It is a string comparison. It is not a claim that we opened the README, hashed the weights, or confirmed the organization that trained the model.
| Field | Stored value |
|---|---|
| Snapshot source | Hugging Face official leaderboard API |
| Snapshot saved | October 6, 2026 (2026-10-06T09:00:11.687Z) |
| Guide URL stored on the file | https://huggingface.co/docs/hub/en/leaderboard-data-guide |
| Benchmark id | swe-bench-verified |
| Benchmark label | SWE-bench Verified |
| Domain label | coding |
| Dataset id | SWE-bench/SWE-bench_Verified |
| Benchmark source URL | https://www.swebench.com/ |
| Metric name | % resolved |
| Dataset timestamp | August 16, 2026 (2026-08-16T04:23:43.000Z) |
| Row rank | 1 |
| Reported value | 86 |
| Verification flag | false — community submission, not marked verified |
| Model id | ornith-ai/Ornith-1.5-397B |
| Card link name | Ornith-1.5-397B model card |
| Card URL | https://huggingface.co/ornith-ai/Ornith-1.5-397B |
| Link marked external | false |
| Lower is better | false |
| Parameter count | 403,397,928,944 (integer stored on the row) |
| Derived check: card URL contains the full model id | Yes |
| Field | Stored value |
|---|---|
| Snapshot source | Hugging Face official leaderboard API |
| Snapshot saved | October 6, 2026 (2026-10-06T09:00:11.687Z) |
| Guide URL stored on the file | https://huggingface.co/docs/hub/en/leaderboard-data-guide |
| Benchmark id | swe-bench-verified |
| Benchmark label | SWE-bench Verified |
| Domain label | coding |
| Dataset id | SWE-bench/SWE-bench_Verified |
| Benchmark source URL | https://www.swebench.com/ |
| Metric name | % resolved |
| Dataset timestamp | August 16, 2026 (2026-08-16T04:23:43.000Z) |
| Row rank | 4 |
| Reported value | 82.4 |
| Verification flag | false — community submission, not marked verified |
| Model id | ornith-ai/Ornith-1.0-397B |
| Card link name | Ornith-1.0-397B model card |
| Card URL | https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B |
| Link marked external | true |
| Lower is better | false |
| Parameter count | 396,802,360,816 (integer stored on the row) |
| Derived check: card URL contains the full model id | No |
Read the two timestamps before the percentage. “Snapshot saved” is when our file was written. “Dataset timestamp” is the time stored for that benchmark’s dataset record. They can differ by weeks. A citation that says only “as of today” is false unless today’s date is one of those two. The metric name sits next to the value so a GPQA accuracy is not later described as a percent of issues resolved. The lower-is-better flag is on the row even when the current benchmarks treat a higher value as better. Quote the flag with the value. A future benchmark in the same shape could flip the direction, and a reader who memorized “up is good” would be wrong.
The parameter count is the integer on the row, grouped for reading. We do not round it to a marketing size such as “400B.” Rounding would be a new number. If a card’s own headline uses a round size, that headline is the card’s claim. Our integer is the file’s claim. They can disagree by a rounding choice or by a real difference in what was counted. Leave them as two figures until you know which one the vendor means in a contract.
The verification flag is the narrow field already discussed in the SWE-bench flag essay. A benchmark label can contain the word Verified because the task subset was filtered. The row’s own flag is a separate fact about the submission record we synced. A false flag beside a high reported value is a normal state of this file. It means the leaderboard record did not mark the submission verified. It does not mean the value was typed in by Model VS, and a true flag would not mean we reproduced the run. Open the link either way.
Does the card URL have to match the model id?
No. The file allows them to differ, and the second table exists to show a case where they do, when one exists. The model id is the leaderboard’s name for the row. The card URL is the evidence link stored beside it. When the URL’s path does not contain that id, you are looking at two names. Sometimes the link points at a repository under a different organization string. Sometimes the link is a blob URL into a README line rather than the repository root. Sometimes the external bit is true, which in this file means the link was marked external. None of those facts tells you the score is wrong. They tell you that copying only the model id, or only the URL, drops half of the record.
The practical habit is to paste both strings into the note, then open the URL. On the card, look for the same benchmark name, the same split, the same metric, and a harness description. If the card shows a different percent, write both percents and both dates. Do not “fix” our table to match the card, and do not “fix” the card to match our table. Tell us if our file has drifted from the leaderboard record. The card can still be right about a different run.
A matching URL is not a reproduction either. The derived check can say Yes because the id string appears in the link, while the README describes a scaffold, a retry budget, or a tool setup that your product does not have. SWE-bench scores in particular are produced by a system that edits a repository and runs tests. Methodology already says a specialist result may include a scaffold or a test-time strategy. The row does not unpack that scaffold into its own columns. The card might. Until you have read it, the honest label is “reported value, harness not recorded in our file.”
What can a README number still fail to be?
A model card README is a publisher document. It can include a table of scores the leaderboard row does not store. Those scores may be real measurements by the publisher. They are not measurements by Model VS, and they are not automatically the same run as the leaderboard value. Common differences: a different benchmark split, a different sample count, a majority vote over several attempts, a tool-using agent versus a bare model, or a number copied from a paper that predates the repository. The card can also omit the split and still look precise.
Treat every README figure as unpublished, relative to this site, until you have written down the split and the date you saw on that page. “Unpublished here” does not mean “false.” It means our snapshot will not silently adopt it. If you want it in a decision, cite the card URL and the date you opened it, beside our row’s value and our sync time. Two citations are cleaner than one blended number.
| A sentence someone might say | What the stored fields can support |
|---|---|
| This model won. | The row has a rank inside one named benchmark list, on one file, at the printed timestamp. |
| The lab proved the number. | The row has a reported value, a verification flag, and a URL. The flag and the URL still have to be read. |
| Model VS recommends it. | No endorsement field exists. The site is not affiliated with the card’s publisher. |
| The README repeats this percent. | The README is not stored. Only the link is. Open the card. |
| It is also the Arena leader. | Arena is a different file and a different unit. This row has no preference rating. |
That boundary is why specialist rows stay off the Arena list and off the LiveBench category table. A percent resolved and a preference rating do not add. A GPQA accuracy and a LiveBench Reasoning average do not add, even though both are about questions with answers. If you need a preference reading, use the interval essay. If you need a task family on the LiveBench release, use the category essay. If the two orderings conflict for one exact id, use the disagreement essay. None of those pages import the specialist percent to break a tie.
How do you quote a row without endorsing it?
A citation that stays true to this file has eight pieces:
- The benchmark id
- The model id
- The reported value
- The metric name
- The verification flag
- The card URL
- The dataset timestamp
- The snapshot time
You can add the derived URL check if you are explaining a mismatch, and you should label it as a string check. You should not add “best,” “recommended,” or “verified by Model VS.” The rank can be included as “rank on this saved list,” which is a position, not a trophy.
Example shape, with the numbers left for the tables to supply: “On the Model VS specialist snapshot saved at the printed time, the first SWE-bench Verified row reported the printed value on the printed metric. The verification flag was the printed flag. The card link was the printed URL. The dataset timestamp was the printed dataset time. Model VS did not reproduce the run and does not endorse the model.” Fill the blanks from the table. If any blank is missing in the file, say it is missing. A missing harness is a result.
After the citation, decide whether the job is even this benchmark. A chat assistant that must follow a policy is not a SWE-bench submission. Quoting a resolved-rate at that team sends them to the wrong exam. How we rate asks you to define the task before you pick a lane. The specialist lane is appropriate when the acceptance test is close to the benchmark’s own task: a repository issue, a patch, a test suite, or a graduate-level question set in the case of GPQA. Even then, run one instance you are allowed to touch. Keep the command and the fail log. The public row remains context for the harness you did not run.
If the card and the row disagree and you cannot tell which run your vendor will sell, the shortlist line is “unresolved citation,” not a guess. Unresolved is allowed. It is how a small team avoids promoting a screenshot. The next essay, the thirty-minute shortlist, tells you to leave that cell blank rather than invent a bridge between the card and the row.
Questions about cards and flags
Does a specialist row mean Model VS recommends the model?
No. The row stores a reported metric, a model id, a verification flag, a parameter count, and a link. Model VS does not reproduce the run and is not affiliated with the publisher of the card.
Is the README on the linked card part of our snapshot?
Only the link is. We do not copy README tables into this page. A README can describe a different split, harness, or date from the leaderboard row. Open it before you treat those figures as the same claim.
What does a false verification flag mean?
The saved leaderboard record did not mark that submission verified. It does not mean the reported number was fabricated, and it does not cancel the benchmark name. The subset name and the submission flag are different fields.
Why are these rows kept off the Arena list?
The units differ. A percent resolved or a GPQA accuracy does not add to a preference rating. Methodology keeps specialist results on their own surface so the scales stay separate.