Corpus and saved runs, 4 October 2026
Public counts for the Builder corpus and for saved answers re-graded on this date. Evidence Table Builder is aligned with RAISE guidance.
Report
Re-grade date: 4 October 2026. Saved answers: 21 September 2026. App version 0.1.0 is the version in the repository at summary commit 34303a0 (34303a04149d1949cc42fb61e40b66cead01d8f0). The saved runs do not record a git SHA or an app version of their own.
These rows are earlier configurations of the previous assistant. Saved answers from 21 September 2026 were re-graded on 4 October 2026 with the current scorer (version 2, the same scorer as the 5 October check). No new answers were generated. Scores under the earlier scorer are not published. The rows used different extraction settings (question parallelism 1–3, 4 or 8 requests in flight, sequential or shared-file processing), so they are not repeats of one configuration. The current release is the 5 October 2026 eval report.
Corpus
The Builder corpus has 27 papers (270 labelled fields). 6 papers (60 fields) have locked reference answers, prepared by the developer, frozen for scoring and not independently reviewed, and were scored. 21 papers (210 fields) have candidate labels awaiting independent review and were not scored.
The same 27 papers also have an expanded list of 60 questions (1,620 fields). Scored fields on that list in the published record: 0.
13 fields are human-only and have no forced answer. The summary records 257 fields that could be scored after the draft papers are reviewed. Those fields have not been scored.
Saved runs
The four six-paper runs scored 52 to 55 of 60; the two four-paper runs scored 34 and 32 of 40.
| Saved run | Papers | Correct | Share |
|---|---|---|---|
| Saved run 1 | 6 | 52 of 60 | 86.7% |
| Saved run 2 | 6 | 53 of 60 | 88.3% |
| Saved run 3 | 6 | 55 of 60 | 91.7% |
| Saved run 4 | 6 | 53 of 60 | 88.3% |
| Saved run 5 | 4 | 34 of 40 | 85.0% |
| Saved run 6 | 4 | 32 of 40 | 80.0% |
Across the six runs, 279 of 320 field answers matched the reference. The runs use different paper counts, so that total is not one accuracy figure.
Limits
- These rows are earlier configurations of the previous assistant. Saved answers from 21 September 2026 were re-graded on 4 October 2026 with the current scorer (version 2, the same scorer as the 5 October check). No new answers were generated. Scores under the earlier scorer are not published. The rows used different extraction settings (question parallelism 1–3, 4 or 8 requests in flight, sequential or shared-file processing), so they are not repeats of one configuration.
- Each row is one saved run. Paper counts are not all the same.
- The saved runs do not record provenance for the original extraction.
- Candidate labels were not scored. Quality assessment and the study-design gate were not scored.
- The counts describe these saved runs. They do not come from a non-inferiority test.