Builder extraction check, 5 October 2026
Public counts for the Builder release checked on this date. Evidence Table Builder is aligned with RAISE guidance.
Run
Date: 5 October 2026. App version 0.1.0. Git SHA 47612c2 (47612c271834be9822825865ddd11cbfd65788f2).
N = 180 field answers (6 papers × 10 fields × 3 repeats). Run date 5 October 2026. Commit 47612c2.
Dataset
6 locked papers, mostly orthopaedic, from a 27-paper corpus, with 10 fields on each paper and 3 repeats. N = 180 field answers. The reference answers are an internal draft. The developer ran the check. It has not been independently replicated or peer-reviewed. The percentages describe this dataset. They do not come from a non-inferiority test. Quality-assessment assistance, screening, and the study-design gate were not scored.
The Builder corpus has 27 papers (270 labelled fields). 6 papers (60 fields) have locked reference answers, prepared by the developer, frozen for scoring and not independently reviewed, and were scored. 21 papers (210 fields) have candidate labels awaiting independent review and were not scored.
The same 27 papers also have an expanded list of 60 questions (1,620 fields). Scored fields on that list in the published record: 0.
The four six-paper runs scored 52 to 55 of 60; the two four-paper runs scored 34 and 32 of 40. Saved answers from 21 September 2026 were re-graded on 4 October 2026 with the current scorer (version 2, the same scorer as the 5 October check). No new answers were generated. Scores under the earlier scorer are not published. The rows used different extraction settings (question parallelism 1–3, 4 or 8 requests in flight, sequential or shared-file processing), so they are not repeats of one configuration. Record: the 4 October 2026 corpus and saved-run report.
Counts
| Result | Count | Share |
|---|---|---|
| Reference agreement | 167 of 180 field answers | 92.8% |
| Repeats | 54 of 60, 56 of 60, and 57 of 60 | 90.0% to 95.0% |
| Hallucination where the reference answer was absent | 1 of 15 | 6.7% |
| Absent-reference hallucinations by repeat | 0 of 5, 0 of 5, and 1 of 5 | 0% to 20% |
| False not reported | 0 of 180 | 0% |
| Missing answers | 0 of 180 | 0% |
| Placeholders | 0 of 180 | 0% |
Counts by field
| Field | Correct |
|---|---|
| Comparator | 18 of 18 |
| Condition or disease | 18 of 18 |
| Follow-up period | 16 of 18 |
| Funding source | 17 of 18 |
| Intervention or exposure | 16 of 18 |
| Key finding | 11 of 18 |
| Population | 17 of 18 |
| Primary outcome | 18 of 18 |
| Sample size | 18 of 18 |
| Study design | 18 of 18 |
Agreement by study design
| Study design | Papers | Correct |
|---|---|---|
| Parallel-group trials | 2 | 59 of 60 |
| Observational studies | 2 | 54 of 60 |
| Quasi-experimental study | 1 | 29 of 30 |
| Unclear study design | 1 | 25 of 30 |
Agreement by text layer
| PDF text | Papers | Correct |
|---|---|---|
| Papers with a text layer | 5 | 142 of 150 |
| Scanned paper | 1 | 25 of 30 |
Agreement by difficulty
| Difficulty | Papers | Correct |
|---|---|---|
| Easy | 1 | 29 of 30 |
| Medium | 4 | 113 of 120 |
| Hard | 1 | 25 of 30 |
Agreement by paper
Each locked paper has 30 field answers (10 fields × 3 repeats). Correct answers on the six papers: 30, 29, 29, 29, 25, 25. N = 180 field answers.
The unclear-design, scanned and hard rows are the same one paper.
Assistant configuration used before the current release
The same run scored the assistant configuration used before the current release on the same 6 papers, 10 fields and 3 repeats (N = 180 field answers), with the same reference answers and scorer. 160 of 180 (88.9%). Grounded citations that named a different page: 35 of 139, against 55 of 120 for the current release. Repeats were 53 of 60, 54 of 60, and 53 of 60. Hallucinations where the reference answer was absent: 3 of 15 (20.0%); by repeat, 2 of 5, 0 of 5, and 1 of 5. False not reported, missing answers, and placeholders were each 0 of 180. Date 5 October 2026. Commit 47612c2.
Cited pages
55 of 120 grounded citations named a page that differed from the PDF page. N = 120 grounded citations. This count is separate from the 167 of 180 answer-agreement count.
Scorer calibration (4 October 2026)
N = 43 adversarial examples. 9 of 43 were false passes. 34 of 43 matched the intended verdict (79.07%). This calibration is not extraction accuracy, and it is a different check from the 5 October 2026 run above.