Resolving Dual-Extraction Disagreements: A Field-Level Adjudication Ladder

Goal

Provide a fast, defensible ladder to resolve cell-by-cell disagreements between two human extractors, while preserving a source-linked audit trail. Keep it mechanical: classify, decide, record — no silent overwrites.

Ladder overview

1. Cell mismatch detected → 2) Classify the disagreement → 3) Apply the decision rule → 4) Record resolution in the audit trail (final value + both source pointers).

1) Classify the mismatch

• Transcription/typo
  • One value reproduces the source; the other is a slip (digit transposed, unit copied incorrectly, punctuation stripped).
  • Examples:
    • Sample size: A=412, B=410; PDF table shows n=410.
    • Dose: A=10 mg, B=10 µg; PDF: 10 mg.
• Interpretation
  • Both values exist in the paper but reflect different choices (population set, timepoint, unit convention, rounding, per-protocol vs ITT).
  • Examples:
    • Follow-up: A=12 weeks vs B=90 days; both appear.
    • N: A=410 ITT vs B=392 PP; both reported.
• Missing in source / Not reported (NR)
  • At least one extractor entered a value that is not demonstrably present in the paper.
  • Examples:
    • Compliance rate present? A=78%, B=NR; no rate anywhere in text/tables.

2) Decision rules

• If Transcription/typo
  • Keep the value that matches the source verbatim.
  • If scan quality or OCR is ambiguous, escalate to a third extractor.
• If Interpretation
  • Apply the protocol rule first (e.g., prefer ITT over PP; prespecified primary timepoint; SI units; rounding to source precision).
  • If protocol is silent and both are defensible, escalate to a third extractor; otherwise mark “uncertain” with rationale.
• If Not reported
  • Record “NR” (or the project’s standard null token) and include a short search note (sections/pages checked). Do not impute.

3) Audit trail — how to record the resolution

• Final value: the resolved cell value (including “NR” when applicable).
• Evidence pointers: keep both extractors’ original source pointers (quote + page/figure/table). If one was wrong, still keep its pointer for traceability.
• Rationale: 1–2 words are enough (“typo—kept B”, “protocol: ITT”, “NR after search”).
• Callout: do not overwrite silently. Adjudication must leave a row-level footprint.

4) Escalation triggers

• Low-quality scan, conflicting tables, or OCR artifacts.
• Protocol conflict (e.g., competing unit conventions).
• Safety-critical fields (primary outcomes, N, mortality events).
• Repeated disagreements by the same extractor (training/feedback needed).

Worked examples

• Sample size (Transcription/typo)
  • A=412, B=410; Table 1 shows n=410. Decision: keep 410. Audit: final=410; pointers=[A: none/incorrect, B: Table 1 p.3]; rationale=“typo—kept B”.
• Timepoint (Interpretation)
  • A=12 weeks, B=90 days; Methods specify primary analysis at 12 weeks; Results include a 90-day sensitivity. Decision: keep 12 weeks per protocol. Audit: final=12 weeks; pointers=[A: Methods p.5, B: Results p.9]; rationale=“protocol: primary timepoint”.
• Adverse events rate (Not reported)
  • A=18%, B=NR; no AE rate in text/tables; only raw counts without denominator. Decision: NR. Audit: final=NR; pointers=[A: claimed value—no source, B: —]; rationale=“NR after search (pp.3–7)”.

Related

If you’re trialing automation alongside humans, see also: Automated Data Extraction for Systematic Reviews (HEOR & Market Access). This ladder is strictly for human–human adjudication and auditability.