- original paired test worlds
- 32
- newly trained paired worlds
- 16
- verified individual readouts in three follow-up studies
- 435,456
What did the inventory say, and what can be entered?
An inventory records Pihene as Zubiru’s storage bay. An audit later rules out that assignment. Asked what the inventory reported, the answer remains Pihene. Asked what to enter in a verified catalogue, the value must remain unspecified. Scoring just the first answer would miss whether the system respected the audit.

Figure 1 traces one inventory fact through the three questions and the two replay policies.
Change the attachment while keeping the questions paired
Both specialists start from the same mixed-practice checkpoint and receive 40 further updates on the same questions and answers. A rehearses closed book; B retains documents. At test time an attachment can be an unrelated synthetic verification record or a length-matched WikiText excerpt. Joint accuracy requires three correct readouts for one fact: the reported value, its verification state and the value permitted by that state.
A third branch continues mixed practice. It matches the number of updates and supervised answers. Its later training mix and token count differ. The paired world is the uncertainty unit. The original specialist comparison uses 32 worlds; a fresh study trains new adapters on 16 new worlds.
The better specialist depends on the test document
On the original 32 worlds, B exceeds A by 19.25 percentage points with synthetic records. With WikiText, the difference is −7.92 points; its paired 95% interval is −12.53 to −3.55. The new-world comparison preserves both directions. On the original worlds, continued mixed practice also beats the stronger specialist on both attachment types.
| Comparison | Worlds | Synthetic | WikiText |
|---|---|---|---|
| B − A | 32 | +19.25 | −7.92 |
| B − A | 16 | +17.16 | −6.89 |
| MIX120 − stronger specialist | 32 | +5.58 | +9.23 |
Keep the donor facts and rewrite their presentation
Rewriting the same synthetic donor facts as prose under explicit verdict cues narrows B’s advantage by 3.20 points. All four combinations of form and review cue still favor B. The larger WikiText reversal therefore cannot be assigned to prose formatting alone from this comparison.
The experiments use one Qwen3-1.7B backbone, synthetic worlds and multiple-choice outputs. Length matching leaves differences in vocabulary, discourse and familiarity. Real catalogues also require open-ended answers, source credibility and a cost model for incorrect entries. Those decisions lie beyond these readouts.
Reconstruct the result on CPU
The public package stores individual choices, world-level scores, frozen question plans and source attribution. CPU replay checks 435,456 readouts in the three follow-up studies, verifies 472 paired effects and writes 17 comparisons. Earlier synthetic analyses and the original WikiText comparison are identified separately in the result map.
The released context schedule verifies 1,440 mixed slots per world across 32 worlds. Recombining the document and question losses also recovers all 7,680 logged training-step objectives in the mixed and new-world studies. The adapter and Adam routines are included.
python -B verify_manifest.py
python -B reproduce.py --out replay
python -B check_methods.pySources & data
- Original manuscript · PDF
- Frozen release and checksums
- Result map and reproduction scope
- Interactive evidence explorer
- Method implementation and worked checks
This report uses the public author attribution. The PDF preserves the anonymous manuscript title page.
BibTeXVersion 1.0.1 · Added the original method implementation and an independent CPU check.