Technical reports

Technical report

When should practice questions keep their documents?

Adapters replay the same questions with or without supporting documents. Switching the unrelated attachment at test time reverses which specialist leads.

  • Technical report 1.0.1
  • First published
  • Updated

When Does Document Replay Help Evidence-State Question Answering?

original paired test worlds
32
newly trained paired worlds
16
verified individual readouts in three follow-up studies
435,456

What did the inventory say, and what can be entered?

An inventory records Pihene as Zubiru’s storage bay. An audit later rules out that assignment. Asked what the inventory reported, the answer remains Pihene. Asked what to enter in a verified catalogue, the value must remain unspecified. Scoring just the first answer would miss whether the system respected the audit.

An inventory example, evidence-status readouts, and practice with or without attached documents.

Figure 1 traces one inventory fact through the three questions and the two replay policies.

Change the attachment while keeping the questions paired

Both specialists start from the same mixed-practice checkpoint and receive 40 further updates on the same questions and answers. A rehearses closed book; B retains documents. At test time an attachment can be an unrelated synthetic verification record or a length-matched WikiText excerpt. Joint accuracy requires three correct readouts for one fact: the reported value, its verification state and the value permitted by that state.

A third branch continues mixed practice. It matches the number of updates and supervised answers. Its later training mix and token count differ. The paired world is the uncertainty unit. The original specialist comparison uses 32 worlds; a fresh study trains new adapters on 16 new worlds.

The better specialist depends on the test document

On the original 32 worlds, B exceeds A by 19.25 percentage points with synthetic records. With WikiText, the difference is −7.92 points; its paired 95% interval is −12.53 to −3.55. The new-world comparison preserves both directions. On the original worlds, continued mixed practice also beats the stronger specialist on both attachment types.

Paired joint-accuracy differences, in percentage points. Positive B−A favors document replay.
ComparisonWorldsSyntheticWikiText
B − A32+19.25−7.92
B − A16+17.16−6.89
MIX120 − stronger specialist32+5.58+9.23

Keep the donor facts and rewrite their presentation

Rewriting the same synthetic donor facts as prose under explicit verdict cues narrows B’s advantage by 3.20 points. All four combinations of form and review cue still favor B. The larger WikiText reversal therefore cannot be assigned to prose formatting alone from this comparison.

The experiments use one Qwen3-1.7B backbone, synthetic worlds and multiple-choice outputs. Length matching leaves differences in vocabulary, discourse and familiarity. Real catalogues also require open-ended answers, source credibility and a cost model for incorrect entries. Those decisions lie beyond these readouts.

Reconstruct the result on CPU

The public package stores individual choices, world-level scores, frozen question plans and source attribution. CPU replay checks 435,456 readouts in the three follow-up studies, verifies 472 paired effects and writes 17 comparisons. Earlier synthetic analyses and the original WikiText comparison are identified separately in the result map.

The released context schedule verifies 1,440 mixed slots per world across 32 worlds. Recombining the document and question losses also recovers all 7,680 logged training-step objectives in the mixed and new-world studies. The adapter and Adam routines are included.

python -B verify_manifest.py
python -B reproduce.py --out replay
python -B check_methods.py

Sources & data

This report uses the public author attribution. The PDF preserves the anonymous manuscript title page.

BibTeX

Version 1.0.1 · Added the original method implementation and an independent CPU check.

Copy text

Copy was unavailable. Select and copy the text below.