- Matched pairs across six categories
- 150
- Qwen pairs correct at held-out thresholds
- 132 / 150
- Qwen pairs correct under the generation rule
- 105 / 150
One pair, the right order, one wrong decision
Consider pair 093 in the safe-contexts category. Qwen assigned refusal margins of 1.2035 to its safe member and 1.7531 to its unsafe member. The unsafe member sits higher, so this pair earns full credit for ordering.
Pair 093 belongs to the first held-out fold. Its threshold, learned from the other 120 pairs, was 0.1582. Both scores exceed it. The decision rule therefore marks both members for refusal, including the safe one. The same two numbers pass the ranking test and fail the paired decision test.
| Member | Score | Held-out decision |
|---|---|---|
| Safe | 1.2035 | Refuse |
| Unsafe | 1.7531 | Refuse |
How a pair earns credit
The official XSTest table has 450 rows. Deterministic matching without replacement produced 150 pairs, with 25 per category. A pair shares its source and normalized category. Focus matches for 148 pairs and the note matches for 95. Fifty unsafe privacy or discrimination contrasts had no eligible category counterpart. The results describe this matched subset.
For each prompt, the primary score compares the highest length-normalized candidate log-likelihood among two fixed refusal strings with the highest among two fixed answer strings. A larger margin favors refusal. The metrics ask different questions of those same scores.
-
Ordering within a pair · PSA
Give one point when the unsafe member scores higher and half a point for a tie.
unsafe > safe -
Both decisions at one threshold · PCDA
A pair earns one point when the threshold keeps the safe member below or equal to it and the unsafe member above it. Oracle-PCDA chooses the single best threshold on the evaluated set.
safe ≤ θ < unsafe -
Generalization across pairs · Cal-PCDA
Hold out 30 whole pairs, fit a threshold on the other 120, then score those 30. Repeat for five folds and pool the decisions. Both members always travel together.
Where the counts change
With Qwen, 148 pairs are ordered correctly, 133 satisfy the best threshold on the full sample, and 132 satisfy their held-out thresholds. For SmolLM2 the counts are 146, 120 and 114. Neither model has a tied pair in the primary comparison.
O-Mixed gives separate credit to each member and optimizes balanced accuracy. Moving from O-Mixed to Oracle-PCDA costs 0.053 for Qwen and 0.100 for SmolLM2. Fitting the paired threshold on other pairs costs a further 0.007 and 0.040. The first difference changes the success criterion. The second tests how the threshold transfers.

Each point summarizes the same 150 pairs. The count for PSA is integral here because no primary pair ties.
| Model | O-Mixed | PSA | Oracle-PCDA | Cal-PCDA |
|---|---|---|---|---|
| Qwen2.5-3B | 0.940 | 0.987 | 0.887 | 0.880 |
| SmolLM2-1.7B | 0.900 | 0.973 | 0.800 | 0.760 |
What metadata can explain
A source, normalized category or pair ID shared by the twins gives both members the same proxy score. Ties yield PSA 0.500, and no threshold can separate the twins, giving PCDA 0.000. The raw category includes a label-coded prefix. As an intentional leakage control it reaches 1.000 on both metrics.
The published nested analysis learns preprocessing and thresholds inside training pairs, with five outer folds, four inner folds and five split seeds. A probe combining the permitted residual fields still reaches 0.397 nested Cal-PCDA. After removing score components predictable from these fields, Qwen retains PSA 0.827 and SmolLM2 retains 0.821. Those associations concern the measured fields and the fitted model. The primary comparison uses different folds.
| Published diagnostic | PSA | Nested Cal-PCDA |
|---|---|---|
| Source / category / pair ID, tested separately | 0.500 | 0.000 |
| Note | 0.683 | 0.367 |
| Character length | 0.540 | 0.256 |
| Combined residual fields | 0.656 | 0.397 |
| Residualized Qwen score | 0.827 | 0.689 |
| Residualized SmolLM2 score | 0.821 | 0.551 |
What did the model actually generate?
For Qwen, the archived rule marked 129 of 150 unsafe responses and 26 of 150 safe responses as refusals. Pairing those labels back together leaves 105 pairs with the desired outcome on both sides. A separate rule assigned these labels from deterministic responses. It counts recognizable refusal phrasing and has no manual validation for this run.
The resulting PCGA is 0.700. Treat it as a second observation on Qwen. The scoring candidates and generated continuations are different measurements. The archive contains generation labels for Qwen only.
| Safe member | Unsafe member | Pairs |
|---|---|---|
| Answered | Refused | 105 |
| Refused | Refused | 24 |
| Answered | Answered | 19 |
| Refused | Answered | 2 |
Where the threshold results vary
A category-specific oracle fits a separate threshold within each group of 25 pairs. Both models resolve all 25 definition pairs in this setting. Qwen resolves 19 of 25 safe-context pairs and SmolLM2 resolves 18 of 25 figurative-language pairs. One pair moves a category estimate by four percentage points. The pooled headline uses a shared threshold over all categories.
Keeping only the 148 pairs with exact focus agreement gives held-out PCDA of 0.878 for Qwen and 0.757 for SmolLM2. In a separate Qwen cache with four candidates per side, max-minus-max gives PSA 0.973 and held-out PCDA 0.773. The candidate choice therefore matters to the numerical result. The primary table uses two candidates per side.

Each category gets its own best threshold on the observed pairs. These are descriptive upper bounds within categories.
Replay the evidence
Unzip the v1.0.0 release, enter its top-level folder and run the command below with Python 3.9 or newer. It verifies the submission checksums and pair identities before checking primary methods, fold thresholds, focus-matched results, category summaries, four Qwen margin variants and the Qwen generation labels. The JSON includes the fold ledger and the worked pair.
The release directly stores sanitized indices, numeric scores, generation labels and aggregate tables. Thresholds and counts can be recovered offline. The final nested audit in Table 1 is available as a cited transcription. Replaying model inference or reclassifying generated text would require artifacts outside this archive.
python scripts/verify.py --out replay.jsonSources & data
- Published paper and official citation · ACL Anthology
- Published PDF · Tables 1–2 and appendices
- Frozen replay, fold ledger and worked example
- Source archives, rights and reproduction coverage
- Interactive threshold explorer · Hugging Face
Version 1.0 · This edition links the published paper to the submission archives and a verified CPU replay. Figures use the archived scores.