Dataset Freeze Record

Dataset: triplets_60.json Version: 2.5.0 (matches dataset_manifest.json) Checksum (SHA-256): b27455067e80a011653ef79f989794ac9923569335f40dc33ddbaba654fe6c23 Status: EXPLORATORY — not yet frozen

Verify with:

shasum -a 256 -c checksums.sha256      # -> triplets_60.json: OK

Freeze status

This dataset is not a sealed test set yet. Per dataset_manifest.json (freeze_status: "exploratory", freeze_date: null), all 60 triplets are agent-generated and were iteratively refined together; held_out_status is "None; no realization is a true held-out test set." No realization is held out.

The real freeze (v3.0) happens only after the Appendix C human-validation study closes and adjudication is complete (see HUMAN_STUDY_RUNBOOK.md). At that point the validated subset is fixed, this record is updated to status: frozen with a freeze_date, and all baselines are re-run and locked against the new checksum.

Composition (three realizations, not held-out)

Realization IDs Count Purpose
A 1–20 20 First realization of each schema; development / error analysis
B 21–40 20 Second realization; surface-variation coverage
C 41–60 20 Third realization; naturalistic style variation

Blocks A/B/C are surface realizations of the same 20 schemas, not a train/test split. Because the schemas recur across blocks, per-block results are not independent evaluations; report block-level numbers descriptively only.

Provenance and limitations

  • Synthetically generated by a single agent (Claude); schemas are agent-designed, not empirically derived.
  • All realizations iteratively refined together — there is no clean held-out block. Any “test” framing must wait for the frozen v3.0 validated subset.
  • English only; 20 structural schemas × 3 realizations.
  • Independent human validation is pending (the study this record gates).

Statistical notes

  • 60 items derive from 20 schemas → observations are not independent. Use schema-clustered bootstrap intervals; item-level intervals understate uncertainty (dataset_manifest.json → statistical_notes).

Requirements before claiming test results

  1. ✓ Checksum recorded and verifiable (checksums.sha256).
  2. ✓ Development vs. realization reporting kept descriptive (not a held-out claim).
  3. ⬜ Independent human validation (PENDING — Appendix C study).
  4. ⬜ Dataset frozen at v3.0 on the validated subset before any test claim.
  5. ⬜ Baselines re-run and locked on the frozen subset.

Change log

  • v2.5.0 — round-4 Q3 domain-drift fix; re-anchored q3_near_disanalogy for ids [23,28,29,30,41,48,53,59] to each query’s domain after annotator-1 graded validation (surface lure restored; structure unchanged; seed-42 A/B randomization preserved). Checksum b2745506…; still exploratory.
  • v2.4.0 — exploratory release; checksum 6d25163b…; freeze deferred to v3.0.
  • Earlier SEALED_TEST_SET.md (v2.2.0) prematurely described Blocks B/C as a frozen “Sealed evaluation” and recorded a stale checksum (73a3f45c…); that claim contradicted the manifest and has been corrected here.