# ARS v3.10 — Issue #184 Extend Local Eval Harness + Ranking-Method Lift Gate

**Date:** 2026-05-21
**Issue:** [#184](https://github.com/Imbad0202/academic-research-skills/issues/184)
**Umbrella:** [#198](https://github.com/Imbad0202/academic-research-skills/issues/198) (v3.10 deterministic verification layer)
**Target release:** v3.10
**Companion specs:** `2026-05-21-v3.10-183-epistemic-status-spec.md` (full, the §3.6 `verification_gate` shared contract), `2026-05-21-v3.10-182-promote-citation-gate-spec.md` (lite, supplies the `lookup_verified` per-citation signal this spec's citation-extraction gold subset measures against)
**Upstream v3.8 reference:** `academic-pipeline/references/claim_audit_calibration_protocol.md` (the existing calibration runner protocol; this spec generalizes the pattern but does not duplicate it)

This is a **lite spec**. It documents the v3.10 extend delta on top of the existing v3.8 claim-audit calibration infrastructure. The umbrella issue gap analysis confirmed that ~60% of #184's original "Proposal" already ships (20-tuple gold set, FNR/FPR runner, T-C1/T-C2/T-C3 threshold tests, benchmark-report honesty schema); v3.10 #184 generalizes the calibration pattern, adds two more gold-set tracks, introduces a ranking-method lift gate, and ties CI path filters to a uniform eval harness.

## 1. What already ships (do not rebuild)

| Capability | Existing path | First shipped |
|---|---|---|
| 20-tuple synthetic gold set (12 alignment + 8 constraint) | `scripts/fixtures/claim_audit_calibration/gold_set.json` | v3.8 |
| Calibration runner with FNR/FPR aggregate + per-class reporting | `scripts/claim_audit_calibration.py` | v3.8 |
| T-C1 (aggregate FNR<0.15, FPR<0.10) / T-C2 (per-class) / T-C3 (shape integrity) tests | `scripts/test_claim_audit_calibration.py` | v3.8 |
| Calibration protocol documentation | `academic-pipeline/references/claim_audit_calibration_protocol.md` | v3.8 |
| Benchmark-report honesty schema + disclosure lint (not a lift gate) | `shared/benchmark_report.schema.json`, `scripts/check_benchmark_report.py` | v3.8 |
| Literature quality scoring rubric (potential baseline metric source for ranking-change comparisons; the exact section heading needs verification against repo state at implementation time before being wired as a baseline) | `academic-paper/agents/literature_strategist_agent.md` | v3.5 |

The umbrella issue's reframing made the #184 scope clear: this child generalizes the v3.8 calibration runner from "FNR/FPR for the claim-audit judge" to "lift/regression for any ranking-method or generation change," and adds two new gold-set tracks (epistemic-status classification, summarization adequacy) on top of the existing claim-audit track. The v3.8 surface is the reference design; v3.10 builds the general harness using the same pattern.

## 2. Phase 1 / Phase 2 dependency order

#198 declared a two-phase dependency for this spec's deliverables:

**Phase 1 (parallel with #182/#183 ship):**
- Citation-extraction gold subset (this spec's deliverable 1) — required as baseline for #182 `lookup_verified` promotion (otherwise the promotion has no measurement target).

**Phase 2 (sequential after #183 ships):**
- Status-classification gold subset — depends on #183 `epistemic_status` schema field landing.
- Summarization-adequacy gold subset — depends on #183 prose-register inference landing (so the gold set can score writer outputs against extractor-assigned status).
- `run_evals.py` general harness — can land in Phase 1 or Phase 2; only depends on the calibration-runner pattern (already shipped).
- Ranking-method lift gate — independent of #182/#183 ship; can land any time.
- Human expert concordance rows — independent infrastructure; can land any time.

The phasing matters because #182's blocking-default promotion (per #182 spec Delta 3) needs a measurement target the day it ships. The citation-extraction subset is the only #184 deliverable on #182's critical path.

## 3. Five v3.10 delta items

### Delta 1 — `evals/gold/` directory + three core-task gold subsets

A new top-level directory `evals/gold/` holds the v3.10 generalized gold sets. The directory mirrors the v3.8 `scripts/fixtures/claim_audit_calibration/` pattern but is not constrained to claim-audit.

```
evals/
├── README.md                          # entry point for human contributors
├── gold/
│   ├── citation_extraction/           # Phase 1 — baseline for #182
│   │   ├── README.md
│   │   ├── manifest.yaml              # task description, threshold contract, sample N
│   │   ├── tuples/
│   │   │   ├── 001-doi-resolves.json
│   │   │   ├── 031-fabricated-synthetic-genome.json
│   │   │   ├── ...
│   │   │   └── 050-fabricated-counterfactual-spectral-knapsack.json
│   │   └── expected_outcomes.json     # ground-truth per tuple
│   ├── status_classification/         # Phase 2 — depends on #183
│   │   └── ...                        # mirror layout
│   └── summarization_adequacy/        # Phase 2 — depends on #183
│       └── ...
```

#### 3.1.1 Citation-extraction track (Phase 1)

Target: measure the per-citation `lookup_verified` 3-class enum (`true` / `false` / `unresolvable`) classification accuracy produced by #182's `verification_gate.verify_citation` against ground-truth resolver outcomes. The metric is classification accuracy (all three classes treated symmetrically), not boolean accuracy — collapsing `unresolvable` into `false` would mis-attribute manual-skip cases as verification failures. Initial corpus: 50 hand-curated tuples covering:

- 20 tuples — citations with valid DOI (no arXiv ID); expected `lookup_verified=true`, `resolver_outcomes.crossref.status="matched"`, `resolver_outcomes.arxiv.status="skipped"` (arXiv resolver does not run on non-arXiv citations per #182 Delta 1 design; the `.status` enum value is `"skipped"`, not a separate `arxiv_unmatched` boolean).
- 10 tuples — citations with valid arXiv ID (no DOI); expected `lookup_verified=true`, `resolver_outcomes.arxiv.status="matched"`.
- 5 tuples — manual entries (`obtained_via: manual`); expected resolvers `.status="skipped"` per v3.7.3 exemption on all four resolvers, `lookup_verified=unresolvable` (every applicable resolver is `skipped` ⇒ the set of adjudicating resolvers is empty per #182 Delta 4).
- 15 tuples — fabricated references (intentionally-bogus DOI + bogus title); expected `lookup_verified=false`, all applicable resolvers `.status="unmatched"`. Mirrors v3.8's fabricated-reference fixture. Per #182 Delta 4 anti-fabrication bias, partial outage on these tuples (some `.status="unreachable"` from a transient API outage during fixture replay) still yields `lookup_verified=false` if at least one resolver returned `.status="unmatched"`.

Threshold contract for citation-extraction: aggregate `accuracy >= 0.90` against the 50-tuple expected outcomes. Per-class threshold (true / false / unresolvable): `accuracy >= 0.85` per class. These are **v3.10.0 binding default thresholds**; changing them before v3.10.0 ship requires a spec amendment, while post-ship tuning happens via manifest version bump per E-V2 / OQ-1.

> **Amendment (2026-05-27, #250):** The `valid_unresolvable` class (10 tuples) was removed before v3.10.0 ship. No stable, first-party-verifiable citation that is unmatched across all four resolvers was found under current index coverage; the class was unbuildable. Tuples 031–040 are reassigned to `fabricated` (total 15). The `false` class is carried entirely by fabricated tuples. The accepted coverage gap (no real-but-unindexed tuple) is tracked in #250. The `lookup_verified` reducer (#182) is unchanged.

#### 3.1.2 Status-classification track (Phase 2)

Target: measure #183's `epistemic_status_extractor` accuracy against human-labeled ground truth. Initial corpus: 30 source paper excerpts with hand-labeled `epistemic_status ∈ {Established, Supported, Preliminary, Speculative, Contested}` (6 per label).

Threshold contract: aggregate `accuracy >= 0.75` across the 5 labels. Per-label `accuracy >= 0.65`. The labels are inherently softer than citation-extraction's binary, so thresholds are deliberately lower. Phase 2 schedule: gold set authoring begins after #183 ships.

#### 3.1.3 Summarization-adequacy track (Phase 2)

Target: measure whether prose generated by `synthesis_agent` / `draft_writer_agent` / `report_compiler_agent` preserves the source register, given the extractor-assigned `epistemic_status`. Each tuple is `(source_text, extractor_assigned_status, writer_emitted_prose, ground_truth_silent_upgrade)`. Initial corpus: 40 tuples covering all 6 upgrade-pair types (Speculative→Preliminary/Supported/Established, Preliminary→Supported/Established, Supported→Established) plus same-level and downgrade negative controls.

Threshold contract: aggregate `silent_upgrade detection accuracy >= 0.85`. Per upgrade-pair `accuracy >= 0.75`. Phase 2 schedule: depends on #183 prose-register inference landing.

### Delta 2 — `scripts/run_evals.py` general harness

Generalize the v3.8 `scripts/claim_audit_calibration.py` pattern from a single-task FNR/FPR runner to a multi-task harness driven by `evals/gold/<task>/manifest.yaml` declarations.

- **Public CLI**: `python -m scripts.run_evals [--task <task_name>] [--baseline <path>] [--compare <path>] [--output <report.json>]`. Default behavior (no `--task`): run all tasks declared under `evals/gold/`. With `--task citation_extraction`: run only that task.
- **Baseline / compare mode** (Delta 4 ranking-method lift gate): when both `--baseline` and `--compare` are provided, the harness emits a side-by-side report showing pre/post metrics per task per class. Without them, the harness emits a single-run report against the gold set.
- **Output shape**: writes to a new `shared/evals_lift_report.schema.json` (specified per #183 §3.6 forward declaration). The schema is purpose-built for multi-task harness output and is structurally distinct from `shared/benchmark_report.schema.json` (which is a single-benchmark-vs-human-baseline disclosure schema, not a multi-task per-class metrics shape). The v3.8 `benchmark_report` schema is unchanged. The new schema's required fields: `harness_version`, `run_id`, `gold_set_version`, `per_task` array, and `caveats` (the last mirrors the v3.8 honesty-disclosure convention). Each `per_task[]` entry carries `task_name`, `manifest_version`, `aggregate_metric`, `per_class` array, optional `lift_pre` / `lift_post` (when invoked in baseline/compare mode), and optional `expert_concordance` per class (when human-labeled tuples exist per Delta 5).
- **Reuse contract**: `run_evals.py` calls `claim_audit_calibration.run_calibration` for the claim-audit task and a parallel runner for each new task. The v3.8 calibration runner is not modified; v3.10 adds new tasks that conform to the same I/O shape (tuple-in, per-tuple-verdict-out, aggregate-metric-emit).
- **Path filter integration** (Delta 3): a separate CI workflow `.github/workflows/eval-harness.yml` triggers `run_evals.py` on PRs touching paths listed in Delta 3.

### Delta 3 — CI path filter triggers eval comparison

Today the v3.8 calibration runner is invoked manually. v3.10 wires it into CI to fire automatically on PRs that change scoring or generation logic.

- **Path filter**: PRs touching any of the following trigger `run_evals.py` against the affected gold-set tasks:
  - `scripts/claim_audit_*.py` → trigger claim-audit task
  - `scripts/verification_gate/**`, `scripts/arxiv_client.py`, `scripts/verification_cache.py`, `scripts/crossref_client.py`, `scripts/openalex_client.py`, `scripts/citation_verification_summary.py`, and the Semantic Scholar lookup module invoked from `deep-research/agents/bibliography_agent.md` (path determined at implementation time per #182 Delta 5 note) → trigger citation-extraction task (per #182 Delta 5). Rationale: any resolver implementation change can affect the `lookup_verified` truth signal that the citation-extraction gold set measures, so all four resolver clients plus the unified summary derivation module are on the same critical path as the verification_gate wrapper.
  - `scripts/epistemic_status_extractor.py`, `scripts/check_silent_upgrade.py`, `shared/references/epistemic_status_vocabulary.md` → trigger status-classification + summarization-adequacy tasks (per #183 deliverable 6/7; the vocabulary doc is included because §5.3.3 confirms it carries the active inference patterns).
  - `deep-research/agents/synthesis_agent.md`, `academic-paper/agents/draft_writer_agent.md`, `deep-research/agents/report_compiler_agent.md` → trigger status-classification + summarization-adequacy tasks (writer prompts that govern prose register)
  - `evals/gold/**` itself → trigger all tasks (gold-set edits are eval-config changes and must regress-test against existing pipeline)
- **Workflow definition**: `.github/workflows/eval-harness.yml` (new). Concurrency group `${{ github.event_name }}-eval-harness-${{ github.ref }}` per repo CI discipline. Runs `python -m scripts.run_evals --task <triggered_task>` and posts a markdown comment to the PR with the run metrics.
- **Block-on-regression policy**: a PR that causes any per-task threshold to fall below the manifest contract is **blocked** unless the PR description includes an explicit `[eval-regression-acknowledged]` token AND a follow-up issue URL link. CI verification mechanism: `.github/workflows/eval-harness.yml` runs a small shell step that parses the PR description (via `gh api /repos/{owner}/{repo}/pulls/{n}` and a grep over the body) for the literal `[eval-regression-acknowledged]` token AND for at least one GitHub issue URL matching `https://github.com/{owner}/{repo}/issues/\d+`. The workflow then calls `gh api` on each found issue URL to verify the issue exists and is OPEN. Missing token, missing issue URL, closed issue, or 404 on the issue URL all fail the gate. This is the **deterministic** part of the lift gate; the LLM-as-judge regression detection lives downstream in #183.

### Delta 4 — Ranking-method lift gate

When a PR changes scoring or ordering logic — e.g., `bibliography_agent` quality scoring, `literature_strategist_agent` quality rubric, the new `verification_gate.verify_citation` heuristic, or future ranking-method experiments — v3.10 requires explicit pre/post comparison rather than "this should be better in theory" claims.

- **`scripts/check_ranking_lift.py`** — new lint. Reads two `run_evals.py` outputs (baseline + compare) and computes per-task per-class lift values. Each metric in the manifest carries an explicit `direction: "higher_is_better" | "lower_is_better"` declaration (e.g., `accuracy: higher_is_better`, `FNR: lower_is_better`, `FPR: lower_is_better`).
- **Lift computation, direction-aware**:
  - For `higher_is_better` metrics: `signed_lift = (compare - baseline) / |baseline|` if `baseline != 0`; positive = improvement, negative = regression.
  - For `lower_is_better` metrics: `signed_lift = (baseline - compare) / |baseline|` if `baseline != 0`; positive = improvement, negative = regression. (The numerator inverts so the polarity convention "negative lift = regression" holds for every metric regardless of direction.)
  - Zero-baseline case (`baseline == 0`): the lift formula is undefined; the lint reports `signed_lift = "+inf"` when `compare` improves from 0 and `signed_lift = "-inf"` when `compare` regresses from 0. Manifest may declare a `zero_baseline_policy` for the metric (default: treat any non-zero `compare` as significant change and require explicit acknowledgement).
- **Gate semantics**: a PR that introduces a ranking-method change is **blocked** if any per-task `signed_lift < -0.05` (5% relative regression on the chosen metric, polarity-corrected) OR any zero-baseline metric changes value, unless the PR carries `[ranking-regression-acknowledged]` + follow-up issue. Positive or zero `signed_lift` passes.
- **Manifest declaration**: PRs that change ranking logic MUST declare the affected metric in PR description (`Affected metric: <task>.<class>.<metric>`). The check_ranking_lift lint validates the declaration matches the observed change.
- **PR template**: `.github/pull_request_template.md` adds an "Eval impact" section instructing contributors how to declare ranking changes.

### Delta 5 — Human expert concordance rows

The v3.8 gold set is synthetic (judge-written tuples with known-ground-truth verdicts). v3.10 #184 adds a parallel **human expert concordance** column to each gold-set task, recording domain-expert verdicts alongside the synthetic ground truth.

- **Schema extension**: each gold-set tuple gains an optional `human_expert_verdict` field. Initial population: 20% of each task's tuples (10 of 50 citation-extraction tuples, 6 of 30 status-classification, 8 of 40 summarization-adequacy) hand-labeled by domain experts.
- **Concordance metric**: `scripts/run_evals.py` emits an `expert_concordance` field per task per class (named per #183 §3.6 contract vocabulary), computed as the agreement rate between the synthetic ground truth and the human expert verdicts on the subset of tuples that carry a `human_expert_verdict`.
- **Use case**: concordance < 0.80 on any task is a signal that the synthetic gold set is mis-calibrated against domain reality. v3.10 does not gate on concordance (the synthetic ground truth remains the source of truth for CI); concordance is an operational signal surfaced in the Stage 6 report and the `evals_lift_report` output.

Concordance scope: the 20% sample is the initial v3.10 ship target. Expansion of the human-labeled coverage is a future workstream; per-tuple expert verdicts are expensive to source. The infrastructure (schema field + concordance computation) lands at v3.10; the data populates over time.

## 4. Deliverable list (paths)

1. `evals/` top-level directory + `evals/README.md` (Delta 1)
2. `evals/gold/citation_extraction/` — Phase 1 corpus (50 tuples + manifest + expected_outcomes) (Delta 1)
3. `evals/gold/status_classification/` — Phase 2 corpus (30 tuples + manifest, Phase-2 schedule) (Delta 1)
4. `evals/gold/summarization_adequacy/` — Phase 2 corpus (40 tuples + manifest, Phase-2 schedule) (Delta 1)
5. `scripts/run_evals.py` — general harness (Delta 2)
6. `shared/evals_lift_report.schema.json` — new schema for multi-task harness output (Delta 2; aligned with #183 §3.6 forward declaration)
7. `.github/workflows/eval-harness.yml` — CI path-filter workflow (Delta 3)
8. `scripts/check_ranking_lift.py` — ranking-lift gate lint (Delta 4)
9. `.github/pull_request_template.md` — Eval impact section (Delta 4)
10. **gold-set schema field** — `human_expert_verdict` optional field on each tuple type (Delta 5)
11. **Tests** — `scripts/test_run_evals.py`, `scripts/test_check_ranking_lift.py`, `scripts/test_evals_citation_extraction.py`, fixtures for Phase 2 tasks added when those tasks ship.
12. CI wiring: extend `.github/workflows/spec-consistency.yml` with the new test paths; `.github/workflows/eval-harness.yml` is the new workflow per Delta 3.
13. `CHANGELOG.md` + `docs/ARCHITECTURE.md` + ROADMAP §3.10 anchor — v3.10 release-alignment 6-invariant set per repo discipline.

## 5. Out of scope

- **Replacing the v3.8 claim-audit calibration runner.** `scripts/claim_audit_calibration.py` and its 20-tuple gold set remain in place; `run_evals.py` calls into the existing runner for the claim-audit task. v3.10 does not duplicate v3.8 calibration logic.
- **Real-corpus evaluation.** v3.10 #184 gold sets are synthetic (or 20% human-labeled). Running the harness against a large real-paper corpus is out of scope; the harness is designed to scale to such corpora when they become available, but no real-corpus deliverable lands at v3.10.
- **Cross-language gold sets.** Initial corpora are English-only. Traditional Chinese / multilingual evaluation tracks are a future workstream tracked separately.
- **`evidence_type` 6-tier biomedical refinement gold set.** Per #198 vocabulary lock, v3.10 stays on the 5-tier cross-disciplinary enum. A 6-tier biomedical track would land alongside the deferred `evidence_type` follow-up if that schema field is ever added.
- **Continuous regression tracking.** v3.10 ships per-PR gate checks; longitudinal trend tracking (e.g., "task accuracy over the last 90 days") is deferred to a future operational-observability workstream.

## 6. INVARIANTS

The I-V1 through I-V15 from `2026-05-21-v3.10-183-epistemic-status-spec.md` §9 and the C-V1 through C-V5 from `2026-05-21-v3.10-182-promote-citation-gate-spec.md` §5 are not duplicated. The following are #184-specific:

**E-V1 — Gold-set immutability per release.** Once a `evals/gold/<task>/manifest.yaml` version ships in a release, the corpus tuples for that version are frozen. Patches may add tuples; modifying or removing existing tuples requires a manifest version bump. This protects calibration from silent drift.

**E-V2 — Per-task threshold contract.** Each task's `manifest.yaml` declares its aggregate threshold and per-class thresholds. The CI lint reads the manifest, not hard-coded values, so threshold tuning over time is a manifest-edit operation, not a code change.

**E-V3 — Synthetic vs expert concordance.** The synthetic ground truth (in `expected_outcomes.json`) is the source of truth for CI gates. Human expert verdicts in `human_expert_verdict` fields are advisory; mismatch between synthetic and expert verdicts surfaces as an `expert_concordance` metric but does not fail the CI gate. v3.10 does not gate on concordance; future versions may.

**E-V4 — Lift-gate metric explicitness.** PRs that touch ranking logic MUST declare `Affected metric: <task>.<class>.<metric>` in PR description. The `check_ranking_lift` lint cross-checks the declaration against observed metric changes; undeclared regressions block ship, declared regressions require explicit `[ranking-regression-acknowledged]` plus follow-up issue.

**E-V5 — Phase-1-first dependency.** The citation-extraction gold subset MUST ship at or before #182 ship cutoff. The status-classification and summarization-adequacy subsets MUST NOT block #182 ship; they are sequential Phase-2 work depending on #183 surfaces.

## 7. Acceptance criteria

**D1 — `evals/` directory shipped.** Top-level `evals/` directory present with `README.md`, `gold/` subdirectory, and at minimum the citation-extraction Phase-1 corpus.

**D2 — Citation-extraction gold subset shipped (Phase 1).** `evals/gold/citation_extraction/` contains 50 hand-curated tuples per §3.1.1, manifest with 0.90 aggregate / 0.85 per-class thresholds, and `expected_outcomes.json` ground truth. The harness can run `python -m scripts.run_evals --task citation_extraction` and emit a report.

### v3.10.0 ship acceptance (blocking)

**D1 — `evals/` directory shipped.** Top-level `evals/` directory present with `README.md`, `gold/` subdirectory, and at minimum the citation-extraction Phase-1 corpus.

**D2 — Citation-extraction gold subset shipped (Phase 1).** `evals/gold/citation_extraction/` contains 50 hand-curated tuples per §3.1.1, manifest with 0.90 aggregate / 0.85 per-class thresholds (v3.10.0 binding defaults per OQ-1), and `expected_outcomes.json` ground truth. The harness can run `python -m scripts.run_evals --task citation_extraction` and emit a report.

**D3 — `run_evals.py` general harness shipped.** Module accepts `--task` flag, defaults to running all tasks. `--baseline` / `--compare` flags produce side-by-side reports. Output conforms to the new `shared/evals_lift_report.schema.json`.

**D4 — `evals_lift_report` schema shipped.** New `shared/evals_lift_report.schema.json` lands per §3.2 Delta 2 output-shape description. The v3.8 `shared/benchmark_report.schema.json` is **unchanged** by v3.10 #184. The two schemas serve different purposes and remain independent surfaces.

**D5 — Eval-harness CI workflow shipped.** `.github/workflows/eval-harness.yml` triggers `run_evals.py` on PRs matching the Delta 3 path filter. PR markdown comment with metrics generated. Concurrency group includes `github.event_name` suffix.

**D6 — Ranking-lift gate shipped, direction-aware.** `scripts/check_ranking_lift.py` operational. Each manifest metric declares `direction: higher_is_better | lower_is_better`. Polarity-corrected `signed_lift < -0.05` blocks PR unless `[ranking-regression-acknowledged]` token + follow-up issue present. Zero-baseline metric changes also block unless acknowledged. `.github/pull_request_template.md` includes Eval impact section per §3.4 Delta 4.

**D7 — Expert concordance field shipped.** Each gold-set tuple schema includes optional `human_expert_verdict` field. `run_evals.py` emits `expert_concordance` metric per task per class (aligned with #183 §3.6 contract vocabulary). Initial 20% sample populated for the citation-extraction Phase-1 corpus.

**D8 — v3.8 calibration runner preserved.** `scripts/claim_audit_calibration.py` and its T-C1/T-C2/T-C3 tests are unchanged by v3.10 #184. The general harness calls into the existing runner for the claim-audit task.

**D9 — Six-doc release alignment.** At tag time, `scripts/check_version_consistency.py` reports zero drift across CHANGELOG, README, ARCHITECTURE, PERFORMANCE, bilingual mirror docs, and package metadata per repo release discipline.

**D10 — Shared-contract claim verified.** The §3.6 `verification_gate` contract per #183 spec is implemented for the lift-gate row: this spec's `run_evals.py` + `check_ranking_lift.py` produce `lift_pre` / `lift_post` / `expert_concordance` columns in the new `evals_lift_report` schema, matching the #183 §3.6 contract table's "output column" property for the lift gate row.

### Post-#183 follow-up acceptance (non-blocking for v3.10.0)

The following depend on #183 schema surface and ship in a follow-up PR after #183 lands. They are **NOT** required for v3.10.0 ship; v3.10.0 with only the citation-extraction track is a valid ship.

**D11 (follow-up) — Status-classification gold subset.** 30-tuple corpus per §3.1.2, manifest with 0.75 aggregate / 0.65 per-class thresholds. Lands in a follow-up PR after #183 `epistemic_status` schema field is in production.

**D12 (follow-up) — Summarization-adequacy gold subset.** 40-tuple corpus per §3.1.3, manifest with 0.85 aggregate / 0.75 per upgrade-pair thresholds. Lands in a follow-up PR after #183 prose-register inference is in production.

Per OQ-3, whether D11 / D12 ship within the v3.10 release window or slip to v3.10.1 depends on #183 implementation pace. The dependency is one-directional: D11 / D12 cannot ship before #183.

## 8. Open questions deferred to implementation

**OQ-1 — Threshold tuning.** The 0.90 / 0.85 (citation) / 0.75 / 0.65 (status) / 0.85 / 0.75 (summarization) numbers are placeholder thresholds. Implementation may tune based on first 30-60 days of operational data; tuning is a manifest edit, not a code change. Recommend re-tuning at v3.10.1 patch release once production data accumulates.

**OQ-2 — Human expert sourcing.** Who labels the human concordance verdicts? v3.10 ships infrastructure for the field; sourcing protocol is operational (e.g., maintainer self-label, recruited domain experts, paid annotators). Out of scope for the schema spec.

**OQ-3 — Phase 2 timing.** Phase-2 deliverables (D8, D9) depend on #183 ship. Whether they ship within the v3.10 release window or slip to v3.10.1 depends on #183 implementation pace. The dependency is one-directional: Phase 2 cannot ship before #183. **CI-workflow sequencing constraint:** the Delta 3 path filter triggers status-classification + summarization-adequacy tasks on PRs touching `epistemic_status_extractor.py` and writer prompts. If `eval-harness.yml` ships in v3.10.0 ahead of D8/D9, the workflow MUST guard each task invocation with a `gold-set-present` precondition (skip task with a non-failing "Phase 2 deliverable pending" notice rather than fail the gate) so PRs that touch upstream writer prompts before the Phase 2 gold sets land do not get blocked by missing harness tasks. Implementation choice between "guard with precondition" and "defer the workflow to ship alongside D8/D9" is open; either resolution is acceptable.

**OQ-4 — Lift-gate threshold value.** The -0.05 absolute-regression threshold is a guess. Implementation may tune based on observed metric volatility; tighter thresholds (-0.02 or stricter) for production-critical metrics, looser (-0.10) for experimental ranking explorations.

**OQ-5 — Gold-set version bumping.** When a manifest version bumps (e.g., 1.0 → 1.1 because a tuple was reformulated), the threshold contract resets. Whether to require a sandbox window (e.g., 7 days running new tuples without gating) before activating the new threshold is deferred.

## 9. References

- Issue #184: https://github.com/Imbad0202/academic-research-skills/issues/184
- Umbrella #198: https://github.com/Imbad0202/academic-research-skills/issues/198
- Companion full spec #183 (`verification_gate` contract source of truth): `docs/design/2026-05-21-v3.10-183-epistemic-status-spec.md`
- Companion lite spec #182 (provides `lookup_verified` signal measured by citation-extraction track): `docs/design/2026-05-21-v3.10-182-promote-citation-gate-spec.md`
- v3.8 calibration protocol (reference design): `academic-pipeline/references/claim_audit_calibration_protocol.md`
- v3.8 source-of-truth (claim-audit pipeline + 8-row finalizer): `docs/design/2026-05-15-issue-103-claim-alignment-audit-spec.md`
- v3.8 calibration runner: `scripts/claim_audit_calibration.py`
- v3.8 benchmark-report honesty schema: `shared/benchmark_report.schema.json`
- External motivation: Ghareeb et al. 2026 (specialized literature agent versus generic frontier model on drug-discovery proposal hallucination rate). Cited as motivation for the lift-gate principle ("borrowed paper numbers are reference only; production decisions wait for local gold-set numbers" per #198 design discipline notes). Production threshold tuning per OQ-1.
