Files
Neutron/.planning/phases/01-reproducible-fidelity-baseline/01-VALIDATION.md
T

111 lines
13 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
phase: "01"
slug: "reproducible-fidelity-baseline"
status: validated
nyquist_compliant: true
wave_0_complete: true
created: "2026-11-12"
---
# Phase 01 — Validation Strategy
## Test Infrastructure
| Property | Value |
|---|---|
| Framework | Existing Python stdlib unittest, standalone .NET 10 console projects; Node built-in tests for pure viewer logic |
| Config file | Existing scripts/test.sh and scripts/env.sh; no new package dependencies |
| Quick run command | `python3 -m unittest discover -s tests -p 'test_fidelity_*.py' -v` |
| Viewer quick command | `node --test tests/fidelity-viewer.test.mjs` |
| Full suite command | `scripts/test.sh` (includes counted Python, Node viewer and managed observer checks) |
| Runtime | Broad fidelity run: 40.072 seconds (30-second target exceeded); focused tooling checks typically 1–5 seconds, process/observer checks can take longer. Complete local suite: 782.049626 seconds. |
## Sampling Rate
- After every task: run its focused tests; require a positive test count, not empty discovery.
- After each plan: run all fidelity tests and relevant existing managed/runtime regressions when their boundaries changed.
- After each wave and before verify-work: run scripts/test.sh and all newly introduced viewer checks.
- Every runnable automated command has an adjacent fails_when predicate in PLAN.md.
- Synthetic media and receipt fixtures validate tooling only. Actual Windows/DxWnd and native gameplay receipts require identified execution.
## Per-Task Verification Map
Final task bindings below reference the exact executable `<automated>` blocks in the named PLAN.md/task. Each block carries its adjacent observable `<fails_when>`; Python commands assert the minimum discovered count before running. The tests exist and have run with positive counts. The status column separates passing automation from actual capture/review still OPEN. Shared EVID requirement acceptance is not implied by any green tooling row.
| Task | Wave | Exact automated command binding | Minimum count / threat refs | Actual result |
|---|---|---|---|---|
| 01-01-01 | 1 | 01-01-PLAN.md task 01-01-01: counted `test_fidelity_sessions.py` runner and mandated original inventory invocation | 16; T-01-01/02/03 | green: 29 session tests; 539 originals |
| 01-01-02 | 1 | 01-01-PLAN.md task 01-01-02: counted `test_fidelity_scenarios.py` runner | 12; T-01-02 | green: 25 scenario tests |
| 01-02-01 | 2 | 01-02-PLAN.md task 01-02-01: counted `test_fidelity_review.py` runner; `node --test tests/fidelity-viewer.test.mjs` | Python 10, Node positive; T-01-04/05/06 | green: 21 review + 27 Node tests |
| 01-02-02 | 2 | 01-02-PLAN.md task 01-02-02: `node --test tests/fidelity-viewer.test.mjs` | 16; T-01-04/05/06 | green: 27 Node tests |
| 01-03-01 | 2 | 01-03-PLAN.md task 01-03-01: counted `test_fidelity_ledger.py` runner | 12; T-01-07/08/09 | green: 31 ledger tests |
| 01-03-02 | 2 | 01-03-PLAN.md task 01-03-02: counted ledger runner; `python3 tools/recon/fidelity_ledger.py validate --ledger docs/evidence/capabilities.json --requirements .planning/REQUIREMENTS.md` | 16 tests, 33 requirements; T-01-07/08/09 | green: 31 tests; 33 requirement mappings validated |
| 01-04-01 | 2 | 01-04-PLAN.md task 01-04-01: exact sourced-SDK observer command and counted `test_fidelity_observation.py` runner | Managed 8, Python 4; T-01-10/11/12 | green: 23 managed + 7 process tests |
| 01-04-02 | 2 | 01-04-PLAN.md task 01-04-02: sourced-SDK observer/build and MovementTests/DirectorTests/CheckpointTests/BodyOwnershipTests commands | Managed 14 plus positive existing groups; T-01-10/11/12 | green: observer/build and protected groups 28/23/4/3 |
| 01-05-01 | 3 | 01-05-PLAN.md task 01-05-01: counted `test_fidelity_checkpoint.py` runner; `bash -n scripts/prepare-fidelity-checkpoint.sh` | 12; T-01-13/14/15 | green: 25 checkpoint tests; shell syntax |
| 01-05-02 | 3 | 01-05-PLAN.md task 01-05-02: counted checkpoint runner; package-free PowerShell command deferred to final human session | Python 16, Windows 8; T-01-13/14/15 | green: 25 Python + 18 Linux-hosted PowerShell checks; native Windows session OPEN |
| 01-06-01 | 4 | 01-06-PLAN.md task 01-06-01: counted `test_fidelity_capture.py` runner | 10; T-01-16/17/18 | green: 23 capture tests; actual Linux capture OPEN |
| 01-06-02 | 4 | 01-06-PLAN.md task 01-06-02: counted capture runner and original inventory command | 14; T-01-16/17/18 | green: 23 capture tests; 539 originals; actual matrix OPEN |
| 01-07-01 | 5 | 01-07-PLAN.md task 01-07-01: counted `test_fidelity_acceptance.py` runner; `python3 tools/recon/fidelity_acceptance.py ready --checkpoint dist/fidelity/phase01` | 14; T-01-19/20/21 | green: 29 acceptance tests; readiness exit 0 |
| 01-07-02 | 5 | `python3 tools/recon/fidelity_acceptance.py audit --checkpoint dist/fidelity/phase01 --results dist/fidelity/phase01/results --require-complete` | Real matrix/native/original/human checks required; T-01-19/20/21 | OPEN: strict audit exit 1, errors=[], 0/120; fail-closed behavior covered by passing acceptance tests |
| 01-07-03 | 5 | `scripts/test.sh`; `python3 tools/recon/inventory.py ../Original --verify docs/original-manifest.json` | Positive consolidated counts, 539 originals; all threats | green: completed full suite exit 0; 539 originals |
All 15 task bindings were checked against the seven PLAN verify blocks and SUMMARY interfaces. No missing automated contract was found. Task 01-07-02 remains an explicit actual-evidence dependency; its expected nonzero strict audit is not a green acceptance result. Focused commands ran during their owning slices; the completed integrated suite and subsequent tooling regressions are recorded below. No full suite or large checkpoint hash walk was repeated for this audit.
The seven explicit classified spec-less boundaries and two unresolved unclassified manual assumptions are reconciled in 01-SOURCE-AUDIT.md. The two research backstop truths abstain to human_needed if actual original audiovisual capture/browser review is insufficient. Descriptor-less project prohibitions remain flagged-unverified, not silently passed. ASVS1 high/critical threats block acceptance; T-01-SC is reserved across plans.
## Wave 0 Requirements — Complete (Automation)
- Each implementation slice creates meaningful tests together with its production code; no empty stubs counted as validation.
- Implemented fidelity test fixtures covering changed/missing/added originals, output overlap, symlink/drive/case escape, reused session, tampered hash, invalid marker and unsupported schema.
- Implemented pure viewer tests including asymmetric durations, missing/buffering tracks, preserved playbackRate 1.0 and exclusive audio switching.
- No external test-framework installation required. Existing full suite remains available.
## Manual-Only Verifications
| Behavior | Requirement | Why Manual | Test Instructions |
|---|---|---|---|
| Original visible and audible through DxWnd | EVID-02 | User Windows PC and wrapper/media behavior | Use generated verified copy; record Windows/DxWnd/recorder versions and settings; execute the defined matrix and import hashed recordings/annotations |
| Native Windows port route/input/audio | EVID-04 | Cannot be replaced by Linux cross-publish or Wine | Run local checkpoint bundle on user PC; capture actual route and physical input/audio; collect exact package/resource identities and logs |
| Native Linux route/input/audio | EVID-04 | Synthetic/headless fixtures have narrower scope | Run equivalent identified package/matrix on Linux with actual input, audible checks and captures |
| Local playback review | D-05–D-08 | Browser codecs, audible output and media interaction | Load real paired recordings, inspect optional overlays, event seeks/drift and source audio switching without retiming |
Unavailable/failed executions are recorded as open dependencies. Completing tools or preparing a bundle does not close EVID-02/EVID-04 or mark Phase 1 accepted. Source-backed later work may continue only under the existing roadmap exception.
## Measured Verification Evidence
- Completed integrated run: `artifacts/fidelity/01-07-full-suite.json` and `.log`, exit 0 in **782.049626 seconds**. Actual output records 556 Python tests (180 fidelity), 27 Node tests, 23 standalone managed observer checks, two separate HUD tests, all 51 managed entry-point invocations/build, 36 rendered Vulkan receipts with zero reported validation errors, SDL dummy-audio checks and 539 unchanged originals. Optional native-window checks were not enabled. Dummy audio does not establish audibility.
- Subsequent code-review fixes: `01-FIX-REPORT.md` and `artifacts/fidelity/01-review-fixes/`. `python-fidelity.log` records **189 passing tests in 40.072 seconds**; `acceptance-final.log` records **29 passing acceptance tests** after the final standalone-transfer regression. These runs overlap; do not add their counts. `node-viewer-direct.log` records 27 behavior tests, `powershell.log` 18 synthetic checks under Linux-hosted PowerShell, and `original-preservation.log` 539 unchanged originals. The initial sandbox socket errors were resolved by the scoped local run; they are not represented as passes.
- Current discovery, without executing runtime loops: acceptance 29, capture 23, checkpoint 25, ledger 31, observation 7, review 21, scenarios 25, sessions 29: **190 current fidelity tests**. The final added acceptance test explains the difference from the earlier 189-test aggregate; it was executed in the final 29-test acceptance run. Discovery alone is not execution evidence.
- Review fixes exercise selected-receipt digest binding, pristine executable/inventory identity, named copy-state lifetime binding, sealed ledger/history projection including standalone transfer, and incomplete-session recovery without captured promotion. They strengthen existing covered contracts; independent review remains separate.
- Fresh immutable checkpoint: **11001 members**, `dist/fidelity/phase01`. Saved `01-review-fixes/readiness.json` reports `materials_ready: true`, `acceptance: human_needed` (command exit 0). Saved `acceptance.json` reports `errors: []`, **0/120**, all ten human findings pending (strict command exit 1). Original-DxWnd, native Windows and native Linux each retain 40 outstanding obligations.
- Focused tooling feedback is generally around 1–5 seconds (session/scenario/ledger around one second, checkpoint 4.742 seconds, review 5.234 seconds in owning summaries); observer process integration measured 19.126 seconds. The broad 40.072-second regression **exceeds** the proposed 30-second quick target. This is a latency WARNING, not a missing behavioral check or evidence acceptance.
## Validation Sign-Off
- [x] Final plans bind every task to automated checks or explicit human/native evidence
- [x] Sampling continuity has no three consecutive tasks without automated verification
- [x] New test files exist and execute nonzero test counts
- [x] No watch-mode flags
- [ ] Broad quick feedback below 30 seconds — measured 40.072 seconds; WARNING retained
- [x] Native/original capture limitations recorded
- [x] Implemented automated validation contract audited; `nyquist_compliant` and `wave_0_complete` refer only to automation coverage
- [ ] Actual combined capture and human review complete — OPEN
- [ ] EVID requirements and Phase 01 acceptance complete — OPEN
**Automation validation:** Validated on 2026-11-13 from executed tests and retained logs; no new automated gap found, no tests or implementation changed. The latency target remains a documented warning. `status: validated` does not advance the phase or promote any EVID requirement.
**Acceptance dependencies:** A1, A2, EVID-02-unclassified, EVID-04-unclassified, P1, P2, P3, browser, paired_review and physical_input remain OPEN. Follow the single combined agenda in `docs/evidence/checkpoint-session.md` and its Windows/Linux session procedures. Collect actual original-DxWnd/native Windows/native Linux recordings, physical input and audible checks, review real paired playback, return the identified local results, then rerun strict acceptance. The Linux preflight's missing usable display/audio is an explicit dependency. Cross-publish, Wine, headless/synthetic fixtures and preparation cannot close it. Independent review/security gates remain owned by their respective audits.
## Final copy-state refinement
All191 fidelity checks pass in43.355s after02d182f (26 scenario,23 capture,29 acceptance); missing/hash-only/wrong-size state records are explicitly rejected for both lifetimes. The43.355s aggregate still exceeds the30s quick target. No manual acceptance was promoted.
## Validation Audit 2026-11-13
| Metric | Count |
|---|---|
| Gaps found | 0 |
| Resolved | 0 |
| Escalated | 0 |