Files
Neutron/.planning/phases/01-reproducible-fidelity-baseline/01-VALIDATION.md
T

86 lines
7.6 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
phase: "01"
slug: "reproducible-fidelity-baseline"
status: draft
nyquist_compliant: false
wave_0_complete: false
created: "2026-11-12"
---
# Phase 01 — Validation Strategy
## Test Infrastructure
| Property | Value |
|---|---|
| Framework | Existing Python stdlib unittest, standalone .NET 10 console projects; proposed Node built-in tests for pure viewer logic |
| Config file | Existing scripts/test.sh and scripts/env.sh; no new package dependencies |
| Quick run command | `python3 -m unittest discover -s tests -p 'test_fidelity_*.py' -v` (new tests created with their owning slice) |
| Viewer quick command | `node --test tests/fidelity-viewer.test.mjs` (new test file created with viewer slice) |
| Full suite command | `scripts/test.sh` plus the viewer quick command |
| Runtime | Quick target below 30 seconds; measure during execution. Full GPU/original-data suite runs at wave boundaries and has a separate measured duration. |
## Sampling Rate
- After every task: run its focused tests; require a positive test count, not empty discovery.
- After each plan: run all fidelity tests and relevant existing managed/runtime regressions when their boundaries changed.
- After each wave and before verify-work: run scripts/test.sh and all newly introduced viewer checks.
- Every runnable automated command has an adjacent fails_when predicate in PLAN.md.
- Synthetic media and receipt fixtures validate tooling only. Actual Windows/DxWnd and native gameplay receipts require identified execution.
## Per-Task Verification Map
Final task bindings below reference the exact executable `<automated>` blocks in the named PLAN.md/task. Each block carries its adjacent observable `<fails_when>`; Python commands assert the minimum discovered count before running. These tests are intended new artifacts, not checks already executed.
| Task | Wave | Exact automated command binding | Minimum count / threat refs |
|---|---|---|---|
| 01-01-01 | 1 | 01-01-PLAN.md task 01-01-01: counted `test_fidelity_sessions.py` runner and mandated original inventory invocation | 16; T-01-01/02/03 |
| 01-01-02 | 1 | 01-01-PLAN.md task 01-01-02: counted `test_fidelity_scenarios.py` runner | 12; T-01-02 |
| 01-02-01 | 2 | 01-02-PLAN.md task 01-02-01: counted `test_fidelity_review.py` runner; `node --test tests/fidelity-viewer.test.mjs` | Python 10, Node positive; T-01-04/05/06 |
| 01-02-02 | 2 | 01-02-PLAN.md task 01-02-02: `node --test tests/fidelity-viewer.test.mjs` | 16; T-01-04/05/06 |
| 01-03-01 | 2 | 01-03-PLAN.md task 01-03-01: counted `test_fidelity_ledger.py` runner | 12; T-01-07/08/09 |
| 01-03-02 | 2 | 01-03-PLAN.md task 01-03-02: counted ledger runner; `python3 tools/recon/fidelity_ledger.py validate --ledger docs/evidence/capabilities.json --requirements .planning/REQUIREMENTS.md` | 16 tests, 33 requirements; T-01-07/08/09 |
| 01-04-01 | 2 | 01-04-PLAN.md task 01-04-01: exact sourced-SDK observer command and counted `test_fidelity_observation.py` runner | Managed 8, Python 4; T-01-10/11/12 |
| 01-04-02 | 2 | 01-04-PLAN.md task 01-04-02: sourced-SDK observer/build and MovementTests/DirectorTests/CheckpointTests/BodyOwnershipTests commands | Managed 14 plus positive existing groups; T-01-10/11/12 |
| 01-05-01 | 3 | 01-05-PLAN.md task 01-05-01: counted `test_fidelity_checkpoint.py` runner; `bash -n scripts/prepare-fidelity-checkpoint.sh` | 12; T-01-13/14/15 |
| 01-05-02 | 3 | 01-05-PLAN.md task 01-05-02: counted checkpoint runner; package-free PowerShell command deferred to final human session | Python 16, Windows 8; T-01-13/14/15 |
| 01-06-01 | 4 | 01-06-PLAN.md task 01-06-01: counted `test_fidelity_capture.py` runner | 10; T-01-16/17/18 |
| 01-06-02 | 4 | 01-06-PLAN.md task 01-06-02: counted capture runner and original inventory command | 14; T-01-16/17/18 |
| 01-07-01 | 5 | 01-07-PLAN.md task 01-07-01: counted `test_fidelity_acceptance.py` runner; `python3 tools/recon/fidelity_acceptance.py ready --checkpoint dist/fidelity/phase01` | 14; T-01-19/20/21 |
| 01-07-02 | 5 | `python3 tools/recon/fidelity_acceptance.py audit --checkpoint dist/fidelity/phase01 --results dist/fidelity/phase01/results --require-complete` | Real matrix/native/original/human checks required; T-01-19/20/21 |
| 01-07-03 | 5 | `scripts/test.sh`; `python3 tools/recon/inventory.py ../Original --verify docs/original-manifest.json` | Positive consolidated counts, 539 originals; all threats |
All plan subcommands/paths are explicitly introduced in their owning tasks before verification. Run every focused task command after that task; at integrated checkpoint run the full suite once and repeat only after changes/failures. Quick checks target below 30 seconds, to be measured. Existing full suite and native runs can take longer and record separate durations. No new runtime/fixture/Windows results are claimed here.
The seven explicit classified spec-less boundaries and two unresolved unclassified manual assumptions are reconciled in 01-SOURCE-AUDIT.md. The two research backstop truths abstain to human_needed if actual original audiovisual capture/browser review is insufficient. Descriptor-less project prohibitions remain flagged-unverified, not silently passed. ASVS1 high/critical threats block acceptance; T-01-SC is reserved across plans.
## Wave 0 Requirements
- Each implementation slice creates meaningful tests together with its production code; no empty stubs counted as validation.
- Create fidelity test fixtures covering changed/missing/added originals, output overlap, symlink/drive/case escape, reused session, tampered hash, invalid marker and unsupported schema.
- Create pure viewer tests including asymmetric durations, missing/buffering tracks, preserved playbackRate 1.0 and exclusive audio switching.
- No external test-framework installation required. Existing full suite remains available.
## Manual-Only Verifications
| Behavior | Requirement | Why Manual | Test Instructions |
|---|---|---|---|
| Original visible and audible through DxWnd | EVID-02 | User Windows PC and wrapper/media behavior | Use generated verified copy; record Windows/DxWnd/recorder versions and settings; execute the defined matrix and import hashed recordings/annotations |
| Native Windows port route/input/audio | EVID-04 | Cannot be replaced by Linux cross-publish or Wine | Run local checkpoint bundle on user PC; capture actual route and physical input/audio; collect exact package/resource identities and logs |
| Native Linux route/input/audio | EVID-04 | Synthetic/headless fixtures have narrower scope | Run equivalent identified package/matrix on Linux with actual input, audible checks and captures |
| Local playback review | D-05–D-08 | Browser codecs, audible output and media interaction | Load real paired recordings, inspect optional overlays, event seeks/drift and source audio switching without retiming |
Unavailable/failed executions are recorded as open dependencies. Completing tools or preparing a bundle does not close EVID-02/EVID-04 or mark Phase 1 accepted. Source-backed later work may continue only under the existing roadmap exception.
## Validation Sign-Off
- [ ] Final plans bind every task to automated checks or explicit human/native evidence
- [ ] Sampling continuity has no three consecutive tasks without automated verification
- [ ] New test files exist and execute nonzero test counts
- [ ] No watch-mode flags
- [ ] Quick feedback latency measured below 30 seconds
- [ ] Native/original capture limitations recorded
- [ ] nyquist_compliant updated only after the implemented validation contract is audited
**Approval:** Pending execution; final task bindings are planned, while nyquist_compliant and wave_0_complete remain false until actual implementation and validation are audited.