86 lines
7.6 KiB
Markdown
86 lines
7.6 KiB
Markdown
---
|
||
phase: "01"
|
||
slug: "reproducible-fidelity-baseline"
|
||
status: draft
|
||
nyquist_compliant: false
|
||
wave_0_complete: false
|
||
created: "2026-11-12"
|
||
---
|
||
|
||
# Phase 01 — Validation Strategy
|
||
|
||
## Test Infrastructure
|
||
|
||
| Property | Value |
|
||
|---|---|
|
||
| Framework | Existing Python stdlib unittest, standalone .NET 10 console projects; proposed Node built-in tests for pure viewer logic |
|
||
| Config file | Existing scripts/test.sh and scripts/env.sh; no new package dependencies |
|
||
| Quick run command | `python3 -m unittest discover -s tests -p 'test_fidelity_*.py' -v` (new tests created with their owning slice) |
|
||
| Viewer quick command | `node --test tests/fidelity-viewer.test.mjs` (new test file created with viewer slice) |
|
||
| Full suite command | `scripts/test.sh` plus the viewer quick command |
|
||
| Runtime | Quick target below 30 seconds; measure during execution. Full GPU/original-data suite runs at wave boundaries and has a separate measured duration. |
|
||
|
||
## Sampling Rate
|
||
|
||
- After every task: run its focused tests; require a positive test count, not empty discovery.
|
||
- After each plan: run all fidelity tests and relevant existing managed/runtime regressions when their boundaries changed.
|
||
- After each wave and before verify-work: run scripts/test.sh and all newly introduced viewer checks.
|
||
- Every runnable automated command has an adjacent fails_when predicate in PLAN.md.
|
||
- Synthetic media and receipt fixtures validate tooling only. Actual Windows/DxWnd and native gameplay receipts require identified execution.
|
||
|
||
## Per-Task Verification Map
|
||
|
||
Final task bindings below reference the exact executable `<automated>` blocks in the named PLAN.md/task. Each block carries its adjacent observable `<fails_when>`; Python commands assert the minimum discovered count before running. These tests are intended new artifacts, not checks already executed.
|
||
|
||
| Task | Wave | Exact automated command binding | Minimum count / threat refs |
|
||
|---|---|---|---|
|
||
| 01-01-01 | 1 | 01-01-PLAN.md task 01-01-01: counted `test_fidelity_sessions.py` runner and mandated original inventory invocation | 16; T-01-01/02/03 |
|
||
| 01-01-02 | 1 | 01-01-PLAN.md task 01-01-02: counted `test_fidelity_scenarios.py` runner | 12; T-01-02 |
|
||
| 01-02-01 | 2 | 01-02-PLAN.md task 01-02-01: counted `test_fidelity_review.py` runner; `node --test tests/fidelity-viewer.test.mjs` | Python 10, Node positive; T-01-04/05/06 |
|
||
| 01-02-02 | 2 | 01-02-PLAN.md task 01-02-02: `node --test tests/fidelity-viewer.test.mjs` | 16; T-01-04/05/06 |
|
||
| 01-03-01 | 2 | 01-03-PLAN.md task 01-03-01: counted `test_fidelity_ledger.py` runner | 12; T-01-07/08/09 |
|
||
| 01-03-02 | 2 | 01-03-PLAN.md task 01-03-02: counted ledger runner; `python3 tools/recon/fidelity_ledger.py validate --ledger docs/evidence/capabilities.json --requirements .planning/REQUIREMENTS.md` | 16 tests, 33 requirements; T-01-07/08/09 |
|
||
| 01-04-01 | 2 | 01-04-PLAN.md task 01-04-01: exact sourced-SDK observer command and counted `test_fidelity_observation.py` runner | Managed 8, Python 4; T-01-10/11/12 |
|
||
| 01-04-02 | 2 | 01-04-PLAN.md task 01-04-02: sourced-SDK observer/build and MovementTests/DirectorTests/CheckpointTests/BodyOwnershipTests commands | Managed 14 plus positive existing groups; T-01-10/11/12 |
|
||
| 01-05-01 | 3 | 01-05-PLAN.md task 01-05-01: counted `test_fidelity_checkpoint.py` runner; `bash -n scripts/prepare-fidelity-checkpoint.sh` | 12; T-01-13/14/15 |
|
||
| 01-05-02 | 3 | 01-05-PLAN.md task 01-05-02: counted checkpoint runner; package-free PowerShell command deferred to final human session | Python 16, Windows 8; T-01-13/14/15 |
|
||
| 01-06-01 | 4 | 01-06-PLAN.md task 01-06-01: counted `test_fidelity_capture.py` runner | 10; T-01-16/17/18 |
|
||
| 01-06-02 | 4 | 01-06-PLAN.md task 01-06-02: counted capture runner and original inventory command | 14; T-01-16/17/18 |
|
||
| 01-07-01 | 5 | 01-07-PLAN.md task 01-07-01: counted `test_fidelity_acceptance.py` runner; `python3 tools/recon/fidelity_acceptance.py ready --checkpoint dist/fidelity/phase01` | 14; T-01-19/20/21 |
|
||
| 01-07-02 | 5 | `python3 tools/recon/fidelity_acceptance.py audit --checkpoint dist/fidelity/phase01 --results dist/fidelity/phase01/results --require-complete` | Real matrix/native/original/human checks required; T-01-19/20/21 |
|
||
| 01-07-03 | 5 | `scripts/test.sh`; `python3 tools/recon/inventory.py ../Original --verify docs/original-manifest.json` | Positive consolidated counts, 539 originals; all threats |
|
||
|
||
All plan subcommands/paths are explicitly introduced in their owning tasks before verification. Run every focused task command after that task; at integrated checkpoint run the full suite once and repeat only after changes/failures. Quick checks target below 30 seconds, to be measured. Existing full suite and native runs can take longer and record separate durations. No new runtime/fixture/Windows results are claimed here.
|
||
|
||
The seven explicit classified spec-less boundaries and two unresolved unclassified manual assumptions are reconciled in 01-SOURCE-AUDIT.md. The two research backstop truths abstain to human_needed if actual original audiovisual capture/browser review is insufficient. Descriptor-less project prohibitions remain flagged-unverified, not silently passed. ASVS1 high/critical threats block acceptance; T-01-SC is reserved across plans.
|
||
|
||
## Wave 0 Requirements
|
||
|
||
- Each implementation slice creates meaningful tests together with its production code; no empty stubs counted as validation.
|
||
- Create fidelity test fixtures covering changed/missing/added originals, output overlap, symlink/drive/case escape, reused session, tampered hash, invalid marker and unsupported schema.
|
||
- Create pure viewer tests including asymmetric durations, missing/buffering tracks, preserved playbackRate 1.0 and exclusive audio switching.
|
||
- No external test-framework installation required. Existing full suite remains available.
|
||
|
||
## Manual-Only Verifications
|
||
|
||
| Behavior | Requirement | Why Manual | Test Instructions |
|
||
|---|---|---|---|
|
||
| Original visible and audible through DxWnd | EVID-02 | User Windows PC and wrapper/media behavior | Use generated verified copy; record Windows/DxWnd/recorder versions and settings; execute the defined matrix and import hashed recordings/annotations |
|
||
| Native Windows port route/input/audio | EVID-04 | Cannot be replaced by Linux cross-publish or Wine | Run local checkpoint bundle on user PC; capture actual route and physical input/audio; collect exact package/resource identities and logs |
|
||
| Native Linux route/input/audio | EVID-04 | Synthetic/headless fixtures have narrower scope | Run equivalent identified package/matrix on Linux with actual input, audible checks and captures |
|
||
| Local playback review | D-05–D-08 | Browser codecs, audible output and media interaction | Load real paired recordings, inspect optional overlays, event seeks/drift and source audio switching without retiming |
|
||
|
||
Unavailable/failed executions are recorded as open dependencies. Completing tools or preparing a bundle does not close EVID-02/EVID-04 or mark Phase 1 accepted. Source-backed later work may continue only under the existing roadmap exception.
|
||
|
||
## Validation Sign-Off
|
||
|
||
- [ ] Final plans bind every task to automated checks or explicit human/native evidence
|
||
- [ ] Sampling continuity has no three consecutive tasks without automated verification
|
||
- [ ] New test files exist and execute nonzero test counts
|
||
- [ ] No watch-mode flags
|
||
- [ ] Quick feedback latency measured below 30 seconds
|
||
- [ ] Native/original capture limitations recorded
|
||
- [ ] nyquist_compliant updated only after the implemented validation contract is audited
|
||
|
||
**Approval:** Pending execution; final task bindings are planned, while nyquist_compliant and wave_0_complete remain false until actual implementation and validation are audited.
|