Files
Neutron/.planning/phases/01-reproducible-fidelity-baseline/01-VALIDATION.md
T

13 KiB
Raw Blame History

phase, slug, status, nyquist_compliant, wave_0_complete, created
phase slug status nyquist_compliant wave_0_complete created
01 reproducible-fidelity-baseline validated true true 2026-11-12

Phase 01 — Validation Strategy

Test Infrastructure

Property Value
Framework Existing Python stdlib unittest, standalone .NET 10 console projects; Node built-in tests for pure viewer logic
Config file Existing scripts/test.sh and scripts/env.sh; no new package dependencies
Quick run command python3 -m unittest discover -s tests -p 'test_fidelity_*.py' -v
Viewer quick command node --test tests/fidelity-viewer.test.mjs
Full suite command scripts/test.sh (includes counted Python, Node viewer and managed observer checks)
Runtime Broad fidelity run: 40.072 seconds (30-second target exceeded); focused tooling checks typically 1–5 seconds, process/observer checks can take longer. Complete local suite: 782.049626 seconds.

Sampling Rate

  • After every task: run its focused tests; require a positive test count, not empty discovery.
  • After each plan: run all fidelity tests and relevant existing managed/runtime regressions when their boundaries changed.
  • After each wave and before verify-work: run scripts/test.sh and all newly introduced viewer checks.
  • Every runnable automated command has an adjacent fails_when predicate in PLAN.md.
  • Synthetic media and receipt fixtures validate tooling only. Actual Windows/DxWnd and native gameplay receipts require identified execution.

Per-Task Verification Map

Final task bindings below reference the exact executable <automated> blocks in the named PLAN.md/task. Each block carries its adjacent observable <fails_when>; Python commands assert the minimum discovered count before running. The tests exist and have run with positive counts. The status column separates passing automation from actual capture/review still OPEN. Shared EVID requirement acceptance is not implied by any green tooling row.

Task Wave Exact automated command binding Minimum count / threat refs Actual result
01-01-01 1 01-01-PLAN.md task 01-01-01: counted test_fidelity_sessions.py runner and mandated original inventory invocation 16; T-01-01/02/03 green: 29 session tests; 539 originals
01-01-02 1 01-01-PLAN.md task 01-01-02: counted test_fidelity_scenarios.py runner 12; T-01-02 green: 25 scenario tests
01-02-01 2 01-02-PLAN.md task 01-02-01: counted test_fidelity_review.py runner; node --test tests/fidelity-viewer.test.mjs Python 10, Node positive; T-01-04/05/06 green: 21 review + 27 Node tests
01-02-02 2 01-02-PLAN.md task 01-02-02: node --test tests/fidelity-viewer.test.mjs 16; T-01-04/05/06 green: 27 Node tests
01-03-01 2 01-03-PLAN.md task 01-03-01: counted test_fidelity_ledger.py runner 12; T-01-07/08/09 green: 31 ledger tests
01-03-02 2 01-03-PLAN.md task 01-03-02: counted ledger runner; python3 tools/recon/fidelity_ledger.py validate --ledger docs/evidence/capabilities.json --requirements .planning/REQUIREMENTS.md 16 tests, 33 requirements; T-01-07/08/09 green: 31 tests; 33 requirement mappings validated
01-04-01 2 01-04-PLAN.md task 01-04-01: exact sourced-SDK observer command and counted test_fidelity_observation.py runner Managed 8, Python 4; T-01-10/11/12 green: 23 managed + 7 process tests
01-04-02 2 01-04-PLAN.md task 01-04-02: sourced-SDK observer/build and MovementTests/DirectorTests/CheckpointTests/BodyOwnershipTests commands Managed 14 plus positive existing groups; T-01-10/11/12 green: observer/build and protected groups 28/23/4/3
01-05-01 3 01-05-PLAN.md task 01-05-01: counted test_fidelity_checkpoint.py runner; bash -n scripts/prepare-fidelity-checkpoint.sh 12; T-01-13/14/15 green: 25 checkpoint tests; shell syntax
01-05-02 3 01-05-PLAN.md task 01-05-02: counted checkpoint runner; package-free PowerShell command deferred to final human session Python 16, Windows 8; T-01-13/14/15 green: 25 Python + 18 Linux-hosted PowerShell checks; native Windows session OPEN
01-06-01 4 01-06-PLAN.md task 01-06-01: counted test_fidelity_capture.py runner 10; T-01-16/17/18 green: 23 capture tests; actual Linux capture OPEN
01-06-02 4 01-06-PLAN.md task 01-06-02: counted capture runner and original inventory command 14; T-01-16/17/18 green: 23 capture tests; 539 originals; actual matrix OPEN
01-07-01 5 01-07-PLAN.md task 01-07-01: counted test_fidelity_acceptance.py runner; python3 tools/recon/fidelity_acceptance.py ready --checkpoint dist/fidelity/phase01 14; T-01-19/20/21 green: 29 acceptance tests; readiness exit 0
01-07-02 5 python3 tools/recon/fidelity_acceptance.py audit --checkpoint dist/fidelity/phase01 --results dist/fidelity/phase01/results --require-complete Real matrix/native/original/human checks required; T-01-19/20/21 OPEN: strict audit exit 1, errors=[], 0/120; fail-closed behavior covered by passing acceptance tests
01-07-03 5 scripts/test.sh; python3 tools/recon/inventory.py ../Original --verify docs/original-manifest.json Positive consolidated counts, 539 originals; all threats green: completed full suite exit 0; 539 originals

All 15 task bindings were checked against the seven PLAN verify blocks and SUMMARY interfaces. No missing automated contract was found. Task 01-07-02 remains an explicit actual-evidence dependency; its expected nonzero strict audit is not a green acceptance result. Focused commands ran during their owning slices; the completed integrated suite and subsequent tooling regressions are recorded below. No full suite or large checkpoint hash walk was repeated for this audit.

The seven explicit classified spec-less boundaries and two unresolved unclassified manual assumptions are reconciled in 01-SOURCE-AUDIT.md. The two research backstop truths abstain to human_needed if actual original audiovisual capture/browser review is insufficient. Descriptor-less project prohibitions remain flagged-unverified, not silently passed. ASVS1 high/critical threats block acceptance; T-01-SC is reserved across plans.

Wave 0 Requirements — Complete (Automation)

  • Each implementation slice creates meaningful tests together with its production code; no empty stubs counted as validation.
  • Implemented fidelity test fixtures covering changed/missing/added originals, output overlap, symlink/drive/case escape, reused session, tampered hash, invalid marker and unsupported schema.
  • Implemented pure viewer tests including asymmetric durations, missing/buffering tracks, preserved playbackRate 1.0 and exclusive audio switching.
  • No external test-framework installation required. Existing full suite remains available.

Manual-Only Verifications

Behavior Requirement Why Manual Test Instructions
Original visible and audible through DxWnd EVID-02 User Windows PC and wrapper/media behavior Use generated verified copy; record Windows/DxWnd/recorder versions and settings; execute the defined matrix and import hashed recordings/annotations
Native Windows port route/input/audio EVID-04 Cannot be replaced by Linux cross-publish or Wine Run local checkpoint bundle on user PC; capture actual route and physical input/audio; collect exact package/resource identities and logs
Native Linux route/input/audio EVID-04 Synthetic/headless fixtures have narrower scope Run equivalent identified package/matrix on Linux with actual input, audible checks and captures
Local playback review D-05–D-08 Browser codecs, audible output and media interaction Load real paired recordings, inspect optional overlays, event seeks/drift and source audio switching without retiming

Unavailable/failed executions are recorded as open dependencies. Completing tools or preparing a bundle does not close EVID-02/EVID-04 or mark Phase 1 accepted. Source-backed later work may continue only under the existing roadmap exception.

Measured Verification Evidence

  • Completed integrated run: artifacts/fidelity/01-07-full-suite.json and .log, exit 0 in 782.049626 seconds. Actual output records 556 Python tests (180 fidelity), 27 Node tests, 23 standalone managed observer checks, two separate HUD tests, all 51 managed entry-point invocations/build, 36 rendered Vulkan receipts with zero reported validation errors, SDL dummy-audio checks and 539 unchanged originals. Optional native-window checks were not enabled. Dummy audio does not establish audibility.
  • Subsequent code-review fixes: 01-FIX-REPORT.md and artifacts/fidelity/01-review-fixes/. python-fidelity.log records 189 passing tests in 40.072 seconds; acceptance-final.log records 29 passing acceptance tests after the final standalone-transfer regression. These runs overlap; do not add their counts. node-viewer-direct.log records 27 behavior tests, powershell.log 18 synthetic checks under Linux-hosted PowerShell, and original-preservation.log 539 unchanged originals. The initial sandbox socket errors were resolved by the scoped local run; they are not represented as passes.
  • Current discovery, without executing runtime loops: acceptance 29, capture 23, checkpoint 25, ledger 31, observation 7, review 21, scenarios 25, sessions 29: 190 current fidelity tests. The final added acceptance test explains the difference from the earlier 189-test aggregate; it was executed in the final 29-test acceptance run. Discovery alone is not execution evidence.
  • Review fixes exercise selected-receipt digest binding, pristine executable/inventory identity, named copy-state lifetime binding, sealed ledger/history projection including standalone transfer, and incomplete-session recovery without captured promotion. They strengthen existing covered contracts; independent review remains separate.
  • Fresh immutable checkpoint: 11001 members, dist/fidelity/phase01. Saved 01-review-fixes/readiness.json reports materials_ready: true, acceptance: human_needed (command exit 0). Saved acceptance.json reports errors: [], 0/120, all ten human findings pending (strict command exit 1). Original-DxWnd, native Windows and native Linux each retain 40 outstanding obligations.
  • Focused tooling feedback is generally around 1–5 seconds (session/scenario/ledger around one second, checkpoint 4.742 seconds, review 5.234 seconds in owning summaries); observer process integration measured 19.126 seconds. The broad 40.072-second regression exceeds the proposed 30-second quick target. This is a latency WARNING, not a missing behavioral check or evidence acceptance.

Validation Sign-Off

  • Final plans bind every task to automated checks or explicit human/native evidence
  • Sampling continuity has no three consecutive tasks without automated verification
  • New test files exist and execute nonzero test counts
  • No watch-mode flags
  • Broad quick feedback below 30 seconds — measured 40.072 seconds; WARNING retained
  • Native/original capture limitations recorded
  • Implemented automated validation contract audited; nyquist_compliant and wave_0_complete refer only to automation coverage
  • Actual combined capture and human review complete — OPEN
  • EVID requirements and Phase 01 acceptance complete — OPEN

Automation validation: Validated on 2026-11-13 from executed tests and retained logs; no new automated gap found, no tests or implementation changed. The latency target remains a documented warning. status: validated does not advance the phase or promote any EVID requirement.

Acceptance dependencies: A1, A2, EVID-02-unclassified, EVID-04-unclassified, P1, P2, P3, browser, paired_review and physical_input remain OPEN. Follow the single combined agenda in docs/evidence/checkpoint-session.md and its Windows/Linux session procedures. Collect actual original-DxWnd/native Windows/native Linux recordings, physical input and audible checks, review real paired playback, return the identified local results, then rerun strict acceptance. The Linux preflight's missing usable display/audio is an explicit dependency. Cross-publish, Wine, headless/synthetic fixtures and preparation cannot close it. Independent review/security gates remain owned by their respective audits.

Final copy-state refinement

All191 fidelity checks pass in43.355s after02d182f (26 scenario,23 capture,29 acceptance); missing/hash-only/wrong-size state records are explicitly rejected for both lifetimes. The43.355s aggregate still exceeds the30s quick target. No manual acceptance was promoted.

Validation Audit 2026-11-13

Metric Count
Gaps found 0
Resolved 0
Escalated 0