docs(01): close review and security gates with acceptance pending
This commit is contained in:
1 parent
02d182f35d
commit
71366e7d45
14 files changed
+417
-77
No files matched your search
@@ -1,9 +1,9 @@
|
||||
---
|
||||
phase: "01"
|
||||
slug: "reproducible-fidelity-baseline"
|
||||
status: draft
|
||||
nyquist_compliant: false
|
||||
wave_0_complete: false
|
||||
status: validated
|
||||
nyquist_compliant: true
|
||||
wave_0_complete: true
|
||||
created: "2026-11-12"
|
||||
---
|
||||
|
||||
@@ -13,12 +13,12 @@ created: "2026-11-12"
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Framework | Existing Python stdlib unittest, standalone .NET 10 console projects; proposed Node built-in tests for pure viewer logic |
|
||||
| Framework | Existing Python stdlib unittest, standalone .NET 10 console projects; Node built-in tests for pure viewer logic |
|
||||
| Config file | Existing scripts/test.sh and scripts/env.sh; no new package dependencies |
|
||||
| Quick run command | `python3 -m unittest discover -s tests -p 'test_fidelity_*.py' -v` (new tests created with their owning slice) |
|
||||
| Viewer quick command | `node --test tests/fidelity-viewer.test.mjs` (new test file created with viewer slice) |
|
||||
| Full suite command | `scripts/test.sh` plus the viewer quick command |
|
||||
| Runtime | Quick target below 30 seconds; measure during execution. Full GPU/original-data suite runs at wave boundaries and has a separate measured duration. |
|
||||
| Quick run command | `python3 -m unittest discover -s tests -p 'test_fidelity_*.py' -v` |
|
||||
| Viewer quick command | `node --test tests/fidelity-viewer.test.mjs` |
|
||||
| Full suite command | `scripts/test.sh` (includes counted Python, Node viewer and managed observer checks) |
|
||||
| Runtime | Broad fidelity run: 40.072 seconds (30-second target exceeded); focused tooling checks typically 1–5 seconds, process/observer checks can take longer. Complete local suite: 782.049626 seconds. |
|
||||
|
||||
## Sampling Rate
|
||||
|
||||
@@ -30,35 +30,35 @@ created: "2026-11-12"
|
||||
|
||||
## Per-Task Verification Map
|
||||
|
||||
Final task bindings below reference the exact executable `<automated>` blocks in the named PLAN.md/task. Each block carries its adjacent observable `<fails_when>`; Python commands assert the minimum discovered count before running. These tests are intended new artifacts, not checks already executed.
|
||||
Final task bindings below reference the exact executable `<automated>` blocks in the named PLAN.md/task. Each block carries its adjacent observable `<fails_when>`; Python commands assert the minimum discovered count before running. The tests exist and have run with positive counts. The status column separates passing automation from actual capture/review still OPEN. Shared EVID requirement acceptance is not implied by any green tooling row.
|
||||
|
||||
| Task | Wave | Exact automated command binding | Minimum count / threat refs |
|
||||
|---|---|---|---|
|
||||
| 01-01-01 | 1 | 01-01-PLAN.md task 01-01-01: counted `test_fidelity_sessions.py` runner and mandated original inventory invocation | 16; T-01-01/02/03 |
|
||||
| 01-01-02 | 1 | 01-01-PLAN.md task 01-01-02: counted `test_fidelity_scenarios.py` runner | 12; T-01-02 |
|
||||
| 01-02-01 | 2 | 01-02-PLAN.md task 01-02-01: counted `test_fidelity_review.py` runner; `node --test tests/fidelity-viewer.test.mjs` | Python 10, Node positive; T-01-04/05/06 |
|
||||
| 01-02-02 | 2 | 01-02-PLAN.md task 01-02-02: `node --test tests/fidelity-viewer.test.mjs` | 16; T-01-04/05/06 |
|
||||
| 01-03-01 | 2 | 01-03-PLAN.md task 01-03-01: counted `test_fidelity_ledger.py` runner | 12; T-01-07/08/09 |
|
||||
| 01-03-02 | 2 | 01-03-PLAN.md task 01-03-02: counted ledger runner; `python3 tools/recon/fidelity_ledger.py validate --ledger docs/evidence/capabilities.json --requirements .planning/REQUIREMENTS.md` | 16 tests, 33 requirements; T-01-07/08/09 |
|
||||
| 01-04-01 | 2 | 01-04-PLAN.md task 01-04-01: exact sourced-SDK observer command and counted `test_fidelity_observation.py` runner | Managed 8, Python 4; T-01-10/11/12 |
|
||||
| 01-04-02 | 2 | 01-04-PLAN.md task 01-04-02: sourced-SDK observer/build and MovementTests/DirectorTests/CheckpointTests/BodyOwnershipTests commands | Managed 14 plus positive existing groups; T-01-10/11/12 |
|
||||
| 01-05-01 | 3 | 01-05-PLAN.md task 01-05-01: counted `test_fidelity_checkpoint.py` runner; `bash -n scripts/prepare-fidelity-checkpoint.sh` | 12; T-01-13/14/15 |
|
||||
| 01-05-02 | 3 | 01-05-PLAN.md task 01-05-02: counted checkpoint runner; package-free PowerShell command deferred to final human session | Python 16, Windows 8; T-01-13/14/15 |
|
||||
| 01-06-01 | 4 | 01-06-PLAN.md task 01-06-01: counted `test_fidelity_capture.py` runner | 10; T-01-16/17/18 |
|
||||
| 01-06-02 | 4 | 01-06-PLAN.md task 01-06-02: counted capture runner and original inventory command | 14; T-01-16/17/18 |
|
||||
| 01-07-01 | 5 | 01-07-PLAN.md task 01-07-01: counted `test_fidelity_acceptance.py` runner; `python3 tools/recon/fidelity_acceptance.py ready --checkpoint dist/fidelity/phase01` | 14; T-01-19/20/21 |
|
||||
| 01-07-02 | 5 | `python3 tools/recon/fidelity_acceptance.py audit --checkpoint dist/fidelity/phase01 --results dist/fidelity/phase01/results --require-complete` | Real matrix/native/original/human checks required; T-01-19/20/21 |
|
||||
| 01-07-03 | 5 | `scripts/test.sh`; `python3 tools/recon/inventory.py ../Original --verify docs/original-manifest.json` | Positive consolidated counts, 539 originals; all threats |
|
||||
| Task | Wave | Exact automated command binding | Minimum count / threat refs | Actual result |
|
||||
|---|---|---|---|---|
|
||||
| 01-01-01 | 1 | 01-01-PLAN.md task 01-01-01: counted `test_fidelity_sessions.py` runner and mandated original inventory invocation | 16; T-01-01/02/03 | green: 29 session tests; 539 originals |
|
||||
| 01-01-02 | 1 | 01-01-PLAN.md task 01-01-02: counted `test_fidelity_scenarios.py` runner | 12; T-01-02 | green: 25 scenario tests |
|
||||
| 01-02-01 | 2 | 01-02-PLAN.md task 01-02-01: counted `test_fidelity_review.py` runner; `node --test tests/fidelity-viewer.test.mjs` | Python 10, Node positive; T-01-04/05/06 | green: 21 review + 27 Node tests |
|
||||
| 01-02-02 | 2 | 01-02-PLAN.md task 01-02-02: `node --test tests/fidelity-viewer.test.mjs` | 16; T-01-04/05/06 | green: 27 Node tests |
|
||||
| 01-03-01 | 2 | 01-03-PLAN.md task 01-03-01: counted `test_fidelity_ledger.py` runner | 12; T-01-07/08/09 | green: 31 ledger tests |
|
||||
| 01-03-02 | 2 | 01-03-PLAN.md task 01-03-02: counted ledger runner; `python3 tools/recon/fidelity_ledger.py validate --ledger docs/evidence/capabilities.json --requirements .planning/REQUIREMENTS.md` | 16 tests, 33 requirements; T-01-07/08/09 | green: 31 tests; 33 requirement mappings validated |
|
||||
| 01-04-01 | 2 | 01-04-PLAN.md task 01-04-01: exact sourced-SDK observer command and counted `test_fidelity_observation.py` runner | Managed 8, Python 4; T-01-10/11/12 | green: 23 managed + 7 process tests |
|
||||
| 01-04-02 | 2 | 01-04-PLAN.md task 01-04-02: sourced-SDK observer/build and MovementTests/DirectorTests/CheckpointTests/BodyOwnershipTests commands | Managed 14 plus positive existing groups; T-01-10/11/12 | green: observer/build and protected groups 28/23/4/3 |
|
||||
| 01-05-01 | 3 | 01-05-PLAN.md task 01-05-01: counted `test_fidelity_checkpoint.py` runner; `bash -n scripts/prepare-fidelity-checkpoint.sh` | 12; T-01-13/14/15 | green: 25 checkpoint tests; shell syntax |
|
||||
| 01-05-02 | 3 | 01-05-PLAN.md task 01-05-02: counted checkpoint runner; package-free PowerShell command deferred to final human session | Python 16, Windows 8; T-01-13/14/15 | green: 25 Python + 18 Linux-hosted PowerShell checks; native Windows session OPEN |
|
||||
| 01-06-01 | 4 | 01-06-PLAN.md task 01-06-01: counted `test_fidelity_capture.py` runner | 10; T-01-16/17/18 | green: 23 capture tests; actual Linux capture OPEN |
|
||||
| 01-06-02 | 4 | 01-06-PLAN.md task 01-06-02: counted capture runner and original inventory command | 14; T-01-16/17/18 | green: 23 capture tests; 539 originals; actual matrix OPEN |
|
||||
| 01-07-01 | 5 | 01-07-PLAN.md task 01-07-01: counted `test_fidelity_acceptance.py` runner; `python3 tools/recon/fidelity_acceptance.py ready --checkpoint dist/fidelity/phase01` | 14; T-01-19/20/21 | green: 29 acceptance tests; readiness exit 0 |
|
||||
| 01-07-02 | 5 | `python3 tools/recon/fidelity_acceptance.py audit --checkpoint dist/fidelity/phase01 --results dist/fidelity/phase01/results --require-complete` | Real matrix/native/original/human checks required; T-01-19/20/21 | OPEN: strict audit exit 1, errors=[], 0/120; fail-closed behavior covered by passing acceptance tests |
|
||||
| 01-07-03 | 5 | `scripts/test.sh`; `python3 tools/recon/inventory.py ../Original --verify docs/original-manifest.json` | Positive consolidated counts, 539 originals; all threats | green: completed full suite exit 0; 539 originals |
|
||||
|
||||
All plan subcommands/paths are explicitly introduced in their owning tasks before verification. Run every focused task command after that task; at integrated checkpoint run the full suite once and repeat only after changes/failures. Quick checks target below 30 seconds, to be measured. Existing full suite and native runs can take longer and record separate durations. No new runtime/fixture/Windows results are claimed here.
|
||||
All 15 task bindings were checked against the seven PLAN verify blocks and SUMMARY interfaces. No missing automated contract was found. Task 01-07-02 remains an explicit actual-evidence dependency; its expected nonzero strict audit is not a green acceptance result. Focused commands ran during their owning slices; the completed integrated suite and subsequent tooling regressions are recorded below. No full suite or large checkpoint hash walk was repeated for this audit.
|
||||
|
||||
The seven explicit classified spec-less boundaries and two unresolved unclassified manual assumptions are reconciled in 01-SOURCE-AUDIT.md. The two research backstop truths abstain to human_needed if actual original audiovisual capture/browser review is insufficient. Descriptor-less project prohibitions remain flagged-unverified, not silently passed. ASVS1 high/critical threats block acceptance; T-01-SC is reserved across plans.
|
||||
|
||||
## Wave 0 Requirements
|
||||
## Wave 0 Requirements — Complete (Automation)
|
||||
|
||||
- Each implementation slice creates meaningful tests together with its production code; no empty stubs counted as validation.
|
||||
- Create fidelity test fixtures covering changed/missing/added originals, output overlap, symlink/drive/case escape, reused session, tampered hash, invalid marker and unsupported schema.
|
||||
- Create pure viewer tests including asymmetric durations, missing/buffering tracks, preserved playbackRate 1.0 and exclusive audio switching.
|
||||
- Implemented fidelity test fixtures covering changed/missing/added originals, output overlap, symlink/drive/case escape, reused session, tampered hash, invalid marker and unsupported schema.
|
||||
- Implemented pure viewer tests including asymmetric durations, missing/buffering tracks, preserved playbackRate 1.0 and exclusive audio switching.
|
||||
- No external test-framework installation required. Existing full suite remains available.
|
||||
|
||||
## Manual-Only Verifications
|
||||
@@ -72,14 +72,39 @@ The seven explicit classified spec-less boundaries and two unresolved unclassifi
|
||||
|
||||
Unavailable/failed executions are recorded as open dependencies. Completing tools or preparing a bundle does not close EVID-02/EVID-04 or mark Phase 1 accepted. Source-backed later work may continue only under the existing roadmap exception.
|
||||
|
||||
## Measured Verification Evidence
|
||||
|
||||
- Completed integrated run: `artifacts/fidelity/01-07-full-suite.json` and `.log`, exit 0 in **782.049626 seconds**. Actual output records 556 Python tests (180 fidelity), 27 Node tests, 23 standalone managed observer checks, two separate HUD tests, all 51 managed entry-point invocations/build, 36 rendered Vulkan receipts with zero reported validation errors, SDL dummy-audio checks and 539 unchanged originals. Optional native-window checks were not enabled. Dummy audio does not establish audibility.
|
||||
- Subsequent code-review fixes: `01-FIX-REPORT.md` and `artifacts/fidelity/01-review-fixes/`. `python-fidelity.log` records **189 passing tests in 40.072 seconds**; `acceptance-final.log` records **29 passing acceptance tests** after the final standalone-transfer regression. These runs overlap; do not add their counts. `node-viewer-direct.log` records 27 behavior tests, `powershell.log` 18 synthetic checks under Linux-hosted PowerShell, and `original-preservation.log` 539 unchanged originals. The initial sandbox socket errors were resolved by the scoped local run; they are not represented as passes.
|
||||
- Current discovery, without executing runtime loops: acceptance 29, capture 23, checkpoint 25, ledger 31, observation 7, review 21, scenarios 25, sessions 29: **190 current fidelity tests**. The final added acceptance test explains the difference from the earlier 189-test aggregate; it was executed in the final 29-test acceptance run. Discovery alone is not execution evidence.
|
||||
- Review fixes exercise selected-receipt digest binding, pristine executable/inventory identity, named copy-state lifetime binding, sealed ledger/history projection including standalone transfer, and incomplete-session recovery without captured promotion. They strengthen existing covered contracts; independent review remains separate.
|
||||
- Fresh immutable checkpoint: **11001 members**, `dist/fidelity/phase01`. Saved `01-review-fixes/readiness.json` reports `materials_ready: true`, `acceptance: human_needed` (command exit 0). Saved `acceptance.json` reports `errors: []`, **0/120**, all ten human findings pending (strict command exit 1). Original-DxWnd, native Windows and native Linux each retain 40 outstanding obligations.
|
||||
- Focused tooling feedback is generally around 1–5 seconds (session/scenario/ledger around one second, checkpoint 4.742 seconds, review 5.234 seconds in owning summaries); observer process integration measured 19.126 seconds. The broad 40.072-second regression **exceeds** the proposed 30-second quick target. This is a latency WARNING, not a missing behavioral check or evidence acceptance.
|
||||
|
||||
## Validation Sign-Off
|
||||
|
||||
- [ ] Final plans bind every task to automated checks or explicit human/native evidence
|
||||
- [ ] Sampling continuity has no three consecutive tasks without automated verification
|
||||
- [ ] New test files exist and execute nonzero test counts
|
||||
- [ ] No watch-mode flags
|
||||
- [ ] Quick feedback latency measured below 30 seconds
|
||||
- [ ] Native/original capture limitations recorded
|
||||
- [ ] nyquist_compliant updated only after the implemented validation contract is audited
|
||||
- [x] Final plans bind every task to automated checks or explicit human/native evidence
|
||||
- [x] Sampling continuity has no three consecutive tasks without automated verification
|
||||
- [x] New test files exist and execute nonzero test counts
|
||||
- [x] No watch-mode flags
|
||||
- [ ] Broad quick feedback below 30 seconds — measured 40.072 seconds; WARNING retained
|
||||
- [x] Native/original capture limitations recorded
|
||||
- [x] Implemented automated validation contract audited; `nyquist_compliant` and `wave_0_complete` refer only to automation coverage
|
||||
- [ ] Actual combined capture and human review complete — OPEN
|
||||
- [ ] EVID requirements and Phase 01 acceptance complete — OPEN
|
||||
|
||||
**Approval:** Pending execution; final task bindings are planned, while nyquist_compliant and wave_0_complete remain false until actual implementation and validation are audited.
|
||||
**Automation validation:** Validated on 2026-11-13 from executed tests and retained logs; no new automated gap found, no tests or implementation changed. The latency target remains a documented warning. `status: validated` does not advance the phase or promote any EVID requirement.
|
||||
|
||||
**Acceptance dependencies:** A1, A2, EVID-02-unclassified, EVID-04-unclassified, P1, P2, P3, browser, paired_review and physical_input remain OPEN. Follow the single combined agenda in `docs/evidence/checkpoint-session.md` and its Windows/Linux session procedures. Collect actual original-DxWnd/native Windows/native Linux recordings, physical input and audible checks, review real paired playback, return the identified local results, then rerun strict acceptance. The Linux preflight's missing usable display/audio is an explicit dependency. Cross-publish, Wine, headless/synthetic fixtures and preparation cannot close it. Independent review/security gates remain owned by their respective audits.
|
||||
|
||||
## Final copy-state refinement
|
||||
|
||||
All191 fidelity checks pass in43.355s after02d182f (26 scenario,23 capture,29 acceptance); missing/hash-only/wrong-size state records are explicitly rejected for both lifetimes. The43.355s aggregate still exceeds the30s quick target. No manual acceptance was promoted.
|
||||
|
||||
## Validation Audit 2026-11-13
|
||||
|
||||
| Metric | Count |
|
||||
|---|---|
|
||||
| Gaps found | 0 |
|
||||
| Resolved | 0 |
|
||||
| Escalated | 0 |
|
||||
Reference in new issue
Block a user