Record original inventory and recover all GAM level entities

This commit is contained in:
UnixNight committed 2026-10-23 17:08:18 -04:00
commit d62e2ab4de
16 files changed
+3954

No files matched your search

+12
View File
@@ -0,0 +1,12 @@
bin/
obj/
dist/
artifacts/
.tools/
.nuget/
.dotnet-home/
__pycache__/
*.pyc
*.user
*.log
original/
+39
View File
@@ -0,0 +1,39 @@
# Neutron GamePort
Follow `../AGENTS.md` and `../prompt.MD`. This is the local Git repository.
Do not configure a remote, push, upload originals or use cloud CI.
Originals are in `../Original/` and must remain byte-identical; verify using
`python3 tools/recon/inventory.py ../Original --verify docs/original-manifest.json`.
Use .NET 10, Silk.NET.Vulkan Vulkan 1.3+, SDL3 and shared Linux/Windows x64 code.
Keep recovered data/algorithms distinct from unverified approximations. Require
runtime evidence before claiming working gameplay or platform compatibility.
## Working map
- `src/GamePort/`: runtime, platform, rendering, assets and game systems.
- `tools/`: Python analysis/extraction tools (stdlib where practical).
- `tests/`: original-data regressions and bounded parser tests.
- `scripts/`: local dependency, build, test and packaging entry points.
- `docs/`: durable evidence and continuation notes.
- `.tools/`, `.nuget/`, `artifacts/`, `dist/`: ignored local generated data.
Build from this directory with `dotnet build`, `dotnet run`; project-local SDK
may require `scripts/dotnet.sh` (consult README for verified invocation).
Python tests: `python3 -m unittest discover -s tests -v`.
Local builds: `scripts/build-linux.sh`, `scripts/build-windows.sh`;
tests: `scripts/test.sh`; packages: `scripts/package.sh`.
## Coordination
Native subagents supported, max three investigators alongside primary. Explicit
exclusive file ownership; primary reviews, integrates, tests and commits milestones.
Binary agent: `tools/binary/`, binary/graphics analysis. Asset agent: extractors,
asset/format docs, extractor tests. Infrastructure agent: runtime/build scripts and
infrastructure report. Primary: GAM loader, gameplay/recon/docs and integration.
Coordinate any ownership transfer before editing shared files. No agent commits
other agents' active changes. Keep updates concise, save detailed evidence to disk.
Use bounded endian-aware parsers, portable paths, clear C#, deterministic tests,
and explicit unmanaged ownership. Build frequently. Update `docs/progress.md`,
`docs/tasks.md`, `docs/decisions.md`, and compatibility evidence before transitions.
Current priority is exact asset recovery and one reconstructed original scene;
read those documents for actual completion status and next actions.
+74
View File
@@ -0,0 +1,74 @@
# Gameplay evidence
## Verified data reconstruction
`tools/levels/gam.py` parses all 35 supplied GAM files and accounts for every byte;
re-encoding each parsed file reproduces its original bytes. Corpus: 3,299 entities,
93 class FourCCs, 84,044 properties (22,505 string, 26,965 float, 31,275 integer,
3,295 opaque type 2, four opaque type 4). Entity type names are inferred from
matching object tags/properties, not universally proven class implementations.
All integer metadata and float32 values in GAM are **big endian** despite the
little endian x86 executable. World positions are large unscaled coordinates;
the Y coordinate is consistent with height, pending binary confirmation.
## Initial scene
`Original/NewGame.tsk` begins with `LV1B` and float32 20.0. This supports LV1B as
the new-game scene; it does not prove all save fields. `gam/level1b.gam` has 129
objects. Its player record at file offset `0x8`:
| Property | Value |
|---|---|
| kind | 3JIM |
| ObjectTag | JIM1 |
| PositionX | 5071.93310546875 |
| PositionY | 26.661895751953125 |
| PositionZ | 824.1039428710938 |
| RotationY | 155 degrees |
| StartPoint | STARTEXP |
The matching STRT record starts at `0x5b0e`. It agrees on player position/yaw;
camera position is (4873.83642578125, 235.64761352539062, 373.21588134765625),
camera rotations are zero, music is `musicsecretlab.omt` index 0, and StartTrigger
is `LABEXP`. Camera look-at behavior and the introductory trigger require code
analysis; do not treat zero stored rotation as proof of a zero effective rotation.
## Small validation scene
`gam/VR01.gam`: 16 entities, one trophy, four pickups, two phone booths and two
LOAD return triggers. Player at `0x8`: (-2678.0380859375, 87.33218383789062,
-2783.401123046875), yaw 290 degrees. STRT at `0x45c` stores camera
(-2709.25634765625, 287.3321838378906, -2390.133056640625), music `musicvr01.omt`
index 0 and start trigger `bu`. Pickups use radius 100.0 and PointValue 5.
## Controls documented by original readme
Source: `Original/readme.txt`, section V.
| Input | Original action |
|---|---|
| Up / Down | Forward / backup (rearview in scooter mode) |
| Left / Right | Turn left / right |
| Enter / Ctrl / left mouse | Primary fire |
| Shift / Insert / right mouse | Jump or jet-pack burst |
| M / O / H / I | Menu / objectives / help / item inventory |
| G | Call Goddard |
| Tab / Space | Gadget screen |
| Escape | Quit / main menu / skip cutscene |
| Mouse movement | Forward, backward, turn |
## Data is not an algorithm
LOAD exposes LevelName, StartPoint, RequiredTask, RequiredLevel, ExactLevel,
Radius, SoundIndex, FadeType and FadeTime. PIC exposes prerequisite fields,
PointValue and pickup index. These are faithfully recoverable inputs; exact
trigger conditions, distance metric, timing, task comparison and state updates
must be established from executable code or observation before implementation is
called faithful. Negative integers commonly act as flags/sentinels; their exact
meaning depends on each property.
Save data has a fixed-looking region followed by Pascal task names and big endian
values (NewGame.tsk tail includes SCENE = 30). Save writing is not yet recovered.
Physics, jumping, collision, AI, RNG, combat, progression and cutscene execution
remain under investigation. No original gameplay session has been captured yet.
+28
View File
@@ -0,0 +1,28 @@
# Architecture
Use a single .NET 10 C# executable, Silk.NET.Vulkan for Vulkan 1.3 rendering,
SDL3 for platform windowing, input and audio. Linux x64 and Windows x64 are
equally required. Source is `src/GamePort/`; root executable csproj allows
`dotnet build` and `dotnet run` from GamePort. No runtime network dependencies.
Original assets stay in `../Original`. Python tools parse undocumented original
formats into generated scene resources under `artifacts/`. The modern application
loads these original-derived resources and big endian GAM entity data. Portable
asset resolution must handle Windows-authored case differences. Extraction must
retain material references, world coordinates, per-corner UVs and animation data;
triangulation/render-state approximations require explicit documentation.
`Assets/GamLevel.cs` independently decodes original gameplay records and exposes
player/startpoint/camera and other typed properties. Exact record bytes/unknown
types remain accessible; do not normalize unknown flags into invented semantics.
`tools/levels/gam.py` provides a lossless forensic reference and JSON export.
Renderer design follows original fixed-function needs, without PBR or invented
visual effects. A headless offscreen rendering path makes actual Vulkan image
verification possible without a desktop; this is distinct from SDL presentation.
Simulation implementation must follow recovered algorithms and timing. A generic
fly camera or rendered model alone does not satisfy the playable-scene requirement.
Subsystem implementation and validated ownership are documented in
`infrastructure.md`, `analysis/graphics.md` and `rendering.md` as they become
available. Persistent docs distinguish planned architecture from verified behavior.
+21
View File
@@ -0,0 +1,21 @@
# Compatibility and validation
Status is evidence-based; compilation alone is not runtime verification.
| Feature | Implemented | Tested | Evidence / limits |
|---|---|---|---|
| Original preservation | Hash baseline | Initial corpus | 539 files in original-manifest.json |
| GAM reader/encoder Python | Yes | Linux, all 35 levels | Byte-for-byte round trip, six tests |
| GAM C# loader | Yes | Pending compilation | Original scene/player/camera exposed |
| OMT geometry recovery | In progress | Original level1 meshes | Asset agent integration pending |
| .NET 10 Linux build | In progress | Pending | Project-local SDK installed |
| Windows x64 build/package | Pending | Unverified | Cross-build will not prove execution |
| Vulkan frame | In progress | Pending | Headless Mesa available; no initial display |
| SDL3 window/input/audio | In progress | Pending | Native dependencies being built locally |
| Playable scene / original fidelity | No | No | Rendering foundation does not establish gameplay |
| Original execution/reference capture | No | No | Wine absent initially; originals not executed |
| Animation/AI/combat/progression/save | No | No | Reverse engineering ongoing |
Current known limitations: no reconstructed complete gameplay loop yet; no native
Windows testing or reference captures. Exact camera, physics, fixed-function
lighting/blending/fog and task behavior need binary/observational validation.
+19
View File
@@ -0,0 +1,19 @@
# Decisions
1. **Preserve input directory.** Keep Original in place, baseline all file hashes,
generate outputs elsewhere. No legacy binary execution in the source directory.
2. **Nested local repository.** Workspace `.git` is reserved and not a functioning
repository. Initialize GamePort/.git, no remotes, commit code/docs/tools only.
3. **Evidence-first data recovery.** Native x86 executable does not imply little
endian formats: GAM and OMT use big endian metadata. Require bounded readers,
full consumption and real original-file regressions.
4. **Initial scene.** NewGame.tsk starts LV1B and player GAM references STARTEXP.
Prioritize that scene; use smaller VR01 for data/render verification. The LV1B
LABEXP start trigger means simple camera spawning is insufficient for fidelity.
5. **Local dependencies.** Missing SDK/native tools installed under `.tools` or
`/tmp` after approved official downloads, without assuming root or altering host.
6. **Useful parallelism.** Three independent agents own binary/rendering, asset
recovery and infrastructure, while primary owns gameplay data and integration.
Shared runtime/format changes must be coordinated. No external coordination.
7. **Honest compatibility.** Build, headless Vulkan frame, SDL presentation,
gameplay fidelity and native Windows execution are separate evidence categories.
+29
View File
@@ -0,0 +1,29 @@
# GAM level entities
Derived from all 35 original files. Implemented reader and lossless encoder:
`tools/levels/gam.py`; corpus regression tests: `tests/test_levels.py`.
All multibyte integers/floats below use big endian byte order.
```
file: char[4] level_code, u32 entity_count, entity[entity_count]
entity: char[4] class_code, u32 property_count, property[property_count], byte[4] trailer
property: u8 name_length, char[name_length] name, u32 type, u32 payload_length,
[u8 string_length if type == 1], byte[payload_length] payload
```
The trailer is opaque: its purpose is unverified. The string prefix must equal
payload_length but is excluded from payload_length. Type 1 is Latin-1-compatible
text (no terminator); type 3 is float32; type 6 is signed int32. Type 2 has four
opaque bytes in this corpus (RotateToDest) and type 4 remains opaque. Retain raw
payload bytes and property order to preserve float bit patterns and unknown types.
No compression or padding occurs between the fields. There is no file footer.
Reader rejects truncation, impossible counts, duplicate properties, scalar length
mismatches, string length mismatches, non-finite known scalars and trailing bytes.
Original corpus passes these constraints; they are not proof of every conceivable
GAM version. Never read a Windows binary using host-endian assumptions.
Examples: `VR01.gam` has header `56 52 30 31 00 00 00 10`; its first object at 8
is `3JIM` with 19 properties. `level1b.gam` starts `LV1B`, 129 objects, with the
first `3JIM` at 8. All 3,299 object trailers are retained without reinterpretation.
File diff suppressed because it is too large. Load diff
+25
View File
@@ -0,0 +1,25 @@
# Progress
## Verified so far
- Read complete primary specification, inspected originals, established ownership.
- Created local-only Git repository under GamePort and persistent project guidance.
- Recorded all 539 original-file SHA-256 hashes (224,601,234 bytes).
- Identified PE32 x86 MSVC C++ game and Open Media Toolkit 2.0 DR4 engine.
- Fully decoded 35 GAM levels, 3,299 entities, 93 class codes, 84,044 properties.
- Python GAM reader/encoder round-trips every original byte; six targeted tests pass.
- C# GAM reader added with explicit endian handling and original startpoint/camera.
- OMT resource index and meshes recovered by asset investigation; integration ongoing.
- Scoped dependency download approved; local .NET 10 SDK/Silk/SDL setup ongoing.
## Current work
Binary/rendering investigation, exact material/texture/audio recovery, C# runtime,
Vulkan offscreen verification, SDL3 presentation, cross-platform packaging.
No claim of full playable reconstruction, original behavior comparison, or Windows
runtime validation. See `tasks.md` and `compatibility.md` for remaining work.
## Continuation
Read AGENTS, this file, tasks, architecture and analysis reports. Check local Git
status and active agents before editing. Avoid repeating completed inventory/GAM
analysis. Original initial-scene evidence points to LV1B / STARTEXP, with VR01 as
a smaller resource-validation scene. Exact movement defaults now recovered in
binary investigation; do not substitute unverified gameplay guesses.
+50
View File
@@ -0,0 +1,50 @@
# Initial reconnaissance
Original game identified from supplied readme: **Jimmy Neutron: Boy Genius**.
Root input directory is `../Original/`, preserved in place. The specification
is `../prompt.MD` (uppercase extension on disk), read in full before work.
539 input files, 224,601,234 bytes. Complete per-file sizes, first bytes and
SHA-256 hashes: `original-manifest.json`. Reproduce or verify via
`tools/recon/inventory.py`. There are 254 ASE, 128 PNG, 100 OMT, 35 GAM,
12 TSK, five EXE, three DLL, one readme TXT and one uninstall ISU.
## Binaries and API evidence
`file` identifies Neutron.exe and OMT2.dll as PE32 x86 Windows GUI files, each
with four sections. Binary analysis identifies MSVC C++ and Open Media Toolkit
2.0 DR4. Imported DirectDrawCreateEx, DirectInput8 and DirectSound plus engine
renderer symbols establish legacy graphics/audio/input infrastructure. The
readme's requirement for DirectX 8 does not alone establish Direct3D version.
See `analysis/binary.md` and `analysis/graphics.md` for addresses and confidence.
## Asset evidence
ASE files are readable 3ds Max ASCII export version 200, including frame
animation. `ASE/jimrun.ASE` names jimmyrun.max and stores 0..4 frames at 10 fps,
480 ticks/frame, with position, normals and UV data. PNG files are unmodified
original textures. Proprietary OMT databases begin `0MF2` and have a big endian
index offset; the asset investigation has recovered the resource table and
original world-space meshes. GAM entities are fully decoded and byte-round-trip
verified across all 35 files. TSK saves remain partially understood.
Details: `analysis/assets.md`, `file-formats.md`, `gam-format.md`,
`analysis/gameplay.md`. Generated outputs belong under ignored `artifacts/`.
## Initial environment
Available initially: git, Python 3.12, file, objdump, strings, gcc, cmake,
curl/wget, Vulkan loader and Mesa ICDs including lavapipe. No dotnet in PATH;
no Wine, Ghidra/Rizin/radare2, glslang, spirv tools, SDL3 or Xvfb in PATH.
No DISPLAY or WAYLAND_DISPLAY was set. No original binary has been executed.
Native isolated-context subagents, parallel tool orchestration and background
commands are supported. Three meaningful independent tasks were delegated
(binary/rendering, assets, infrastructure); primary owns gameplay/integration.
Git workspace metadata is reserved read-only, so a separate local repository
was initialized under GamePort, without remotes. Local worktrees are possible
but exclusive directory ownership is sufficient for these initial tasks.
Dependency downloads initially fail sandbox DNS. Scoped escalation for official
SDK/library downloads succeeded; dependencies are installed project-locally.
See infrastructure and compatibility reports for final actual versions/checks.
+28
View File
@@ -0,0 +1,28 @@
# Reverse-engineering index
Authoritative reference is supplied originals. Findings must be supported by file
offsets, virtual addresses, API symbols or observed behavior. Decompiled output is
evidence, not original source. Runtime behavior has not yet been observed.
- `original-manifest.json`: complete original SHA-256 baseline.
- `reconnaissance.md`: input/tool inventory, game identity and initial environment.
- `analysis/binary.md`: PE/compiler, entry points, imports and function mappings.
- `analysis/graphics.md`: renderer/API, geometry, camera and Vulkan equivalents.
- `analysis/assets.md`, `file-formats.md`: recovered OMT/ASE/texture/audio evidence.
- `gam-format.md`: fully accounted GAM wire format and exact round-trip validation.
- `analysis/gameplay.md`: original controls, startpoint/camera, entities and limits.
## Function map under investigation
Provisional mappings from binary agent (disassembly must accompany final report):
C3DJimmy update 0x424600, input 0x426030; player input 0x438A60, walking branch
0x438BC0. Player defaults ctor region 0x4372A0 sets acceleration 30, deceleration
35, maximum speed 600; increments occur per simulation invocation, while movement
uses elapsed time. A future simulation must recover update cadence before calling
these coefficients timestep-independent. Jump/gravity values remain provisional.
## Unresolved
Original execution under isolated Wine/Windows; complete gameplay algorithms;
archive animations and render states; save schema; exact historical timing and
Windows runtime compatibility. Follow `tasks.md` for concrete next actions.
+30
View File
@@ -0,0 +1,30 @@
# Tasks
## Completed
- [x] Full specification read; original inventory, hashes and local Git established.
- [x] Capabilities/tool reconnaissance; independent investigators with file ownership.
- [x] Executable format and engine identification.
- [x] Decode/round-trip all 35 GAM files; reader corruption and original-data tests.
- [x] Implement C# GAM asset reader (compilation/runtime checks pending).
## Active
- [ ] Complete binary/graphics report with addresses, rendering state and camera.
- [ ] Recover player timing, movement, gravity, collision and input algorithms.
- [ ] Extract OMT meshes, textures, materials, animations, sound and music robustly.
- [ ] Verify asset schema and integration into native Vulkan renderer.
- [ ] Compile .NET 10 runtime and verify real offscreen rendering/readback.
- [ ] Integrate SDL3 window/input/audio; exercise local display where possible.
- [ ] Local Linux and Windows self-contained builds and dependency packaging.
- [ ] Test original preservation after tools; checkpoint stable work locally.
## Required after foundation
- [ ] Run original in isolated copied-data Wine prefix or local Windows VM.
- [ ] Capture original startup, controls, camera, animation, audio and gameplay.
- [ ] Reconstruct LV1B opening sequence and one complete playable scene.
- [ ] Implement/verify collision, interactions, trigger/task system and progression.
- [ ] Implement original animation playback and correct rendering states.
- [ ] Implement menus, inventory, gadgets, combat/AI and all remaining scenes.
- [ ] Recover save/load behavior, configuration, audio mixing and transitions.
- [ ] Compare reconstruction to original references; resolve fidelity differences.
- [ ] Run native Windows tests; distinguish cross-publishing from runtime evidence.
- [ ] Produce verified local release packages and accurate known limitations.
+110
View File
@@ -0,0 +1,110 @@
using System.Buffers.Binary;
using System.Numerics;
using System.Text;
namespace GamePort.Assets;
/// <summary>Original, big endian level entities. See docs/gam-format.md.</summary>
public sealed record GamProperty(string Name, uint Type, byte[] Raw)
{
public string Text => Type == 1 ? Encoding.Latin1.GetString(Raw)
: throw new InvalidDataException($"{Name} is not a string");
public float Float => Type == 3 && Raw.Length == 4
? BitConverter.Int32BitsToSingle(BinaryPrimitives.ReadInt32BigEndian(Raw))
: throw new InvalidDataException($"{Name} is not a float32");
public int Integer => Type == 6 && Raw.Length == 4
? BinaryPrimitives.ReadInt32BigEndian(Raw)
: throw new InvalidDataException($"{Name} is not an int32");
}
public sealed record GamEntity(string Kind, int Offset,
IReadOnlyDictionary<string, GamProperty> Properties, byte[] Trailer)
{
public string? Text(string name) => Properties.TryGetValue(name, out var p) ? p.Text : null;
public float Number(string name, float fallback = 0) =>
Properties.TryGetValue(name, out var p) ? p.Float : fallback;
public int Integer(string name, int fallback = 0) =>
Properties.TryGetValue(name, out var p) ? p.Integer : fallback;
public string? Tag => Text("ObjectTag");
public Vector3 Position => new(Number("PositionX"), Number("PositionY"), Number("PositionZ"));
public Vector3 Rotation => new(Number("RotationX"), Number("RotationY"), Number("RotationZ"));
public Vector3 CameraPosition => new(Number("ViewportPx"), Number("ViewportPy"), Number("ViewportPz"));
}
public sealed record GamLevel(string Code, IReadOnlyList<GamEntity> Entities)
{
public GamEntity? Player => Entities.FirstOrDefault(e => e.Kind == "3JIM");
public GamEntity? FindStart(string? tag = null)
{
tag ??= Player?.Text("StartPoint");
return Entities.FirstOrDefault(e => e.Kind == "STRT" &&
string.Equals(e.Tag, tag, StringComparison.OrdinalIgnoreCase));
}
public static GamLevel Load(string path) => Parse(File.ReadAllBytes(path));
public static GamLevel Parse(ReadOnlySpan<byte> data)
{
var reader = new GamReader(data);
var code = reader.Text(4);
if (code.Any(c => !char.IsAsciiLetterOrDigit(c)))
throw new InvalidDataException("Invalid level FourCC");
var count = reader.Count(12);
var entities = new List<GamEntity>(count);
for (var i = 0; i < count; i++)
{
var offset = reader.Offset;
var kind = reader.Text(4);
var properties = new Dictionary<string, GamProperty>(StringComparer.Ordinal);
var propertyCount = reader.Count(10);
for (var j = 0; j < propertyCount; j++)
{
var name = reader.Pascal();
var type = reader.U32();
var length = reader.U32();
if (length > int.MaxValue)
throw new InvalidDataException($"Property too long at {reader.Offset:x}");
if (type == 1 && reader.Byte() != length)
throw new InvalidDataException($"String size mismatch at {reader.Offset:x}");
if (type is 3 or 6 && length != 4)
throw new InvalidDataException($"Scalar size mismatch at {reader.Offset:x}");
var property = new GamProperty(name, type, reader.Take((int)length).ToArray());
if (type == 3 && !float.IsFinite(property.Float))
throw new InvalidDataException($"Non-finite scalar at {reader.Offset:x}");
if (!properties.TryAdd(name, property))
throw new InvalidDataException($"Duplicate property {name}");
}
entities.Add(new GamEntity(kind, offset, properties, reader.Take(4).ToArray()));
}
if (reader.Remaining != 0)
throw new InvalidDataException($"Unexpected bytes at {reader.Offset:x}");
return new GamLevel(code, entities);
}
private ref struct GamReader(ReadOnlySpan<byte> data)
{
private readonly ReadOnlySpan<byte> data = data;
public int Offset { get; private set; }
public int Remaining => data.Length - Offset;
public ReadOnlySpan<byte> Take(int length)
{
if (length < 0 || length > Remaining)
throw new InvalidDataException($"Truncated GAM at {Offset:x}");
var result = data.Slice(Offset, length);
Offset += length;
return result;
}
public byte Byte() => Take(1)[0];
public uint U32() => BinaryPrimitives.ReadUInt32BigEndian(Take(4));
public string Text(int length) => Encoding.Latin1.GetString(Take(length));
public string Pascal() => Text(Byte());
public int Count(int minimumItemBytes)
{
var count = U32();
if (count > Remaining / minimumItemBytes)
throw new InvalidDataException($"Count exceeds remaining GAM at {Offset:x}");
return (int)count;
}
}
}
+67
View File
@@ -0,0 +1,67 @@
import json
from pathlib import Path
import struct
import sys
import unittest
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tools/levels"))
import gam
def fixture(properties=b"", count=0):
return b"TST1" + struct.pack(">I", 1) + b"3JIM" + struct.pack(">I", count) + properties + b"\x12\x34\x56\x78"
class GamTests(unittest.TestCase):
def test_real_levels_roundtrip_and_counts(self):
paths = sorted((ROOT.parent / "Original/gam").glob("*.gam"))
self.assertEqual(len(paths), 35, "original corpus is required for this integration test")
total = 0
for path in paths:
with self.subTest(path=path.name):
raw = path.read_bytes()
level = gam.parse(raw)
self.assertEqual(gam.encode(level), raw)
total += len(level.entities)
self.assertEqual(total, 3299)
def test_exact_original_start(self):
level = gam.parse((ROOT.parent / "Original/gam/level1b.gam").read_bytes())
self.assertEqual(level.code, "LV1B")
player = level.entities[0].values()
self.assertEqual(player["StartPoint"], "STARTEXP")
self.assertEqual(player["PositionX"], 5071.93310546875)
start = next(e.values() for e in level.entities
if e.kind == "STRT" and e.values()["ObjectTag"] == "STARTEXP")
self.assertEqual(start["StartTrigger"], "LABEXP")
self.assertEqual(start["MusicDatabase"], "musicsecretlab.omt")
def test_endianness_and_string_prefix(self):
props = b"\x01x" + struct.pack(">II f", 3, 4, -12.5)
props += b"\x01s" + struct.pack(">II", 1, 3) + b"\x03abc"
raw = fixture(props, 2)
level = gam.parse(raw)
self.assertEqual(level.entities[0].values(), {"x": -12.5, "s": "abc"})
self.assertEqual(gam.encode(level), raw)
def test_unknown_type_preserved(self):
raw = fixture(b"\x01x" + struct.pack(">II", 999, 3) + b"abc", 1)
self.assertEqual(gam.encode(gam.parse(raw)), raw)
def test_all_truncations_rejected(self):
raw = fixture(b"\x01x" + struct.pack(">II f", 3, 4, 12.5), 1)
for end in range(len(raw)):
with self.subTest(end=end), self.assertRaises(gam.FormatError):
gam.parse(raw[:end])
def test_overflows_trailing_data_and_bad_strings(self):
for data in [b"TST1\xff\xff\xff\xff", fixture() + b"x",
fixture(b"\x01x" + struct.pack(">II", 1, 2) + b"\x03abc", 1),
fixture(b"\x01x" + struct.pack(">II", 3, 3) + b"abc", 1)]:
with self.subTest(data=data), self.assertRaises(gam.FormatError):
gam.parse(data)
if __name__ == "__main__":
unittest.main()
+151
View File
@@ -0,0 +1,151 @@
#!/usr/bin/env python3
"""Lossless, bounded reader for the supplied Neutron GAM entity files.
No original file is modified. Unknown property types stay opaque. The four bytes
after each object are retained without assigning an unverified interpretation.
"""
from __future__ import annotations
import argparse
import json
import math
import struct
from dataclasses import asdict, dataclass
from pathlib import Path
class FormatError(ValueError):
pass
class Reader:
def __init__(self, data: bytes):
self.data = data
self.offset = 0
def take(self, count: int) -> bytes:
if count < 0 or count > len(self.data) - self.offset:
raise FormatError(f"truncated data at 0x{self.offset:x}: need {count} bytes")
result = self.data[self.offset:self.offset + count]
self.offset += count
return result
def u32(self) -> int:
return struct.unpack(">I", self.take(4))[0]
def text(self, count: int) -> str:
return self.take(count).decode("latin-1")
def pascal(self) -> str:
return self.text(self.take(1)[0])
@dataclass
class Property:
name: str
type: int
length: int
value: str | int | float
raw: str
offset: int
@dataclass
class Entity:
kind: str
offset: int
properties: list[Property]
trailer: str
def values(self) -> dict:
return {p.name: p.value for p in self.properties}
@dataclass
class Level:
code: str
entities: list[Entity]
size: int
def parse(data: bytes) -> Level:
r = Reader(data)
code = r.text(4)
if not code.isascii() or not code.isalnum():
raise FormatError("invalid level FourCC")
count = r.u32()
if count > (len(data) - 8) // 12:
raise FormatError("object count exceeds file capacity")
entities = []
for _ in range(count):
offset = r.offset
kind, property_count = r.text(4), r.u32()
if property_count > (len(data) - r.offset) // 10:
raise FormatError(f"property count exceeds capacity at 0x{offset:x}")
properties = []
names = set()
for _ in range(property_count):
property_offset = r.offset
name, type_id, length = r.pascal(), r.u32(), r.u32()
if name in names:
raise FormatError(f"duplicate property {name!r} at 0x{property_offset:x}")
names.add(name)
if type_id == 1:
actual = r.take(1)[0]
if actual != length:
raise FormatError(f"string size mismatch at 0x{property_offset:x}")
raw = r.take(length)
if type_id == 1:
value = raw.decode("latin-1")
elif type_id in (3, 6):
if length != 4:
raise FormatError(f"scalar size {length} at 0x{property_offset:x}")
value = struct.unpack(">f" if type_id == 3 else ">i", raw)[0]
if isinstance(value, float) and not math.isfinite(value):
raise FormatError(f"non-finite scalar at 0x{property_offset:x}")
else:
value = raw.hex()
properties.append(Property(name, type_id, length, value, raw.hex(), property_offset))
entities.append(Entity(kind, offset, properties, r.take(4).hex()))
if r.offset != len(data):
raise FormatError(f"{len(data) - r.offset} unexpected trailing bytes at 0x{r.offset:x}")
return Level(code, entities, len(data))
def encode(level: Level) -> bytes:
"""Re-encode parsed data to prove every byte has been accounted for."""
out = bytearray(level.code.encode("latin-1") + struct.pack(">I", len(level.entities)))
for e in level.entities:
out += e.kind.encode("latin-1") + struct.pack(">I", len(e.properties))
for p in e.properties:
name = p.name.encode("latin-1")
raw = bytes.fromhex(p.raw)
out += bytes([len(name)]) + name + struct.pack(">II", p.type, p.length)
if p.type == 1:
out += bytes([p.length])
out += raw
out += bytes.fromhex(e.trailer)
return bytes(out)
def main() -> None:
cli = argparse.ArgumentParser(description=__doc__)
cli.add_argument("input", type=Path)
cli.add_argument("--output", type=Path)
args = cli.parse_args()
raw = args.input.read_bytes()
level = parse(raw)
if encode(level) != raw:
raise FormatError("round trip differs")
text = json.dumps(asdict(level), indent=2, ensure_ascii=True, allow_nan=False) + "\n"
if args.output:
if args.output.resolve() == args.input.resolve():
cli.error("output must differ from original input")
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(text)
else:
print(text, end="")
if __name__ == "__main__":
main()
+32
View File
@@ -0,0 +1,32 @@
#!/usr/bin/env python3
"""Inventory or verify originals with SHA-256; never writes to the source tree."""
import argparse
import hashlib
import json
from pathlib import Path
def inventory(root):
return [{"path": p.relative_to(root).as_posix(), "size": p.stat().st_size,
"sha256": hashlib.sha256(p.read_bytes()).hexdigest(),
"header": p.read_bytes()[:16].hex()}
for p in sorted(root.rglob("*")) if p.is_file()]
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("root", type=Path)
parser.add_argument("--verify", type=Path)
args = parser.parse_args()
files = inventory(args.root)
if args.verify:
previous = json.loads(args.verify.read_text())["files"]
if files != previous:
raise SystemExit("Original inventory differs from recorded baseline")
print(f"Verified {len(files)} unchanged original files")
else:
print(json.dumps({"root": str(args.root), "files": files}, indent=2))
if __name__ == "__main__":
main()