Record original inventory and recover all GAM level entities
This commit is contained in:
commit
d62e2ab4de
16 files changed
+3954
No files matched your search
+12
@@ -0,0 +1,12 @@
|
||||
bin/
|
||||
obj/
|
||||
dist/
|
||||
artifacts/
|
||||
.tools/
|
||||
.nuget/
|
||||
.dotnet-home/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.user
|
||||
*.log
|
||||
original/
|
||||
@@ -0,0 +1,39 @@
|
||||
# Neutron GamePort
|
||||
|
||||
Follow `../AGENTS.md` and `../prompt.MD`. This is the local Git repository.
|
||||
Do not configure a remote, push, upload originals or use cloud CI.
|
||||
Originals are in `../Original/` and must remain byte-identical; verify using
|
||||
`python3 tools/recon/inventory.py ../Original --verify docs/original-manifest.json`.
|
||||
|
||||
Use .NET 10, Silk.NET.Vulkan Vulkan 1.3+, SDL3 and shared Linux/Windows x64 code.
|
||||
Keep recovered data/algorithms distinct from unverified approximations. Require
|
||||
runtime evidence before claiming working gameplay or platform compatibility.
|
||||
|
||||
## Working map
|
||||
- `src/GamePort/`: runtime, platform, rendering, assets and game systems.
|
||||
- `tools/`: Python analysis/extraction tools (stdlib where practical).
|
||||
- `tests/`: original-data regressions and bounded parser tests.
|
||||
- `scripts/`: local dependency, build, test and packaging entry points.
|
||||
- `docs/`: durable evidence and continuation notes.
|
||||
- `.tools/`, `.nuget/`, `artifacts/`, `dist/`: ignored local generated data.
|
||||
|
||||
Build from this directory with `dotnet build`, `dotnet run`; project-local SDK
|
||||
may require `scripts/dotnet.sh` (consult README for verified invocation).
|
||||
Python tests: `python3 -m unittest discover -s tests -v`.
|
||||
Local builds: `scripts/build-linux.sh`, `scripts/build-windows.sh`;
|
||||
tests: `scripts/test.sh`; packages: `scripts/package.sh`.
|
||||
|
||||
## Coordination
|
||||
Native subagents supported, max three investigators alongside primary. Explicit
|
||||
exclusive file ownership; primary reviews, integrates, tests and commits milestones.
|
||||
Binary agent: `tools/binary/`, binary/graphics analysis. Asset agent: extractors,
|
||||
asset/format docs, extractor tests. Infrastructure agent: runtime/build scripts and
|
||||
infrastructure report. Primary: GAM loader, gameplay/recon/docs and integration.
|
||||
Coordinate any ownership transfer before editing shared files. No agent commits
|
||||
other agents' active changes. Keep updates concise, save detailed evidence to disk.
|
||||
|
||||
Use bounded endian-aware parsers, portable paths, clear C#, deterministic tests,
|
||||
and explicit unmanaged ownership. Build frequently. Update `docs/progress.md`,
|
||||
`docs/tasks.md`, `docs/decisions.md`, and compatibility evidence before transitions.
|
||||
Current priority is exact asset recovery and one reconstructed original scene;
|
||||
read those documents for actual completion status and next actions.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Gameplay evidence
|
||||
|
||||
## Verified data reconstruction
|
||||
|
||||
`tools/levels/gam.py` parses all 35 supplied GAM files and accounts for every byte;
|
||||
re-encoding each parsed file reproduces its original bytes. Corpus: 3,299 entities,
|
||||
93 class FourCCs, 84,044 properties (22,505 string, 26,965 float, 31,275 integer,
|
||||
3,295 opaque type 2, four opaque type 4). Entity type names are inferred from
|
||||
matching object tags/properties, not universally proven class implementations.
|
||||
|
||||
All integer metadata and float32 values in GAM are **big endian** despite the
|
||||
little endian x86 executable. World positions are large unscaled coordinates;
|
||||
the Y coordinate is consistent with height, pending binary confirmation.
|
||||
|
||||
## Initial scene
|
||||
|
||||
`Original/NewGame.tsk` begins with `LV1B` and float32 20.0. This supports LV1B as
|
||||
the new-game scene; it does not prove all save fields. `gam/level1b.gam` has 129
|
||||
objects. Its player record at file offset `0x8`:
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| kind | 3JIM |
|
||||
| ObjectTag | JIM1 |
|
||||
| PositionX | 5071.93310546875 |
|
||||
| PositionY | 26.661895751953125 |
|
||||
| PositionZ | 824.1039428710938 |
|
||||
| RotationY | 155 degrees |
|
||||
| StartPoint | STARTEXP |
|
||||
|
||||
The matching STRT record starts at `0x5b0e`. It agrees on player position/yaw;
|
||||
camera position is (4873.83642578125, 235.64761352539062, 373.21588134765625),
|
||||
camera rotations are zero, music is `musicsecretlab.omt` index 0, and StartTrigger
|
||||
is `LABEXP`. Camera look-at behavior and the introductory trigger require code
|
||||
analysis; do not treat zero stored rotation as proof of a zero effective rotation.
|
||||
|
||||
## Small validation scene
|
||||
|
||||
`gam/VR01.gam`: 16 entities, one trophy, four pickups, two phone booths and two
|
||||
LOAD return triggers. Player at `0x8`: (-2678.0380859375, 87.33218383789062,
|
||||
-2783.401123046875), yaw 290 degrees. STRT at `0x45c` stores camera
|
||||
(-2709.25634765625, 287.3321838378906, -2390.133056640625), music `musicvr01.omt`
|
||||
index 0 and start trigger `bu`. Pickups use radius 100.0 and PointValue 5.
|
||||
|
||||
## Controls documented by original readme
|
||||
|
||||
Source: `Original/readme.txt`, section V.
|
||||
|
||||
| Input | Original action |
|
||||
|---|---|
|
||||
| Up / Down | Forward / backup (rearview in scooter mode) |
|
||||
| Left / Right | Turn left / right |
|
||||
| Enter / Ctrl / left mouse | Primary fire |
|
||||
| Shift / Insert / right mouse | Jump or jet-pack burst |
|
||||
| M / O / H / I | Menu / objectives / help / item inventory |
|
||||
| G | Call Goddard |
|
||||
| Tab / Space | Gadget screen |
|
||||
| Escape | Quit / main menu / skip cutscene |
|
||||
| Mouse movement | Forward, backward, turn |
|
||||
|
||||
## Data is not an algorithm
|
||||
|
||||
LOAD exposes LevelName, StartPoint, RequiredTask, RequiredLevel, ExactLevel,
|
||||
Radius, SoundIndex, FadeType and FadeTime. PIC exposes prerequisite fields,
|
||||
PointValue and pickup index. These are faithfully recoverable inputs; exact
|
||||
trigger conditions, distance metric, timing, task comparison and state updates
|
||||
must be established from executable code or observation before implementation is
|
||||
called faithful. Negative integers commonly act as flags/sentinels; their exact
|
||||
meaning depends on each property.
|
||||
|
||||
Save data has a fixed-looking region followed by Pascal task names and big endian
|
||||
values (NewGame.tsk tail includes SCENE = 30). Save writing is not yet recovered.
|
||||
Physics, jumping, collision, AI, RNG, combat, progression and cutscene execution
|
||||
remain under investigation. No original gameplay session has been captured yet.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Architecture
|
||||
|
||||
Use a single .NET 10 C# executable, Silk.NET.Vulkan for Vulkan 1.3 rendering,
|
||||
SDL3 for platform windowing, input and audio. Linux x64 and Windows x64 are
|
||||
equally required. Source is `src/GamePort/`; root executable csproj allows
|
||||
`dotnet build` and `dotnet run` from GamePort. No runtime network dependencies.
|
||||
|
||||
Original assets stay in `../Original`. Python tools parse undocumented original
|
||||
formats into generated scene resources under `artifacts/`. The modern application
|
||||
loads these original-derived resources and big endian GAM entity data. Portable
|
||||
asset resolution must handle Windows-authored case differences. Extraction must
|
||||
retain material references, world coordinates, per-corner UVs and animation data;
|
||||
triangulation/render-state approximations require explicit documentation.
|
||||
|
||||
`Assets/GamLevel.cs` independently decodes original gameplay records and exposes
|
||||
player/startpoint/camera and other typed properties. Exact record bytes/unknown
|
||||
types remain accessible; do not normalize unknown flags into invented semantics.
|
||||
`tools/levels/gam.py` provides a lossless forensic reference and JSON export.
|
||||
|
||||
Renderer design follows original fixed-function needs, without PBR or invented
|
||||
visual effects. A headless offscreen rendering path makes actual Vulkan image
|
||||
verification possible without a desktop; this is distinct from SDL presentation.
|
||||
Simulation implementation must follow recovered algorithms and timing. A generic
|
||||
fly camera or rendered model alone does not satisfy the playable-scene requirement.
|
||||
|
||||
Subsystem implementation and validated ownership are documented in
|
||||
`infrastructure.md`, `analysis/graphics.md` and `rendering.md` as they become
|
||||
available. Persistent docs distinguish planned architecture from verified behavior.
|
||||
@@ -0,0 +1,21 @@
|
||||
# Compatibility and validation
|
||||
|
||||
Status is evidence-based; compilation alone is not runtime verification.
|
||||
|
||||
| Feature | Implemented | Tested | Evidence / limits |
|
||||
|---|---|---|---|
|
||||
| Original preservation | Hash baseline | Initial corpus | 539 files in original-manifest.json |
|
||||
| GAM reader/encoder Python | Yes | Linux, all 35 levels | Byte-for-byte round trip, six tests |
|
||||
| GAM C# loader | Yes | Pending compilation | Original scene/player/camera exposed |
|
||||
| OMT geometry recovery | In progress | Original level1 meshes | Asset agent integration pending |
|
||||
| .NET 10 Linux build | In progress | Pending | Project-local SDK installed |
|
||||
| Windows x64 build/package | Pending | Unverified | Cross-build will not prove execution |
|
||||
| Vulkan frame | In progress | Pending | Headless Mesa available; no initial display |
|
||||
| SDL3 window/input/audio | In progress | Pending | Native dependencies being built locally |
|
||||
| Playable scene / original fidelity | No | No | Rendering foundation does not establish gameplay |
|
||||
| Original execution/reference capture | No | No | Wine absent initially; originals not executed |
|
||||
| Animation/AI/combat/progression/save | No | No | Reverse engineering ongoing |
|
||||
|
||||
Current known limitations: no reconstructed complete gameplay loop yet; no native
|
||||
Windows testing or reference captures. Exact camera, physics, fixed-function
|
||||
lighting/blending/fog and task behavior need binary/observational validation.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Decisions
|
||||
|
||||
1. **Preserve input directory.** Keep Original in place, baseline all file hashes,
|
||||
generate outputs elsewhere. No legacy binary execution in the source directory.
|
||||
2. **Nested local repository.** Workspace `.git` is reserved and not a functioning
|
||||
repository. Initialize GamePort/.git, no remotes, commit code/docs/tools only.
|
||||
3. **Evidence-first data recovery.** Native x86 executable does not imply little
|
||||
endian formats: GAM and OMT use big endian metadata. Require bounded readers,
|
||||
full consumption and real original-file regressions.
|
||||
4. **Initial scene.** NewGame.tsk starts LV1B and player GAM references STARTEXP.
|
||||
Prioritize that scene; use smaller VR01 for data/render verification. The LV1B
|
||||
LABEXP start trigger means simple camera spawning is insufficient for fidelity.
|
||||
5. **Local dependencies.** Missing SDK/native tools installed under `.tools` or
|
||||
`/tmp` after approved official downloads, without assuming root or altering host.
|
||||
6. **Useful parallelism.** Three independent agents own binary/rendering, asset
|
||||
recovery and infrastructure, while primary owns gameplay data and integration.
|
||||
Shared runtime/format changes must be coordinated. No external coordination.
|
||||
7. **Honest compatibility.** Build, headless Vulkan frame, SDL presentation,
|
||||
gameplay fidelity and native Windows execution are separate evidence categories.
|
||||
@@ -0,0 +1,29 @@
|
||||
# GAM level entities
|
||||
|
||||
Derived from all 35 original files. Implemented reader and lossless encoder:
|
||||
`tools/levels/gam.py`; corpus regression tests: `tests/test_levels.py`.
|
||||
|
||||
All multibyte integers/floats below use big endian byte order.
|
||||
|
||||
```
|
||||
file: char[4] level_code, u32 entity_count, entity[entity_count]
|
||||
entity: char[4] class_code, u32 property_count, property[property_count], byte[4] trailer
|
||||
property: u8 name_length, char[name_length] name, u32 type, u32 payload_length,
|
||||
[u8 string_length if type == 1], byte[payload_length] payload
|
||||
```
|
||||
|
||||
The trailer is opaque: its purpose is unverified. The string prefix must equal
|
||||
payload_length but is excluded from payload_length. Type 1 is Latin-1-compatible
|
||||
text (no terminator); type 3 is float32; type 6 is signed int32. Type 2 has four
|
||||
opaque bytes in this corpus (RotateToDest) and type 4 remains opaque. Retain raw
|
||||
payload bytes and property order to preserve float bit patterns and unknown types.
|
||||
No compression or padding occurs between the fields. There is no file footer.
|
||||
|
||||
Reader rejects truncation, impossible counts, duplicate properties, scalar length
|
||||
mismatches, string length mismatches, non-finite known scalars and trailing bytes.
|
||||
Original corpus passes these constraints; they are not proof of every conceivable
|
||||
GAM version. Never read a Windows binary using host-endian assumptions.
|
||||
|
||||
Examples: `VR01.gam` has header `56 52 30 31 00 00 00 10`; its first object at 8
|
||||
is `3JIM` with 19 properties. `level1b.gam` starts `LV1B`, 129 objects, with the
|
||||
first `3JIM` at 8. All 3,299 object trailers are retained without reinterpretation.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,25 @@
|
||||
# Progress
|
||||
|
||||
## Verified so far
|
||||
- Read complete primary specification, inspected originals, established ownership.
|
||||
- Created local-only Git repository under GamePort and persistent project guidance.
|
||||
- Recorded all 539 original-file SHA-256 hashes (224,601,234 bytes).
|
||||
- Identified PE32 x86 MSVC C++ game and Open Media Toolkit 2.0 DR4 engine.
|
||||
- Fully decoded 35 GAM levels, 3,299 entities, 93 class codes, 84,044 properties.
|
||||
- Python GAM reader/encoder round-trips every original byte; six targeted tests pass.
|
||||
- C# GAM reader added with explicit endian handling and original startpoint/camera.
|
||||
- OMT resource index and meshes recovered by asset investigation; integration ongoing.
|
||||
- Scoped dependency download approved; local .NET 10 SDK/Silk/SDL setup ongoing.
|
||||
|
||||
## Current work
|
||||
Binary/rendering investigation, exact material/texture/audio recovery, C# runtime,
|
||||
Vulkan offscreen verification, SDL3 presentation, cross-platform packaging.
|
||||
No claim of full playable reconstruction, original behavior comparison, or Windows
|
||||
runtime validation. See `tasks.md` and `compatibility.md` for remaining work.
|
||||
|
||||
## Continuation
|
||||
Read AGENTS, this file, tasks, architecture and analysis reports. Check local Git
|
||||
status and active agents before editing. Avoid repeating completed inventory/GAM
|
||||
analysis. Original initial-scene evidence points to LV1B / STARTEXP, with VR01 as
|
||||
a smaller resource-validation scene. Exact movement defaults now recovered in
|
||||
binary investigation; do not substitute unverified gameplay guesses.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Initial reconnaissance
|
||||
|
||||
Original game identified from supplied readme: **Jimmy Neutron: Boy Genius**.
|
||||
Root input directory is `../Original/`, preserved in place. The specification
|
||||
is `../prompt.MD` (uppercase extension on disk), read in full before work.
|
||||
|
||||
539 input files, 224,601,234 bytes. Complete per-file sizes, first bytes and
|
||||
SHA-256 hashes: `original-manifest.json`. Reproduce or verify via
|
||||
`tools/recon/inventory.py`. There are 254 ASE, 128 PNG, 100 OMT, 35 GAM,
|
||||
12 TSK, five EXE, three DLL, one readme TXT and one uninstall ISU.
|
||||
|
||||
## Binaries and API evidence
|
||||
|
||||
`file` identifies Neutron.exe and OMT2.dll as PE32 x86 Windows GUI files, each
|
||||
with four sections. Binary analysis identifies MSVC C++ and Open Media Toolkit
|
||||
2.0 DR4. Imported DirectDrawCreateEx, DirectInput8 and DirectSound plus engine
|
||||
renderer symbols establish legacy graphics/audio/input infrastructure. The
|
||||
readme's requirement for DirectX 8 does not alone establish Direct3D version.
|
||||
See `analysis/binary.md` and `analysis/graphics.md` for addresses and confidence.
|
||||
|
||||
## Asset evidence
|
||||
|
||||
ASE files are readable 3ds Max ASCII export version 200, including frame
|
||||
animation. `ASE/jimrun.ASE` names jimmyrun.max and stores 0..4 frames at 10 fps,
|
||||
480 ticks/frame, with position, normals and UV data. PNG files are unmodified
|
||||
original textures. Proprietary OMT databases begin `0MF2` and have a big endian
|
||||
index offset; the asset investigation has recovered the resource table and
|
||||
original world-space meshes. GAM entities are fully decoded and byte-round-trip
|
||||
verified across all 35 files. TSK saves remain partially understood.
|
||||
|
||||
Details: `analysis/assets.md`, `file-formats.md`, `gam-format.md`,
|
||||
`analysis/gameplay.md`. Generated outputs belong under ignored `artifacts/`.
|
||||
|
||||
## Initial environment
|
||||
|
||||
Available initially: git, Python 3.12, file, objdump, strings, gcc, cmake,
|
||||
curl/wget, Vulkan loader and Mesa ICDs including lavapipe. No dotnet in PATH;
|
||||
no Wine, Ghidra/Rizin/radare2, glslang, spirv tools, SDL3 or Xvfb in PATH.
|
||||
No DISPLAY or WAYLAND_DISPLAY was set. No original binary has been executed.
|
||||
|
||||
Native isolated-context subagents, parallel tool orchestration and background
|
||||
commands are supported. Three meaningful independent tasks were delegated
|
||||
(binary/rendering, assets, infrastructure); primary owns gameplay/integration.
|
||||
Git workspace metadata is reserved read-only, so a separate local repository
|
||||
was initialized under GamePort, without remotes. Local worktrees are possible
|
||||
but exclusive directory ownership is sufficient for these initial tasks.
|
||||
|
||||
Dependency downloads initially fail sandbox DNS. Scoped escalation for official
|
||||
SDK/library downloads succeeded; dependencies are installed project-locally.
|
||||
See infrastructure and compatibility reports for final actual versions/checks.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Reverse-engineering index
|
||||
|
||||
Authoritative reference is supplied originals. Findings must be supported by file
|
||||
offsets, virtual addresses, API symbols or observed behavior. Decompiled output is
|
||||
evidence, not original source. Runtime behavior has not yet been observed.
|
||||
|
||||
- `original-manifest.json`: complete original SHA-256 baseline.
|
||||
- `reconnaissance.md`: input/tool inventory, game identity and initial environment.
|
||||
- `analysis/binary.md`: PE/compiler, entry points, imports and function mappings.
|
||||
- `analysis/graphics.md`: renderer/API, geometry, camera and Vulkan equivalents.
|
||||
- `analysis/assets.md`, `file-formats.md`: recovered OMT/ASE/texture/audio evidence.
|
||||
- `gam-format.md`: fully accounted GAM wire format and exact round-trip validation.
|
||||
- `analysis/gameplay.md`: original controls, startpoint/camera, entities and limits.
|
||||
|
||||
## Function map under investigation
|
||||
|
||||
Provisional mappings from binary agent (disassembly must accompany final report):
|
||||
C3DJimmy update 0x424600, input 0x426030; player input 0x438A60, walking branch
|
||||
0x438BC0. Player defaults ctor region 0x4372A0 sets acceleration 30, deceleration
|
||||
35, maximum speed 600; increments occur per simulation invocation, while movement
|
||||
uses elapsed time. A future simulation must recover update cadence before calling
|
||||
these coefficients timestep-independent. Jump/gravity values remain provisional.
|
||||
|
||||
## Unresolved
|
||||
|
||||
Original execution under isolated Wine/Windows; complete gameplay algorithms;
|
||||
archive animations and render states; save schema; exact historical timing and
|
||||
Windows runtime compatibility. Follow `tasks.md` for concrete next actions.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Tasks
|
||||
|
||||
## Completed
|
||||
- [x] Full specification read; original inventory, hashes and local Git established.
|
||||
- [x] Capabilities/tool reconnaissance; independent investigators with file ownership.
|
||||
- [x] Executable format and engine identification.
|
||||
- [x] Decode/round-trip all 35 GAM files; reader corruption and original-data tests.
|
||||
- [x] Implement C# GAM asset reader (compilation/runtime checks pending).
|
||||
|
||||
## Active
|
||||
- [ ] Complete binary/graphics report with addresses, rendering state and camera.
|
||||
- [ ] Recover player timing, movement, gravity, collision and input algorithms.
|
||||
- [ ] Extract OMT meshes, textures, materials, animations, sound and music robustly.
|
||||
- [ ] Verify asset schema and integration into native Vulkan renderer.
|
||||
- [ ] Compile .NET 10 runtime and verify real offscreen rendering/readback.
|
||||
- [ ] Integrate SDL3 window/input/audio; exercise local display where possible.
|
||||
- [ ] Local Linux and Windows self-contained builds and dependency packaging.
|
||||
- [ ] Test original preservation after tools; checkpoint stable work locally.
|
||||
|
||||
## Required after foundation
|
||||
- [ ] Run original in isolated copied-data Wine prefix or local Windows VM.
|
||||
- [ ] Capture original startup, controls, camera, animation, audio and gameplay.
|
||||
- [ ] Reconstruct LV1B opening sequence and one complete playable scene.
|
||||
- [ ] Implement/verify collision, interactions, trigger/task system and progression.
|
||||
- [ ] Implement original animation playback and correct rendering states.
|
||||
- [ ] Implement menus, inventory, gadgets, combat/AI and all remaining scenes.
|
||||
- [ ] Recover save/load behavior, configuration, audio mixing and transitions.
|
||||
- [ ] Compare reconstruction to original references; resolve fidelity differences.
|
||||
- [ ] Run native Windows tests; distinguish cross-publishing from runtime evidence.
|
||||
- [ ] Produce verified local release packages and accurate known limitations.
|
||||
@@ -0,0 +1,110 @@
|
||||
using System.Buffers.Binary;
|
||||
using System.Numerics;
|
||||
using System.Text;
|
||||
|
||||
namespace GamePort.Assets;
|
||||
|
||||
/// <summary>Original, big endian level entities. See docs/gam-format.md.</summary>
|
||||
public sealed record GamProperty(string Name, uint Type, byte[] Raw)
|
||||
{
|
||||
public string Text => Type == 1 ? Encoding.Latin1.GetString(Raw)
|
||||
: throw new InvalidDataException($"{Name} is not a string");
|
||||
public float Float => Type == 3 && Raw.Length == 4
|
||||
? BitConverter.Int32BitsToSingle(BinaryPrimitives.ReadInt32BigEndian(Raw))
|
||||
: throw new InvalidDataException($"{Name} is not a float32");
|
||||
public int Integer => Type == 6 && Raw.Length == 4
|
||||
? BinaryPrimitives.ReadInt32BigEndian(Raw)
|
||||
: throw new InvalidDataException($"{Name} is not an int32");
|
||||
}
|
||||
|
||||
public sealed record GamEntity(string Kind, int Offset,
|
||||
IReadOnlyDictionary<string, GamProperty> Properties, byte[] Trailer)
|
||||
{
|
||||
public string? Text(string name) => Properties.TryGetValue(name, out var p) ? p.Text : null;
|
||||
public float Number(string name, float fallback = 0) =>
|
||||
Properties.TryGetValue(name, out var p) ? p.Float : fallback;
|
||||
public int Integer(string name, int fallback = 0) =>
|
||||
Properties.TryGetValue(name, out var p) ? p.Integer : fallback;
|
||||
public string? Tag => Text("ObjectTag");
|
||||
public Vector3 Position => new(Number("PositionX"), Number("PositionY"), Number("PositionZ"));
|
||||
public Vector3 Rotation => new(Number("RotationX"), Number("RotationY"), Number("RotationZ"));
|
||||
public Vector3 CameraPosition => new(Number("ViewportPx"), Number("ViewportPy"), Number("ViewportPz"));
|
||||
}
|
||||
|
||||
public sealed record GamLevel(string Code, IReadOnlyList<GamEntity> Entities)
|
||||
{
|
||||
public GamEntity? Player => Entities.FirstOrDefault(e => e.Kind == "3JIM");
|
||||
|
||||
public GamEntity? FindStart(string? tag = null)
|
||||
{
|
||||
tag ??= Player?.Text("StartPoint");
|
||||
return Entities.FirstOrDefault(e => e.Kind == "STRT" &&
|
||||
string.Equals(e.Tag, tag, StringComparison.OrdinalIgnoreCase));
|
||||
}
|
||||
|
||||
public static GamLevel Load(string path) => Parse(File.ReadAllBytes(path));
|
||||
|
||||
public static GamLevel Parse(ReadOnlySpan<byte> data)
|
||||
{
|
||||
var reader = new GamReader(data);
|
||||
var code = reader.Text(4);
|
||||
if (code.Any(c => !char.IsAsciiLetterOrDigit(c)))
|
||||
throw new InvalidDataException("Invalid level FourCC");
|
||||
var count = reader.Count(12);
|
||||
var entities = new List<GamEntity>(count);
|
||||
for (var i = 0; i < count; i++)
|
||||
{
|
||||
var offset = reader.Offset;
|
||||
var kind = reader.Text(4);
|
||||
var properties = new Dictionary<string, GamProperty>(StringComparer.Ordinal);
|
||||
var propertyCount = reader.Count(10);
|
||||
for (var j = 0; j < propertyCount; j++)
|
||||
{
|
||||
var name = reader.Pascal();
|
||||
var type = reader.U32();
|
||||
var length = reader.U32();
|
||||
if (length > int.MaxValue)
|
||||
throw new InvalidDataException($"Property too long at {reader.Offset:x}");
|
||||
if (type == 1 && reader.Byte() != length)
|
||||
throw new InvalidDataException($"String size mismatch at {reader.Offset:x}");
|
||||
if (type is 3 or 6 && length != 4)
|
||||
throw new InvalidDataException($"Scalar size mismatch at {reader.Offset:x}");
|
||||
var property = new GamProperty(name, type, reader.Take((int)length).ToArray());
|
||||
if (type == 3 && !float.IsFinite(property.Float))
|
||||
throw new InvalidDataException($"Non-finite scalar at {reader.Offset:x}");
|
||||
if (!properties.TryAdd(name, property))
|
||||
throw new InvalidDataException($"Duplicate property {name}");
|
||||
}
|
||||
entities.Add(new GamEntity(kind, offset, properties, reader.Take(4).ToArray()));
|
||||
}
|
||||
if (reader.Remaining != 0)
|
||||
throw new InvalidDataException($"Unexpected bytes at {reader.Offset:x}");
|
||||
return new GamLevel(code, entities);
|
||||
}
|
||||
|
||||
private ref struct GamReader(ReadOnlySpan<byte> data)
|
||||
{
|
||||
private readonly ReadOnlySpan<byte> data = data;
|
||||
public int Offset { get; private set; }
|
||||
public int Remaining => data.Length - Offset;
|
||||
public ReadOnlySpan<byte> Take(int length)
|
||||
{
|
||||
if (length < 0 || length > Remaining)
|
||||
throw new InvalidDataException($"Truncated GAM at {Offset:x}");
|
||||
var result = data.Slice(Offset, length);
|
||||
Offset += length;
|
||||
return result;
|
||||
}
|
||||
public byte Byte() => Take(1)[0];
|
||||
public uint U32() => BinaryPrimitives.ReadUInt32BigEndian(Take(4));
|
||||
public string Text(int length) => Encoding.Latin1.GetString(Take(length));
|
||||
public string Pascal() => Text(Byte());
|
||||
public int Count(int minimumItemBytes)
|
||||
{
|
||||
var count = U32();
|
||||
if (count > Remaining / minimumItemBytes)
|
||||
throw new InvalidDataException($"Count exceeds remaining GAM at {Offset:x}");
|
||||
return (int)count;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tools/levels"))
|
||||
import gam
|
||||
|
||||
|
||||
def fixture(properties=b"", count=0):
|
||||
return b"TST1" + struct.pack(">I", 1) + b"3JIM" + struct.pack(">I", count) + properties + b"\x12\x34\x56\x78"
|
||||
|
||||
|
||||
class GamTests(unittest.TestCase):
|
||||
def test_real_levels_roundtrip_and_counts(self):
|
||||
paths = sorted((ROOT.parent / "Original/gam").glob("*.gam"))
|
||||
self.assertEqual(len(paths), 35, "original corpus is required for this integration test")
|
||||
total = 0
|
||||
for path in paths:
|
||||
with self.subTest(path=path.name):
|
||||
raw = path.read_bytes()
|
||||
level = gam.parse(raw)
|
||||
self.assertEqual(gam.encode(level), raw)
|
||||
total += len(level.entities)
|
||||
self.assertEqual(total, 3299)
|
||||
|
||||
def test_exact_original_start(self):
|
||||
level = gam.parse((ROOT.parent / "Original/gam/level1b.gam").read_bytes())
|
||||
self.assertEqual(level.code, "LV1B")
|
||||
player = level.entities[0].values()
|
||||
self.assertEqual(player["StartPoint"], "STARTEXP")
|
||||
self.assertEqual(player["PositionX"], 5071.93310546875)
|
||||
start = next(e.values() for e in level.entities
|
||||
if e.kind == "STRT" and e.values()["ObjectTag"] == "STARTEXP")
|
||||
self.assertEqual(start["StartTrigger"], "LABEXP")
|
||||
self.assertEqual(start["MusicDatabase"], "musicsecretlab.omt")
|
||||
|
||||
def test_endianness_and_string_prefix(self):
|
||||
props = b"\x01x" + struct.pack(">II f", 3, 4, -12.5)
|
||||
props += b"\x01s" + struct.pack(">II", 1, 3) + b"\x03abc"
|
||||
raw = fixture(props, 2)
|
||||
level = gam.parse(raw)
|
||||
self.assertEqual(level.entities[0].values(), {"x": -12.5, "s": "abc"})
|
||||
self.assertEqual(gam.encode(level), raw)
|
||||
|
||||
def test_unknown_type_preserved(self):
|
||||
raw = fixture(b"\x01x" + struct.pack(">II", 999, 3) + b"abc", 1)
|
||||
self.assertEqual(gam.encode(gam.parse(raw)), raw)
|
||||
|
||||
def test_all_truncations_rejected(self):
|
||||
raw = fixture(b"\x01x" + struct.pack(">II f", 3, 4, 12.5), 1)
|
||||
for end in range(len(raw)):
|
||||
with self.subTest(end=end), self.assertRaises(gam.FormatError):
|
||||
gam.parse(raw[:end])
|
||||
|
||||
def test_overflows_trailing_data_and_bad_strings(self):
|
||||
for data in [b"TST1\xff\xff\xff\xff", fixture() + b"x",
|
||||
fixture(b"\x01x" + struct.pack(">II", 1, 2) + b"\x03abc", 1),
|
||||
fixture(b"\x01x" + struct.pack(">II", 3, 3) + b"abc", 1)]:
|
||||
with self.subTest(data=data), self.assertRaises(gam.FormatError):
|
||||
gam.parse(data)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,151 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Lossless, bounded reader for the supplied Neutron GAM entity files.
|
||||
|
||||
No original file is modified. Unknown property types stay opaque. The four bytes
|
||||
after each object are retained without assigning an unverified interpretation.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import struct
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class FormatError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class Reader:
|
||||
def __init__(self, data: bytes):
|
||||
self.data = data
|
||||
self.offset = 0
|
||||
|
||||
def take(self, count: int) -> bytes:
|
||||
if count < 0 or count > len(self.data) - self.offset:
|
||||
raise FormatError(f"truncated data at 0x{self.offset:x}: need {count} bytes")
|
||||
result = self.data[self.offset:self.offset + count]
|
||||
self.offset += count
|
||||
return result
|
||||
|
||||
def u32(self) -> int:
|
||||
return struct.unpack(">I", self.take(4))[0]
|
||||
|
||||
def text(self, count: int) -> str:
|
||||
return self.take(count).decode("latin-1")
|
||||
|
||||
def pascal(self) -> str:
|
||||
return self.text(self.take(1)[0])
|
||||
|
||||
|
||||
@dataclass
|
||||
class Property:
|
||||
name: str
|
||||
type: int
|
||||
length: int
|
||||
value: str | int | float
|
||||
raw: str
|
||||
offset: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class Entity:
|
||||
kind: str
|
||||
offset: int
|
||||
properties: list[Property]
|
||||
trailer: str
|
||||
|
||||
def values(self) -> dict:
|
||||
return {p.name: p.value for p in self.properties}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Level:
|
||||
code: str
|
||||
entities: list[Entity]
|
||||
size: int
|
||||
|
||||
|
||||
def parse(data: bytes) -> Level:
|
||||
r = Reader(data)
|
||||
code = r.text(4)
|
||||
if not code.isascii() or not code.isalnum():
|
||||
raise FormatError("invalid level FourCC")
|
||||
count = r.u32()
|
||||
if count > (len(data) - 8) // 12:
|
||||
raise FormatError("object count exceeds file capacity")
|
||||
entities = []
|
||||
for _ in range(count):
|
||||
offset = r.offset
|
||||
kind, property_count = r.text(4), r.u32()
|
||||
if property_count > (len(data) - r.offset) // 10:
|
||||
raise FormatError(f"property count exceeds capacity at 0x{offset:x}")
|
||||
properties = []
|
||||
names = set()
|
||||
for _ in range(property_count):
|
||||
property_offset = r.offset
|
||||
name, type_id, length = r.pascal(), r.u32(), r.u32()
|
||||
if name in names:
|
||||
raise FormatError(f"duplicate property {name!r} at 0x{property_offset:x}")
|
||||
names.add(name)
|
||||
if type_id == 1:
|
||||
actual = r.take(1)[0]
|
||||
if actual != length:
|
||||
raise FormatError(f"string size mismatch at 0x{property_offset:x}")
|
||||
raw = r.take(length)
|
||||
if type_id == 1:
|
||||
value = raw.decode("latin-1")
|
||||
elif type_id in (3, 6):
|
||||
if length != 4:
|
||||
raise FormatError(f"scalar size {length} at 0x{property_offset:x}")
|
||||
value = struct.unpack(">f" if type_id == 3 else ">i", raw)[0]
|
||||
if isinstance(value, float) and not math.isfinite(value):
|
||||
raise FormatError(f"non-finite scalar at 0x{property_offset:x}")
|
||||
else:
|
||||
value = raw.hex()
|
||||
properties.append(Property(name, type_id, length, value, raw.hex(), property_offset))
|
||||
entities.append(Entity(kind, offset, properties, r.take(4).hex()))
|
||||
if r.offset != len(data):
|
||||
raise FormatError(f"{len(data) - r.offset} unexpected trailing bytes at 0x{r.offset:x}")
|
||||
return Level(code, entities, len(data))
|
||||
|
||||
|
||||
def encode(level: Level) -> bytes:
|
||||
"""Re-encode parsed data to prove every byte has been accounted for."""
|
||||
out = bytearray(level.code.encode("latin-1") + struct.pack(">I", len(level.entities)))
|
||||
for e in level.entities:
|
||||
out += e.kind.encode("latin-1") + struct.pack(">I", len(e.properties))
|
||||
for p in e.properties:
|
||||
name = p.name.encode("latin-1")
|
||||
raw = bytes.fromhex(p.raw)
|
||||
out += bytes([len(name)]) + name + struct.pack(">II", p.type, p.length)
|
||||
if p.type == 1:
|
||||
out += bytes([p.length])
|
||||
out += raw
|
||||
out += bytes.fromhex(e.trailer)
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
cli = argparse.ArgumentParser(description=__doc__)
|
||||
cli.add_argument("input", type=Path)
|
||||
cli.add_argument("--output", type=Path)
|
||||
args = cli.parse_args()
|
||||
raw = args.input.read_bytes()
|
||||
level = parse(raw)
|
||||
if encode(level) != raw:
|
||||
raise FormatError("round trip differs")
|
||||
text = json.dumps(asdict(level), indent=2, ensure_ascii=True, allow_nan=False) + "\n"
|
||||
if args.output:
|
||||
if args.output.resolve() == args.input.resolve():
|
||||
cli.error("output must differ from original input")
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(text)
|
||||
else:
|
||||
print(text, end="")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,32 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Inventory or verify originals with SHA-256; never writes to the source tree."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def inventory(root):
|
||||
return [{"path": p.relative_to(root).as_posix(), "size": p.stat().st_size,
|
||||
"sha256": hashlib.sha256(p.read_bytes()).hexdigest(),
|
||||
"header": p.read_bytes()[:16].hex()}
|
||||
for p in sorted(root.rglob("*")) if p.is_file()]
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("root", type=Path)
|
||||
parser.add_argument("--verify", type=Path)
|
||||
args = parser.parse_args()
|
||||
files = inventory(args.root)
|
||||
if args.verify:
|
||||
previous = json.loads(args.verify.read_text())["files"]
|
||||
if files != previous:
|
||||
raise SystemExit("Original inventory differs from recorded baseline")
|
||||
print(f"Verified {len(files)} unchanged original files")
|
||||
else:
|
||||
print(json.dumps({"root": str(args.root), "files": files}, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in new issue
Block a user