longitudinal agentic coding benchmark via real reverse-engineering project
Analyzing Frontier Model Progress with My Favourite Game: Prince of Persia (external)
A controlled, no-human-edits coding experiment where the same reverse-engineering task is handed to each new frontier model, giving you a concrete read on what changed between Opus 4.6, Codex, Opus 5, and Opus 5.5: the jump to agentic self-testing against a running reference turned "polishes the surface" into pixel-exact.
- Task: port Jordan Mechner's 1989 Prince of Persia Apple II 6502 assembly (published 2012) to C#; the author only prompts and plays, never reads or edits the code.
- Opus 4.6 (Feb 2026) and OpenAI Codex (Mar 2026) both failed on architecture: they built a tile-grid, one-tile-per-move engine instead of the original's per-frame animation sequences; Codex made cosmetic fixes but never ran the game.
- Opus 5 (Sep 2026) was given two Claude Code skills (drive DOSBox and any Windows program via launch/keystrokes/screenshots), ran autonomously overnight, rebuilt the engine around frame sequences, and extracted animation tables from PRINCE.EXE by byte-pattern matching the Apple II source, confirming each table against a second known value. No copyrighted data entered the repo; the C# reads it live from the user's own DOS install.
Full summary
A developer ran a longitudinal experiment: hand the published Apple II 6502 source of Prince of Persia to each new frontier model and ask for a C# port, editing nothing himself and only playing the result. Claude Opus 4.6 and OpenAI Codex both chose a tile-grid engine rather than the game's frame-sequence animation model and could not recover. Claude Opus 5, equipped with author-written Claude Code skills to drive and screenshot DOSBox, ran overnight, diagnosed the architecture error, rebuilt the engine, and pulled real animation and level data out of PRINCE.EXE by pattern-matching the Apple II source, validating each table against a second known value. Claude Opus 5.5, from a single prompt, found the executable was EXEPACK-compressed, wrote an unpacker, and ported SDLPoP's reverse-engineered room-drawing routine (seg008.c), cutting the pixel difference on level 1's first screen from 8,429 to 2. The author and the model both credit the decisive leap to agentic tooling that let the model see and test against the real game, plus the prior reverse-engineering in SDLPoP, rather than raw model intelligence alone. The project is GPL-3.0-or-later, ships no copyrighted game data, and still lacks guards, sword fighting, and palace levels.