Source code

MapleBench

A benchmark for AI agents playing MapleStory

CodeExplore results
Leafre · recorded gameplay

Recorded attempts appear here once their evidence is verified.

Latest verified recordings, shown chronologically. Select one to inspect the full replay.

Agents write code to earn XP that survives logout.

Waiting for results

Simulation labInside the distributed setup

One game world per worker.

4 vCPU / 16 GB each

Swipe to follow the connections

Agent runtime and game environment on each simulation worker The illustrated worker pool contains separate virtual machines; the labels A, B and C do not report a live worker count. Each VM owns one Cosmic game world, its MySQL database, Journey WASM browser and local asset service, WebSocket-to-TCP proxy, host controller, program sandbox and verifier. Operator-managed dispatch assigns frozen comparison plans to qualified workers. The fan-out shows those assignments, not an automated fleet scheduler. Inside each worker, the worker runner executes the assigned plan in order, restores the baseline, drives each trial and collects verified evidence. This node groups local plan execution and the durable trial runner; it does not assign work to other workers. It enforces group limits and stops on failure. Exclusive world, queue and runner locks permit one active trial per configured worker world, including verification and cleanup. Separate qualified worker VMs can execute independently. The model inference provider is remote and shared; the current implementation uses the OpenAI API. Within each VM, Agent runtime groups the trusted MapleBench controller and the isolated model-written program. Game environment groups the Journey WASM frontend, the WebSocket-to-TCP proxy, the Cosmic Java backend and MySQL persistence. These group boxes describe software responsibilities, not additional machines or security boundaries. The worker runner and outcome collection belong to the evaluation harness, outside the agent runtime group. The trusted controller has network access and holds API credentials outside the Docker sandbox. Only model-written JavaScript runs in that networkless sandbox; validated SDK requests use stdin/stdout pipes. Cosmic saves character state in the worker-local MySQL database. This database is reused across trials with the frozen baseline restored before each trial. The controller writes run metrics, model-call evidence, SDK receipts and captured video to private files keyed by attempt ID. The trial runner collects these with initial and final database exports and native save receipts into a separate evidence bundle for every attempt. Verification checks the saved outcome and artifact bindings. A separate serialized publication process rechecks the evidence and exports allowlisted results.json summaries, recording manifests and videos to the static site on Vercel. Raw prompts, generated programs and private logs remain private. Each model in a comparison group uses the same declared worker and restored character baseline; this does not make live combat deterministic. Worker C · separate VM Worker B · separate VM Worker A · one VM 1 active trial per worker Independent world, database & run state Agent runtime Game environment Operator-managed dispatchExperiment coordinationFrozen plans → assigned workers External serviceModel inference providerOpenAI API Local plan executionWorker runnerReset · run · verify · collect evidence Task + state / code Trusted host process · PythonMapleBench controllerSDK relay, budgets & receiptsNetwork access · host-held credentials Frontend · browser clientJourney WASMClient physics, controls & renderingRead-only assets · gameplay recording Game state / key inputs Validated SDK calls stdin / stdout Model-written JavaScript onlyNode.js + DockerNo network · no credentials WebSocket ↔ TCP proxy Backend · Java game serverCosmic · MapleStory v83Combat, monsters & character state Save on logout Persistence · local to workerMySQLBaseline restored before each trial Initial / final snapshots Metrics / video Collected & verified by the runnerPer-attempt evidence filesMetrics, receipts & recording Export checked artifacts Separate publication processVerify & publishAllowlisted export · SHA-256 manifest Public evidence package Static site · VercelResults & recordingsresults.json + manifests + WebM
Historical single-request protocol. Three workers provisioned; all retired. 12 newer cloud pilots.

What the agent sees

Position, HP/MP, XP, level, map, alive status, and nearby monsters. The model receives structured state; the recording is for us.

What it controls

A program calls observe(), pressKeys(), and wait() in Node.js / Docker. Normal game inputs only; no stat edits.

An agent’s program Illustrative

const state = await sdk.observe();
if (state.character.hp < state.character.maxHp / 2) {
  await sdk.pressKeys(['HP_POTION'], 100);
}
await sdk.pressKeys(['LEFT'], 300);
await sdk.pressKeys(['BRANDISH'], 650);

Each worker owns its world, database, baseline, credentials, and locks. Trials run serially per worker; cohorts stay on one host. Provisioning alone does not establish qualification. An external controller verifies worker deletion at lease expiry.

Results

Same character, knowledge, controls and time budget. Every planned attempt stays visible.

Pilot · unranked

Five-minute adaptive runs count model wait inside the controller budget. The score is signed session net XP, verified after ordinary logout and including its bounded tail. Watch the recording to inspect model calls and inputs. Earlier protocols remain separate.

Model × declared class / task

Leaderboard

Mean saved XP per model
Saved XP by model
#ModelVerifiedMeanMedianBest

A protocol-declared cohort is required for the research matrix. Existing run evidence remains below.

Every planned attempt remains in the denominator. Valid zero, negative, death and no-input outcomes are retained; missing evidence has no score. Sample spread is descriptive. This small pilot does not establish an overall model ranking, and initial/final XP does not establish a peak XP rate.

Recordings & run evidence

No active attempt

The next run starts here.

This page shows actual full-client attempts as their evidence arrives.

    API model returned
    —
    Input actions
    —
    Live XP change · diagnostic
    —
    Last observed state
    —

    Results by frozen inputs

    Every group is shown. Matching inputs do not prove identical live monster scenes.

    Unranked

    Verified persisted results appear here after normal logout.

    Persisted XP includes penalties and is measured after normal logout. Publication evidence is checked separately; a blocked check does not erase the verified score. All displayed runs remain unranked.

    Every visible attempt

    Failures, recoveries, and unscored integration runs stay in the history.

    0 attempts
    Attempt / exact modelStatusPersisted XPDiagnostic XPInput actionsPublication evidenceRecording

    No attempts have been exported yet.

    Methodology

    What the agent sees

    Structured game state and a frozen knowledge pack. It writes programs that observe the world and press ordinary game keys. Recordings show the run for human inspection.

    What the score means

    XP saved after ordinary logout minus starting XP, including penalties and the bounded logout tail. The five-minute controller budget includes model wait. Missing evidence stays unscored.

    What this pilot can tell us

    Results describe these declared fixtures and model settings. Live monster scenes vary. Four samples per model describe observed spread; they do not establish statistical significance or an overall ranking.

    Earlier protocols stay separate. Input delivery, native skill qualification and persisted score are distinct evidence checks; a key press alone does not prove a successful cast or hit.

    Read the full protocol and limitations

    Recorded attempt