English | Deutsch

All Lane Arena reports · Pilot edition

opencode 1.19 vs. opencode 2: same model, different harness

The lane, the locally hosted model from the pilot edition, runs inside opencode. opencode 2 has been in development for weeks. Before switching, we measured whether the new version behaves the same way, whether it is faster and whether the results get better or worse. The model, hardware and server stayed exactly the same; only the agent harness changed.

This is not an arena edition: there are no competing participants, no blind review and no hidden tests. It is a harness comparison with two parts: a micro benchmark on toy tasks and the replay of one real ticket.

1. Results at a glance

2. Setup

Micro benchmark

Four scenarios, 3 repetitions each, alternating v1 and v2 (24 runs). Each run starts in a fresh copy of a tiny git repository with a sibling folder outside/ next to it.

Real ticket

The real ticket is a replay: a ticket that the lane had already solved in production on 1.19.0 (first implementation round 605 s), run again with the verbatim original brief on the same base commit, one fresh clone per arm, one run each. The production run gives a third data point for 1.19 next to the replay. The ticket (backend Hono/Prisma, frontend Next.js, docs): a dashboard summary should mark a connected service whose stored credential can no longer be decrypted as "unreadable", instead of failing as a whole or showing "Not configured". The brief demanded new tests, doc updates and a mutation check for each new branch (break the branch, the new test must fail).

In the original production run, the review had found one medium-severity gap: the skip of downstream calls was only tested for some services, the deployment service lines had surviving mutants.

3. Behaviour: what opencode 2 changes

The permission configuration we used for v2 on the real ticket, passed as OPENCODE_CONFIG_CONTENT together with --standalone (later rules win, so path-specific allow rules can follow the ask). The micro benchmark ran with the first rule only; the question rule was added after the micro benchmark showed the question call:

{"permissions": [
  {"action": "external_directory", "resource": "*", "effect": "ask"},
  {"action": "question", "resource": "*", "effect": "deny"}
]}

The source references (default rules, file access, AGENTS.md discovery in packages/core/src/config/plugin/instruction.ts, shell directory scanning in packages/core/src/shell/parse.ts) were read on the upstream development branch, not exactly at 2.0.25. Every row above except the source reading was also observed in runs.

This configuration does not restore the 1.19 boundary completely: a shell command that reads a path outside the project through its arguments runs without a permission request in v2 (tested with cat; by the source, writes through arguments are not checked either). The micro benchmark did not catch this, because its shell scenario changed directory first (cd ../..), which v2 does check. The upstream security policy states that the permission system is a UX feature, not a sandbox, and recommends a container or VM for real isolation.

4. Speed

Micro benchmark (median of 3)

• Harness overhead (wall time minus model time) is small in both: 0.7 s (v1) and 0.5 s (v2).
• The gap is almost entirely the first prompt of a session: v1 sends about 11.9k tokens that miss the server's prompt cache every time. v2 sends a smaller system prompt (7.8k on its very first run). After that, in an ok run (two steps), about 15.2k of its 15.7k prompt tokens come from the cache, although the working directory changes per run. Why v1 misses the cache is not verified.
• edit: tests green in 3 of 3 runs in both versions, only the two intended files changed. v2 needed 5 steps instead of 6 to 8 and about half the prompt tokens.

Real ticket (one run each)

The 1.19 replay came within 2 % of the original production run (596 s vs. 605 s), so on 1.19 the time for this ticket reproduces; v2's 1,040 s lies well outside that. Only the wall time of the production run is compared here; it ran in a regular worktree, where 1.19 loads no extra AGENTS.md either.

Per step the two versions are equally fast, and inside one long session both use the prompt cache well. v2 was slower because the model did more under v2, not because the harness is slower. The extra time is spread out, with no single hotspot. Each step is counted by its first tool call; shell commands that run tests, lint, type checks, builds or code generation count as "tests, lint, build":

What v2 did in addition: more tests, two failed edits (the search string did not match), repeated npm ci after parallel installs left a partial tree, reading a tool's --help twice, a production build and a throwaway end-to-end check against the real downstream module. The micro-benchmark saving of about 15 s per session is about 2.5 % of a 600 s ticket and disappears in that extra work.

5. Result quality on the real ticket

• Both versions changed the same 9 files. Backend 270 of 270 tests green; frontend 30 of 30 (v1) and 33 of 33 (v2).
• Both reported the three required mutation checks as killed. Our own probes on the skip for the deployment service lines found 2 surviving mutants in both versions, exactly the gap of the original production run. Neither version extended the tests to this symmetric case on its own; both tested what the brief named literally.
• v1 removed a guard that only checks readability for configured services. Unconfigured services now trigger an extra credential lookup and a downstream call, and the summary reports an error instead of null for them. The UI is unaffected, the change is untested and not purely additive. The original production run on the same version did not do this: run-to-run variance, not a version trait.
• v2 kept the response contract, added a test for one more service and an extra mutation check, and ran the build and an end-to-end check.

Reading: v2 was more thorough here and paid for it with 1.75 times the time, 2.4 times the prompt tokens and therefore 1.75 times the server slot occupancy. With one run per version and visible variance within 1.19 itself, "v2 is more thorough" cannot be separated from chance.

6. Limits

• One run per version on the real ticket, and not alternating: v2 ran first, then v1 (a loop bug in the benchmark script ran only v2 on the first invocation; the data was relabelled, nothing was lost). With the production run, 1.19 has two time measurements for this ticket, v2 only one.
• The micro scenarios repeat identical prompts, which favours prompt caching. A busy server or changing briefs were not measured.
• The cause of v1's cache misses on the first prompt is a hypothesis (a per-run value placed early in the system prompt), not verified by a request capture.
• Reasoning tokens are reported as 0 by both versions although the model thinks (as in the pilot). Output tokens therefore do not separate reasoning.
• One build of 2.x (2.0.25). The development branch moves quickly; 1.19 has since received patch releases that were not re-measured.
• Both arms ran in clones outside the home directory. In the production layout (checkouts below the workspace folder) v2 would additionally have loaded the workspace AGENTS.md.

7. Conclusion

1. opencode 2 is not a drop-in replacement for an unattended lane. Its default permissions allow everything, run without --standalone ignores permission overrides, AGENTS.md discovery reaches beyond the repository, the question tool is available in unattended runs, and shell commands can reach paths outside the project through their arguments even with an explicit ask rule. Each of these changes the lane without any visible error.
2. The silent abort is fixed, but the exit code stays 0. A wrapper that fails a run without a final report is needed in both versions.
3. Faster on toy tasks, slower on real work. The toy-task gain is a fixed prompt-cache saving per session. On a real ticket the number of steps dominates, and the model took more of them under v2.
4. The brief is the more likely quality lever than the harness. Both versions skipped the same symmetric test case. Naming symmetric branches explicitly in the brief is the next thing to test; this comparison did not try it.

We stay on 1.19 and repeat the real-ticket comparison with several runs per version once 2.x is the regular npm release, with --standalone, the permission rules above, checkouts outside the home directory and the run wrapper's report check. Since the permission rules do not cover shell arguments in v2, a lane on v2 would need a container or VM as its actual boundary.

Updated after publication (October 2026): added the shell path-argument row and its consequence, and the documentation status of the default permissions and the AGENTS.md walk. A feature request for a configurable discovery boundary is upstream as anomalyco/opencode#55999.