TokenSpendEvals

Evaluation report

Stagehand Model Scorecard

We tested 8 models and 7 coding-agent harnesses on 12 merged pull requests from browserbase/stagehand, grading each attempt with the pull request's own tests.

12 tasks · 8 models · 7 harnesses · 492 graded attempts

Summary

We replayed 12 pull requests merged into browserbase/stagehand between 2026-06-10 and 2026-09-27 as coding tasks. Each of 8 models ran in TokenSpend's reference harness, and 6 of them also ran through 6 other agent harnesses. A task counts as resolved only when the pull request's own tests pass on the model's change. Costs come from the token counts the router metered for each request.

  1. The harness mattered most for open models. GLM 5.3 resolved between 3 and 11 of 12 tasks, depending on which one it ran in.
  2. In every harness, Opus 5.5 resolved 11 or 12 tasks and GPT-6 Astra resolved 12. Their cost per resolved task still changed by up to 2× between harnesses.
  3. In the reference harness, Opus 5.5 resolved 36 of 36 attempts at $0.184 per resolved task.
  4. DeepSeek V4.1 Flash in tokenspend-mini has the highest arena rating (1707). The arena counts a win for the cheaper setup when both solve a task.
  5. DeepSeek V4.1 Flash in tokenspend-mini was the cheapest setup to resolve at least 80% of tasks. It resolved 12 of 12 on first attempts, at $0.021 per resolved task.

Results by model

Every model ran in the same reference harness (tokenspend-mini@1), with five tools, one prompt, and 60 turns and 30 minutes per attempt.

ModelResolvedPass rateCost per resolved taskMedian timeMedian turnsWhy attempts failed
Opus 5.536 of 36100% ± 0$0.18464s10none
GPT-6 Astra36 of 36100% ± 0$0.4952.3 min12none
DeepSeek V4.1 Flash35 of 3697% ± 3$0.0213.8 min47Tests failed 1
Fable 5.112 of 12100%$1.123.4 min16none
GPT-6 Sol11 of 1292%$0.1502.4 min20Tests failed 1
Grok 4.711 of 1292%$0.9983.0 min29Tests failed 1
Kimi K36 of 1250%$1.7110.1 min45Tests failed 3, Time limit 2, Turn limit 1
GLM 5.35 of 1242%$1.4612.9 min60Time limit 3, Broke the build 3, Turn limit 1

Table 1. Results by model, reference harness.

Running out of turns or time counts as a failure. Cost per resolved task includes all requests from all attempts, including retries. Opus 5.5, GPT-6 Astra and DeepSeek V4.1 Flash ran 3 times; ± is the standard error across runs.

$0.01$0.1$1$10 0255075100 Cost per resolved task (US$, log scale) Pass rate (%) Opus 5.5GPT-6 AstraDeepSeek V4.1 FlashFable 5.1GPT-6 SolGrok 4.7Kimi K3GLM 5.3
Figure 1. Pass rate against cost per resolved task, reference harness. Up and to the left is better. Hover a point for details. Source: TokenSpend router receipts; graded by each pull request's own tests.
Model#2213#2220#2306#2335#2347#2361#2542#2752#2787#2799#3034#3052
Opus 5.53/33/33/33/33/33/33/33/33/33/33/33/3
GPT-6 Astra3/33/33/33/33/33/33/33/33/33/33/33/3
DeepSeek V4.1 Flash3/33/33/33/33/33/33/33/33/32/33/33/3
Fable 5.1✓✓✓✓✓✓✓✓✓✓✓✓
GPT-6 Sol✓✓✓✓✓✓✓✓✓–✓✓
Grok 4.7✓✓✓✓✓✓✓✓✓–✓✓
Kimi K3✓✓✓✓–––✓✓–––
GLM 5.3✓––✓–✓✓✓––––

Table 2. Results per task, reference harness. ✓ resolved, – not resolved. For models that ran more than once, the cell shows runs resolved. Hover a cell for the reason.

Results by harness

We ran the same 12 tasks through other coding-agent harnesses on 6 of the models. Each harness uses its own prompt, tools and loop. Containers, limits, router and grader match the reference runs.

HarnessOpus 5.5GPT-6 AstraDeepSeek V4.1 FlashGrok 4.7Kimi K3GLM 5.3
tokenspend-miniv112/12$0.17812/12$0.48512/12$0.02111/12$0.9986/12$1.715/12$1.46
mini-swe-agent2.4.611/12$0.23812/12$0.73610/12$0.0318/12$1.888/12$1.725/12$1.32
Claude Code2.1.28612/12$0.242–10/12$0.02510/12$1.5010/12$1.3010/12$0.756
Codex0.159.3–12/12$0.559––––
Pi0.99.212/12$0.16812/12$0.5708/12$0.03410/12$1.3210/12$0.86211/12$0.658
Hermesmain-612d8e412/12$0.33812/12$1.149/12$0.0478/12$3.5810/12$1.183/12$2.70
Cursor2026.09.28-64d204312/12$0.365*––10/12$0.911*11/12$0.978*–

Table 3. Tasks resolved and cost per resolved task, by harness and model.

Each cell shows tasks resolved out of 12, then cost per resolved task, from first attempts. The reference-harness row can differ from Table 1, which counts every run. The best harness for each model (most tasks resolved, then lowest cost) is shaded and bold. – means not run. * Estimated from Cursor's token counts at its published prices.

tokenspend-mini, the reference harness other harnesses

GLM 5.3 3 to 11 of 12
036912 5/12tokenspend-mini5/12mini-swe-agent10/12Claude Code11/12Pi3/12Hermes
Kimi K3 6 to 11 of 12
036912 6/12tokenspend-mini8/12mini-swe-agent10/12Claude Code10/12Pi10/12Hermes11/12Cursor
DeepSeek V4.1 Flash 8 to 12 of 12
036912 12/12tokenspend-mini10/12mini-swe-agent10/12Claude Code8/12Pi9/12Hermes
Grok 4.7 8 to 11 of 12
036912 11/12tokenspend-mini8/12mini-swe-agent10/12Claude Code10/12Pi8/12Hermes10/12Cursor
Opus 5.5 11 to 12 of 12
036912 12/12tokenspend-mini11/12mini-swe-agent12/12Claude Code12/12Pi12/12Hermes12/12Cursor
GPT-6 Astra 12 of 12 in every harness
036912 12/12tokenspend-mini12/12mini-swe-agent12/12Codex12/12Pi12/12Hermes
Figure 2. Tasks resolved in each harness, per model, with the widest spread first. Hover a bar to see that harness in every panel. Source: TokenSpend router receipts; first attempts only.

tokenspend-mini other harnesses estimated cost

GLM 5.3 4× from cheapest to priciest
0$2.5$5 $1.46tokenspend-mini$1.32mini-swe-agent$0.756Claude Code$0.658Pi$2.70Hermes
Kimi K3 2× from cheapest to priciest
0$1$2 $1.71tokenspend-mini$1.72mini-swe-agent$1.30Claude Code$0.862Pi$1.18Hermes$0.978Cursor
DeepSeek V4.1 Flash 2× from cheapest to priciest
0$0.05$0.1 $0.021tokenspend-mini$0.031mini-swe-agent$0.025Claude Code$0.034Pi$0.047Hermes
Grok 4.7 4× from cheapest to priciest
0$2.5$5 $0.998tokenspend-mini$1.88mini-swe-agent$1.50Claude Code$1.32Pi$3.58Hermes$0.911Cursor
Opus 5.5 2× from cheapest to priciest
0$0.25$0.5 $0.178tokenspend-mini$0.238mini-swe-agent$0.242Claude Code$0.168Pi$0.338Hermes$0.365Cursor
GPT-6 Astra 2× from cheapest to priciest
0$1$2 $0.485tokenspend-mini$0.736mini-swe-agent$0.559Codex$0.570Pi$1.14Hermes
Figure 3. Cost per resolved task by each harness, per model. Each panel has its own scale. Source: TokenSpend router receipts; Cursor estimated at its published prices.
HarnessResolvedCost per resolved taskMedian turnsMedian timePrompt read from cache
Opus 5.5
Pi 0.99.212 of 12$0.1681061s89%
tokenspend-mini v112 of 12$0.1781165s87%
Claude Code 2.1.28612 of 12$0.242878s93%
Hermes main-612d8e412 of 12$0.338141.6 min94%
Cursor 2026.09.28-64d204312 of 12$0.365*n/a2.1 min93%
mini-swe-agent 2.4.611 of 12$0.238989s89%
GPT-6 Astra
tokenspend-mini v112 of 12$0.485122.0 min84%
Codex 0.159.312 of 12$0.5591166s90%
Pi 0.99.212 of 12$0.570121.6 min89%
mini-swe-agent 2.4.612 of 12$0.736121.6 min87%
Hermes main-612d8e412 of 12$1.14122.3 min88%
Kimi K3
Cursor 2026.09.28-64d204311 of 12$0.978*n/a4.7 min95%
Pi 0.99.210 of 12$0.862356.5 min95%
Hermes main-612d8e410 of 12$1.18396.4 min97%
Claude Code 2.1.28610 of 12$1.303310.4 min96%
mini-swe-agent 2.4.68 of 12$1.725014.9 min96%
tokenspend-mini v16 of 12$1.714510.1 min93%
Grok 4.7
tokenspend-mini v111 of 12$0.998293.0 min84%
Cursor 2026.09.28-64d204310 of 12$0.911*n/a5.1 min85%
Pi 0.99.210 of 12$1.32375.4 min82%
Claude Code 2.1.28610 of 12$1.50263.4 min88%
mini-swe-agent 2.4.68 of 12$1.88284.5 min86%
Hermes main-612d8e48 of 12$3.58345.6 min91%
GLM 5.3
Pi 0.99.211 of 12$0.6585211.9 min97%
Claude Code 2.1.28610 of 12$0.756568.2 min97%
mini-swe-agent 2.4.65 of 12$1.325713.2 min95%
tokenspend-mini v15 of 12$1.466012.9 min97%
Hermes main-612d8e43 of 12$2.702930.0 min96%
DeepSeek V4.1 Flash
tokenspend-mini v112 of 12$0.021453.6 min96%
Claude Code 2.1.28610 of 12$0.025302.7 min97%
mini-swe-agent 2.4.610 of 12$0.031384.4 min95%
Hermes main-612d8e49 of 12$0.047455.2 min97%
Pi 0.99.28 of 12$0.034326.5 min96%

Table 4. Results per harness and model.

* Estimated from Cursor's token counts at its published prices; Cursor reports no turn count.

Notes

  1. Only first attempts count in this section.
  2. mini-swe-agent comes from the team behind SWE-bench.
  3. Codex ran its GPT-6 Astra profile in the standard Responses format and edited files through its shell tool.
  4. Pi does not retry a failed model request. Attempts cut off by a provider error were run again.
  5. Cursor runs model calls on its own servers, so its costs are estimates, made from its token counts at Cursor's published prices. It reports no turn count. Opus 5.5 ran as Cursor's medium-effort 300K-context variant and Kimi K3 as its max-effort variant. Cursor does not offer GLM 5.3, DeepSeek V4.1 Flash or GPT-6.
  6. Claude Code ran on non-Claude models after a router fix that keeps its turns separate and its prompt cache working.
  7. Hermes asks for medium reasoning effort with no output limit. GLM 5.3 then reasoned past 8,000 tokens on about one request in seven, and most of those attempts hit the time limit.
  8. At most 5 attempts ran at once, on one 8-core machine. Attempts that hit the time limit while the machine was overloaded were run again.

Overall scores

Several setups solved every task, so pass rate alone cannot rank them. We add two scores. The difficulty-adjusted score (item response theory, the method behind the Epoch Capabilities Index) puts models, harnesses and tasks on one scale and separates the model's effect from the harness's effect. The arena rating (Bradley-Terry, the method behind LMArena) compares setups two at a time on each task. Solving beats failing, and when both solve a task the cheaper one wins.

ModelScore90% rangeNote
GPT-6 Astra1909+1834 to 1972Lower bound, since it never missed a task.
Opus 5.518031690 to 1964
Fable 5.11615+1571 to 1646Lower bound, since it never missed a task.
GPT-6 Sol14431248 to 1632
DeepSeek V4.1 Flash13951275 to 1573
Grok 4.712821109 to 1487
Kimi K312461141 to 1379
GLM 5.31059953 to 1174

Table 5. Model ability across all harnesses, difficulty-adjusted.

The average task scores 1000. A score 200 points above a task's is about a 76% chance of solving it. Ranges come from refitting on resampled tasks.

HarnessPointsA task tokenspend-mini solves half the time90% range
Cursor+11766%−38 to +251
Claude Code+10765%−14 to +230
Pi+8362%−56 to +220
CodexNot measurable: every attempt was resolved
tokenspend-mini reference050%
Hermes−10835%−214 to −17
mini-swe-agent−10835%−206 to −3

Table 6. What each harness adds, against tokenspend-mini.

This assumes a harness helps every model equally. Table 3 shows pairs where it does not, such as Hermes on GLM 5.3.

TaskTitleDifficultySolved
#2799fix(cli): resolve template sources and align setup instructions138224 of 41
#2306fix(core): declare CUA screenshot media type at capture boundary (#2300)135725 of 41
#3034feat: add internal PDF capture to the extension115832 of 41
#2347feat(core): support OpenAI Chat Completions via openaiEndpointFormat108634 of 41
#2542fix: verify cached local Chrome debug port before trusting it100236 of 41
#2220STG-2196 Shadow DOM fix95237 of 41
#2361feat: warn and add opt-out flag for browse CLI .env auto-loading89538 of 41
#3052fix(extension): read text() from direct child text nodes (#2728)89538 of 41
#2335[STG-2511] feat: add browse skills show and agent-facing --help header89538 of 41
#2787ci: publish commit-addressed alpha releases of the Python SDK89538 of 41
#2752Allow SDKs to reattach to Stagehand runtimes74140 of 41
#2213[STG-2183] fix(cli): list full template catalog (send scope=all)74140 of 41

Table 7. Task difficulty, on the same scale as Table 5. A task every setup solved cannot tell setups apart. Solved counts every graded attempt on the task, repeat runs included.

RankSetupRating90% rangeWonLostDrawn
1DeepSeek V4.1 Flash in tokenspend-mini17071675 to 1904393150
2DeepSeek V4.1 Flash in Claude Code14231204 to 1936328601
3DeepSeek V4.1 Flash in mini-swe-agent14061232 to 1731321651
4Opus 5.5 in Pi13891373 to 1467335694
5Opus 5.5 in tokenspend-mini13811364 to 1479332706
6GPT-6 Sol in tokenspend-mini13681262 to 1535317733
7Opus 5.5 in mini-swe-agent13001204 to 1440298914
8Opus 5.5 in Claude Code12891264 to 1355306966
9DeepSeek V4.1 Flash in Hermes12871068 to 1627280940
10Opus 5.5 in Hermes12441219 to 12992921106
11Opus 5.5 in Cursor12121182 to 12732821215
12DeepSeek V4.1 Flash in Pi1211952 to 16002581160
13GPT-6 Astra in tokenspend-mini10951042 to 115723915811
14GPT-6 Astra in Pi1059990 to 112122517013
15GPT-6 Astra in Codex1022942 to 10932131869
16GLM 5.3 in Pi1019892 to 111920118111
17GPT-6 Astra in mini-swe-agent1011897 to 10932091909
18GLM 5.3 in Claude Code949874 to 101416519716
19Kimi K3 in Pi903813 to 97014621121
20Kimi K3 in Hermes895758 to 98915022314
21Grok 4.7 in tokenspend-mini857720 to 95314023716
22Grok 4.7 in Cursor842753 to 90512823614
23Kimi K3 in Cursor838733 to 90913124319
24Fable 5.1 in tokenspend-mini793659 to 87112427113
25Grok 4.7 in Claude Code763572 to 8821052737
26Grok 4.7 in Pi761552 to 8991052737
27GPT-6 Astra in Hermes755625 to 83810928415
28GLM 5.3 in mini-swe-agent753484 to 926922644
29Kimi K3 in Claude Code724584 to 8028728018
30Grok 4.7 in mini-swe-agent712518 to 822872894
31Grok 4.7 in Hermes655465 to 768652939
32GLM 5.3 in tokenspend-mini649429 to 758592908
33Kimi K3 in mini-swe-agent636478 to 732663116
34Kimi K3 in tokenspend-mini613401 to 737563096
35GLM 5.3 in Hermes476216 to 593263214

Table 8. Arena rating, cost-aware. Hover a row for its record.

Based on 6,815 comparisons of setup pairs across the 12 tasks, first attempts only. Solving beats failing. When both solve a task the cheaper one wins, and costs within 5% are a draw. Two failures do not count. The average setup rates 1000, and 200 points is about a 76% chance of winning a comparison. This ranks value, so an expensive model that solves everything ranks below a cheap one that does too.

Methodology

  1. Each task is a merged pull request. We check out the repository at the commit before it. The model gets a description of the change and the code, but not the merged change or its tests.
  2. We kept a task only if the merged change passed the pull request's tests and an empty change failed them, in three runs out of three. One model completed each description from the tests, two models from other providers audited it, and a person reviewed it.
  3. Every model ran in our reference agent: five tools (shell, view file, replace text, create file, submit), one prompt, and limits of 60 turns, 30 minutes and a spend cap per attempt. The other harnesses ran as shipped, with the same limits. Containers had 4 CPUs, 5 GB of memory, no network access and no git history.
  4. To grade an attempt, we apply the model's change to a fresh copy, add the pull request's tests, rebuild the project and run the tests. An attempt is resolved when every test the pull request fixed passes and nothing that passed before breaks.
  5. Each request carries a tag for its run, task and model. Cost is priced from the token counts on the router's receipts for that tag.

Limitations

  • A sample of 12 tasks is small. A difference of fewer than about 3 tasks between two models is not evidence that one is better.
  • Completing each description from the tests makes the tasks fair but also easier, and several models solved every task. This set separates models on cost and time more than on ability.
  • The arena rating weighs cost only when both setups solve a task. It does not credit code quality, speed or how close a failed attempt came.
  • Results by model use the reference harness. For open models it is not the best one, since Kimi K3 and GLM 5.3 resolved more tasks in other harnesses.
  • The tasks were merged between 2026-06-10 and 2026-09-27. Some may have been in a model's training data.
  • Slow serving counts against a model. An attempt that hits the time limit fails, even when the delay came from the host.

Tasks

  1. #2213 [STG-2183] fix(cli): list full template catalog (send scope=all) Merged 2026-06-10, 19 lines changed, 2 target tests.
  2. #2220 STG-2196 Shadow DOM fix Merged 2026-06-10, 125 lines changed, 5 target tests.
  3. #2306 fix(core): declare CUA screenshot media type at capture boundary (#2300) Merged 2026-07-06, 139 lines changed, 3 target tests.
  4. #2335 [STG-2511] feat: add browse skills show and agent-facing --help header Merged 2026-07-09, 59 lines changed, 6 target tests.
  5. #2347 feat(core): support OpenAI Chat Completions via openaiEndpointFormat Merged 2026-07-21, 49 lines changed, 4 target tests.
  6. #2361 feat: warn and add opt-out flag for browse CLI .env auto-loading Merged 2026-07-14, 42 lines changed, 6 target tests.
  7. #2542 fix: verify cached local Chrome debug port before trusting it Merged 2026-09-24, 90 lines changed, 3 target tests.
  8. #2752 Allow SDKs to reattach to Stagehand runtimes Merged 2026-08-20, 17 lines changed, 3 target tests.
  9. #2787 ci: publish commit-addressed alpha releases of the Python SDK Merged 2026-08-24, 161 lines changed, 7 target tests.
  10. #2799 fix(cli): resolve template sources and align setup instructions Merged 2026-09-23, 73 lines changed, 8 target tests.
  11. #3034 feat: add internal PDF capture to the extension Merged 2026-09-27, 33 lines changed, 6 target tests.
  12. #3052 fix(extension): read text() from direct child text nodes (#2728) Merged 2026-09-26, 78 lines changed, 3 target tests.