BlogResearch31 min read

Karpo-browser: An Accurate, Fast and Economical System 1 + System 2 Browser Agent

karpo-browser is an open-source System 1 + System 2 browser agent: a general LLM plans goals and reads results, while Karpo Decision 27B Act, a decision model trained for the web, chooses each step on the page. On all 102 Browserbase Benchmark v2 tasks: 89% accuracy, 201 s and $0.096 per task.

Karpo Research

On this page
  1. 1Accurate, fast and economical, built to scale
  2. 2Today's browser agents are slow and expensive
  3. 3How karpo-browser works: System 1 + System 2
  4. 4KBB: an open benchmark for web action decisions
  5. 5Karpo Decision 27B Act: a System 1 trained for the web
  6. 6Detailed results
  7. Try it
  8. References

We are open-sourcing karpo-browser, a browser agent that grew out of Karpo's production work. It divides the work the way psychology distinguishes "fast thinking" (System 1) from "slow thinking" (System 2): a general LLM is System 2, which understands the task, splits it into goals, writes the text to type and reads the results; a decision model is System 1, which chooses the actions on the page step by step, one forward pass per step, about 1 second each. This is how Karpo's Personal Agent operates the browser when it books tickets, places orders and looks things up for its users.

With DeepSeek V4.1 Flash as System 2 and Karpo Decision 27B Act as System 1, karpo-browser was evaluated on all 102 tasks of Browserbase Benchmark v2[1].

  • 89%: task accuracy, the share of tasks done in full.

  • 201 s: average time per task, counting only the agent's own running time.

  • $0.096: model cost per task.

The public leaderboard[1] uses only the 38 official core tasks. On those 38, karpo-browser averages 84%, 218 s and $0.096 over 8 runs; none of the 91 records on the leaderboard reaches 80% accuracy while staying within 250 s and $0.15 per task. Against the 12 of them above 80% accuracy, its accuracy is in the same tier (theirs run from 84% to 92%); it is slightly faster than the fastest of them (223 s) and takes about 40% of their median time (539 s); and it gets about 4 times as much accuracy per dollar as the most economical of them (gpt-6.2-sol + codex) and about 18 times their median, with Gemini at its list price after the introductory rate. See Detailed results at the end for the method and the full numbers. The framework's code, the decision model's weights and the benchmark are all open source, and we invite you to try them. Released together with the framework are two more parts:

  • Karpo Decision 27B Act, the model weights of System 1 in the framework, a version of Karpo's decision model post-trained for browser tasks

  • KBB (Karpo BrowserBench), a benchmark that measures web action decisions only, covering 584 real websites and 23,075 steps, with answers aligned with real human operations and its own evaluation tools.

Accurate, fast and economical, built to scale

When a Personal Agent books tickets, places orders or looks things up for its users, the work ends up on web pages: read the page, decide where to click and what to type, judge whether the job is done and whether to ask the user first. A task takes twenty-odd steps, and every step is a decision. Today's browser agents hand every step to a general LLM: a cheap model makes many mistakes per step, and they accumulate over twenty-odd steps; a strong model is expensive and slow at every step. To solve these problems and let browser agents scale in production, we built karpo-browser: which element to choose and what to do with it at each step goes to a specially trained, fast and cheap decision model; understanding the task, splitting it into goals, giving the values to fill in and reading the results go to a general LLM.

We evaluated karpo-browser on Browserbase Benchmark v2[1], Browserbase's public leaderboard. Its tasks run on real websites, and besides accuracy it measures each task's time and cost. Most other browser agent benchmarks look at accuracy alone, but in production the other two matter just as much.

  • Accurate: 89% accuracy on all 102 tasks.

  • Low cost: $0.096 per task on average, at public API prices. Among the 12 leaderboard records with 80% accuracy or more, the cheapest costs $0.39 per task (gpt-6.2-sol + Codex); karpo-browser costs about a quarter of that.

  • Fast: 218 s per task on average; the 12 leaderboard records above 80% accuracy have a median of 539 s, and the fastest takes 223 s.

  • Built for real web pages: it reopens a page after a network error, waits for pages that have not finished loading, does not click by mistake when the page changes during a decision, does not pay twice after a command times out, and recognizes going in circles and hands control back to the general LLM. These rules and thresholds come from evaluations on a large number of real websites around the world.

Today's browser agents are slow and expensive

Current frameworks generate every step with an LLM

Mainstream browser agent frameworks work in different ways, but "what to do next" on a web page is always decided by what a general LLM generates:

  • browser-use[2] (1.0): the LLM writes code in a persistent Python environment that calls the Chrome debugging interface and browser helper functions (Accessibility.getFullAXTree, click_at_xy, type_text, screenshot and others), writing a piece per turn, looking at the result, then writing the next. The 0.x versions show the LLM a numbered list of elements and a screenshot, and at each step it outputs the next actions (up to 5 by default).

  • Stagehand[3] (the 3.x agent): the LLM picks one tool per step (ariaTree, act, fillForm, goto, scroll, extract and others); act then hands a natural-language action and the accessibility tree to an LLM to locate the element, so one page action often takes two calls.

  • Computer-use models: they look at screenshots and click at coordinates, one multimodal inference per step.

  • General coding agents (Claude Code, Codex and others): they connect to browser tools and operate the web step by step, the way they write code.

Every step needs the LLM to read the page and generate some text, so cost and time grow linearly with the number of steps. In our evaluation, one general LLM turn takes about 3.4 s, reading about 19.5k input tokens and writing 476 output tokens on average.

LLMs are slow and expensive, and paying more does not buy accuracy

The 91 records on the public leaderboard (updated November 18, 2026), grouped by accuracy:

Accuracy

Records

Median (time / cost)

Fastest record (time / cost)

Cheapest record (cost / time)

85% or more

10

452 s / $1.19

223 s / $1.46

$0.39 / 267 s

75% to 85%

9

552 s / $0.85

268 s / $0.41

$0.12 / 312 s

60% to 75%

38

469 s / $1.38

200 s / $0.05

$0.02 / 311 s

Below 60%

34

469 s / $1.05

20 s / $0.09

$0.01 / 764 s

The median time of every tier is around 450 to 550 s, and the median cost around $1. The rank correlation of accuracy with time, and of accuracy with cost, is close to 0, while time and cost correlate at about 0.55: spending more time and money does not buy accuracy, and slow records tend to be expensive too. Records with 60% accuracy or more take at least 200 s; among the 10 records with 85% or more, the slowest takes 937 s per task (gemini-4-carbon + antigravity) and the most expensive costs $3.07 (gpt-6-astra + codex). Costs are as the leaderboard shows them now; for 7 Gemini 4 records they are introductory prices valid through November 30, after which they double.

The strongest models only trade time and money for accuracy

Changing the framework around the same model changes little: deepseek-v4.1-flash with three agent frameworks, Eve, Mastra and Deep Agents (records from September 12), scores 61% to 63% at 713 to 982 s and $0.22 to $0.56 per task. With the strongest models, accuracy goes up, and so do time and money: the two most accurate records are claude-fable-5-2 + claude code at 92%, 378 s and $1.24 per task, and gpt-6.2-astra + codex at 89%, 223 s and $1.46.

Looking at single steps, even the strongest LLMs often choose wrong. On KBB (a page-by-page replay of operations on real websites, described at the end), Claude Fable 5.2 chooses right at 82.1% of steps; an instruction averages 7.6 steps, and only 40.8% of instructions are right at every step. The four general LLMs on the KBB leaderboard complete 38.1% to 41.0% of instructions, and each page decision takes 2.6 to 5.0 s and costs $0.009 to $0.083. Hand every step to them, and errors and costs pile up with the number of steps.

How karpo-browser works: System 1 + System 2

Since the problem is handing every step to an LLM, karpo-browser has a specially trained decision model (System 1) carry out the steps on the page, and the general LLM (System 2) takes care only of goals and results. Each model does what it is good at, which is how karpo-browser manages to be accurate, fast and economical at once.

System 2 plans, System 1 decides

System 2 plans the goals and gets the results. It splits the user's task into goals, gives the values to fill in, and hands them to System 1. A goal states the result to reach and the limits to keep, not where the controls are or how to scroll, and it stops where a judgment is needed: for example, System 1 is told to stop at a list of results, System 2 reads the list and names the item to choose in its next goal. Reading lists, comparing, summarizing, writing the final answer and remembering information across goals are also System 2's job.

System 1 decides each step on the page. From the current state of the page, that is, the numbered page list, a screenshot, a summary of the goal so far and the page changes the last few actions caused, it takes the most probable of the listed options: one forward pass, about 1 s, no sampling and no generated text. Its options include no "open an address", no free typing and no clicking at coordinates; it does not read long lists or compare, and the text it types can only come from values System 2 gave. This is a deliberate trade-off: the more determined the options, the faster the model and the less likely an unexpected action; the cost is that someone has to give it goals and values.

Job

Who does it

Split the task into goals

System 2

Write the text to type (search words, form values)

System 2

Open an address

System 2

Choose the element and the action at each step

System 1

Choose which value goes into which input

System 1, among the values System 2 gave

Judge whether the goal is reached, or why it cannot be

System 1 judges, System 2 checks

Read lists, compare, summarize, write the final answer

System 2

Remember information across goals

System 2

System 1 stops, System 2 takes over

System 1 hands control back to System 2 in two cases: it chooses an ending itself, or the harness stops it. There are 7 endings:

Ending

Meaning

DONE done

The page shows the goal is reached; a goal that submits something needs a confirmation, clicking the button is not enough

NEEDS_FROM_PERSON ask the person

The next step needs something only the person can give: a value not provided, a login, a code sent to the person, another card after a decline

NEEDS_APPROVAL needs approval

The next step is irreversible, such as paying, booking, sending or cancelling, and the goal does not allow it

SITE_CANNOT site cannot

Sold out, no slot available, or the site does not offer the service

BLOCKED blocked

The site refuses this browser, or it still cannot get through after a verification is handled

PAGE_UNUSABLE page unusable

Still an error page after a refresh, or the part to work on can neither be read nor clicked

OTHER_FAILURE other failure

Cannot go on, for none of the reasons above

The harness stops it in 4 cases: it judges that System 1 is going in circles; a goal reaches its limit of 150 steps or 600 s; the caller stops it; or the harness itself fails. That makes 11 ways to end, and the statuses returned to the caller use the same words as the endings.

When System 1 stops, System 2 receives a report in four fixed parts: how it ended; the page it stopped on, with the steps and seconds used; the last 5 actions and the page changes they caused; and the last page read (up to 400 lines). System 2 takes over from there without reading the page again:

  • When System 1 judges the goal done: its "done" is only its judgment, so System 2 checks it against the page, then hands over the next goal or gives the answer.

  • When System 1 cannot go on: the ending says why it stopped, and System 2 decides whether to hand System 1 a different goal, ask the user, or report truthfully; it can also act by hand with click, type, select, scroll, back and others. Acting by hand uses the same numbered page list and the same execution code, up to 15 times per task by default.

System 2 also has tools such as open (open an address), read_page (read the current page) and finish (give the answer). When its turns run out, only finish is left, so a task always ends with an answer.

Harness: fewer turns, better choices, steadier actions

The harness is the code in karpo-browser outside the models. System 1 only chooses: it generates no text and remembers nothing. Reading the page, turning it into numbered options, carrying out the chosen action, working out the changes it caused and remembering what happened in the round are all the harness's work. System 2 can be a general LLM that karpo-browser calls directly, or the developer's own agent. Either way, System 2 opens addresses, reads pages and hands goals to System 1 through tools the harness provides. Overall, the harness pays off in three ways:

First, it saves System 2 turns.

  • The harness passes goals and results between the two models: a goal goes out, System 1 works through it step by step, and the harness sums up the process in the report above. System 2 does not have to spend a turn on every page action; the time and cost saved are in "The measured division of work" below.

Second, it lets System 1 read less and choose better

  • Writing the web page as a numbered list: it merges a DOM snapshot (layout, paint order, styles, clickability) with the full accessibility tree, one element per line as "number role name [state] href=…". Only elements that can be operated get numbers; elements that are covered, clipped or behind a modal dialog are left out; clickable elements the accessibility tree misses, such as a div used as a button, are added; the list is cut into three parts: the current screen, above it and below it. The screenshot shows only the current screen, scaled to 768 pixels wide.

  • Keeping requests small: a request holds at most about 32,000 tokens and each question at most 255 options; when not everything fits, priorities decide: the part on screen is kept whole, the parts off screen are summarized. An off-screen element written into the request can still be chosen, and is scrolled into view before it is clicked. In practice the median input per request is 7.3k to 7.7k tokens (screenshot included).

  • Code works out what each action did: each page read gets a page state identifier; after each action, the reads before and after are compared and the changes written under the action; on returning to a page read before, the elements already tried are marked with what happened last time. System 1 remembers nothing, yet knows whether its last action worked.

The page list System 1 reads (an illustration; the line format matches real requests):

viewport (scrolled 629/2373px):
  banner
    4 link Home href=/
    search
      5 button Destination: area, station or restaurant [expanded=false]
  heading Enter your booking details
  text A deposit of 5,000 yen per person is required.
  9 textbox Name [required] [value=]
  10 textbox Email [required] [value=sample@example.com]
  11 div [pointer]
  dialog Privacy settings
    13 button Reject all
above:
  banner
    2 link List your restaurant href=/join/
below:
  35 link Private rooms href=/shops/sample/reserve
  36 button Next
  37 div See more reviews [clickable]

Third, it makes actions on real web pages reliable and safe

  • Check before acting, never click blind: a decision takes about 1 s, and the page may redraw meanwhile. Before acting, the harness confirms that the numbered element is still there, with the same role and name; before clicking, it probes 25 points on the element and sends real mouse events only once a point is confirmed to hit that element. Native dropdowns and date inputs are set directly, not through the interface.

  • Failures come with reasons, and the harness does not retry for the model: an action that did not go through is written into the history with its reason, which the model reads at the next step. A command that times out is recorded as "may have taken effect", not as a failure: retrying "pay" on a busy page could pay twice.

  • Handling the surprises of real web pages: a network error when opening an address gets up to 2 more attempts; after each action the harness waits for the page to settle, waiting only for the page's own requests, not for ads and trackers; a browser verification interstitial gets a few seconds, and the run goes on if it lets the browser through; crashed tabs and closed pop-ups are handled too. These rules and thresholds were tuned one by one in evaluations on a large number of real websites.

  • Holding the boundaries: text on the page is treated as data, never as instructions. Secret values are never sent to any model as text; both models see only their names; after a secret value is typed, System 1 no longer receives screenshots (System 2 still does if its image input is on, a gap not yet closed). When the goal does not allow an irreversible action such as paying, booking or sending, System 1 should choose "needs approval" and stop; this relies on the model's choice, the code does not block it, and KBB measures 0.99 overreaches per thousand steps.

  • Recognizing circles and handing back in time: at each step System 1 also answers a yes/no question, "are the recent actions going in circles", and stops when that probability is at least 0.8 for 2 steps in a row. The harness's code stops it as well when an action brings the page back to the same state for the third time, or when the last 6 steps or more repeat the same small loop of 2 to 4 steps, with nothing new read in between. Control then goes back to System 2 instead of running up to the step and time limits. The code looks at whether anything new was read, not at whether actions repeat: going through the items of a list repeats actions (open, back) but is normal work.

System 2 can also be an existing agent. karpo-browser can run as an MCP server and give three tools, open, run_browser_agent and read_page, to Claude Code, Codex or any agent that supports MCP, which then acts as System 2: it understands the task, splits it into goals and reads the results, while the steps on the page still go to System 1. Developers can choose to plug karpo-browser straight into their own System 2 agent.

The measured division of work

In the 3 runs over all 102 tasks of Browserbase Benchmark v2 (see "Detailed results"), on average per task:

  • System 2 took 22.5 turns and handed 3.1 goals to System 1.

  • System 1 was called 26.4 times and did 21.9 page actions; System 2 acted by hand 6.2 times.

  • Leaving out opening addresses, System 1 did about 78% of the page actions, in only 14% of each task's time (27.5 of 201 s) and about 13% of its model cost.

  • The 3 runs made 8,089 decision calls in all, and the server recorded 0 output tokens for every one of them.

One System 2 turn

One System 1 call

Time

About 3.4 s

About 1 s

Average input

About 19.5k tokens

About 8.0k tokens

Average output

476 tokens

0

If each of those 21.9 page actions took a System 2 turn, the averages above put the extra at about 75 s per task (System 1 actually took 27.5 s in total) and about $0.08 more System 2 cost (System 1 actually cost $0.013).

In 30 of the 306 task runs, System 2 handed no goal to System 1: 10 got by with opening addresses and reading pages alone, and in the other 20 System 2 clicked or typed by itself.

KBB: an open benchmark for web action decisions

What existing benchmarks miss

To tell whether a browser agent can go into production, an evaluation has to answer four questions: can it get things done on real websites; how much does each task cost and how long does it take; can results be reproduced and different systems compared; and when it fails, at which step. Whether an agent stops to ask for approval or goes ahead on irreversible steps such as paying and submitting also needs measuring. Existing leaderboards and benchmarks each answer only part of this:

Benchmark

Websites

Reproducible

Cost and time

Grading

Browserbase Benchmark v2[1]

Live sites

No

Published for every record

Whole task; Browserbase's grader applies a rubric

Online-Mind2Web[4] (136 sites, 300 tasks)

Live sites

No

Some leaderboards report total cost, none report time

Whole task; the WebJudge grading model (about 85% agreement with humans) or humans

WebVoyager[5] (15 sites, 643 tasks)

Live sites

No

Not counted

Whole task; a grading model or humans

Mind2Web[6] (137 sites, 2,350 tasks)

Page snapshots from 2023

Yes

Not counted

Per step; matched against crowd-annotated elements and operations

WebArena[7], VisualWebArena[8]

Self-hosted replicas

Yes

Not counted

Whole task; programs check the result

KBB (584 sites, 23,075 steps)

Recorded on real sites, replayed offline

Yes

Part of the total score

Per step; answers aligned with real humans, seven endings included

End-to-end evaluations show whether the job got done, but are hard to reproduce and hard to compare. Websites keep changing, so two runs of the same system meet different pages; results are judged by a grading model, and the numbers move a lot when the runner or the grading changes. On Online-Mind2Web, the best third-party reproduction by Princeton's HAL is 42.33%[9], and HAL has paused adding new models; the best vendor-reported score is 97%, with the vendor's own grading model; the same agent scored 78.7% with human grading and 64.7% with WebJudge[10]. Benchmarks differ in difficulty, and time and cost cannot be compared across them. So this post compares only within one leaderboard and repeats every configuration.

The most-cited leaderboard mostly measures coding agents. The 5 records at 89% accuracy or more on Browserbase Benchmark v2 all come from Claude Code, Codex and Antigravity, at $0.51 to $1.46 and 223 to 937 s per task. Coding agents are productivity tools that developers use interactively on their own machines, carrying for every task a full set of coding tools that read and write files and run commands; their ranking answers how accurate the strongest model with the strongest coding agent can be. A product's backend needs something else: a service embedded in the product that handles many users' tasks at once, with predictable cost and time per task. Browser frameworks built for products are rare on the leaderboard: browser-use is not listed, and Stagehand's own agent has only 4 records; Browserbase's Online-Mind2Web leaderboard[11] has only 10 records, all of them Stagehand's own agent. Browser frameworks mostly report their own numbers: browser-use's model comparison uses its own 180 tasks[12], run by itself. We found no evaluation that compares browser frameworks on the same tasks with the same LLM while reporting accuracy, cost and time together.

End-to-end evaluations do not show which step went wrong, and rarely test irreversible steps. A task takes twenty-odd steps, and a failure can come from any of them; the final result alone can neither locate it nor guide improvements one by one. Paying and submitting cannot really be done on live sites, so most benchmarks avoid such tasks; ClawBench intercepts the final submission request so that ordering and submitting can be carried to the last step, but it still grades only the final result, and its best scores today are 44.6% under the lenient standard and 24.6% under the strict one[13].

KBB adds reproducibility and step-level diagnosis, and counts cost, latency and overreach in the evaluation. Its limit is that the replay follows the recorded path, so it cannot measure an agent correcting itself after going astray; that still takes end-to-end evaluation. So we evaluate on two layers: Browserbase Benchmark v2 measures whether the whole job gets done and at what cost and time, and KBB measures whether every step is chosen right.

How KBB evaluates

KBB (Karpo BrowserBench) measures web action decisions only: whether each step is chosen right. The request at each step of KBB is exactly the one sent during recording, with only two times added.

  • Scale: KBB 1.0 has 584 real websites, 787 tasks, 3,026 instructions and 23,075 steps, split by brand family into two parts: a private part of 15,000 steps used for ranking, and a public part scored separately.

  • Page-by-page offline replay: the process recorded on a real website is cut into one step per page and replayed page by page. At each step the model gets what System 1 saw during recording: the goal, the values it may type, the actions already taken, the page text and the screenshot, together with every action available on that page (99 at the median) and the seven endings. Whatever the model chooses, the next step is the next recorded page, so every model faces exactly the same pages. A goal handed to System 1 is one instruction, 7.6 steps on average.

  • Every step counts: an instruction is complete only when every one of its steps is chosen right.

  • Cost and speed are in the total score: the total score is the completion rate multiplied by a cost factor and a latency factor; cost and the latency of a single decision each have a budget ($0.0063 per instruction, 444 ms per decision), and going over it discounts the completion rate.

  • Answers aligned with real humans: the standard is whether a sensible person doing this for someone else would act this way at this point. 5,000 human-labeled steps are the baseline; the rest are synthetic labels, which agree with the human labels more than 95% of the time.

  • Evaluation tools included: a model scorer grades the model under test step by step against KBB's answer set and computes the completion rate and the step accuracy.

How different models do on KBB

System 1 has to choose right at every step, and also be fast and cheap. We used KBB to compare how different models do as System 1: KBB replays operations recorded on real websites page by page, at each step the model gets the kind of request System 1 receives in karpo-browser, and an instruction is complete only when every step is right (described at the end). The table below covers the private part, 1,794 scored instructions; the leaderboard figures are from the KBB page on November 17, 2026.

Model

Completion rate

Step accuracy

Cost per instruction

Median latency

The four general LLMs on the leaderboard

38.1% to 41.0%

79.2% to 82.1%

$0.069 to $0.635

2,579 to 4,952 ms

Karpo Decision 27B Act

35.5%

78.5%

$0.0039

931 ms

The decision model with the highest completion rate on the leaderboard

28.8%

70.5%

$0.015

Measured differently, not listed

  • General LLMs choose well, but are expensive and slow: $0.069 to $0.635 per instruction, 2.6 to 5.0 s per decision.

  • General decision models are cheap, but not accurate enough on the web: each of the 25 decision models on the leaderboard costs less per instruction than the cheapest general LLM, yet the highest completion rate among them is only 28.8%. They read only the page text, while the general LLMs also see the screenshot, which accounts for part of the gap.

  • Karpo Decision 27B Act comes close to the general LLMs' accuracy at a decision model's cost: a completion rate of 35.5%, above every decision model on the leaderboard; 2.6 to 5.5 points below the four general LLMs, at 1/18 to 1/163 of their cost per instruction (self-hosted systems are priced by KBB's common method: RTX PRO 6000 at $1.80 per GPU-hour).

  • Little overreach: 0.99 unauthorized payments or submissions per thousand steps; the decision models on the leaderboard make 4.20 to 13.22, and the four general LLMs 0.61 to 1.91.

  • 33.7% by KBB's total score formula: cost is within the budget, latency about 2.1 times the budget; measured the way the leaderboard currently measures self-hosted systems (4 requests at once), it is 30.4%. The highest total score on the leaderboard today is 28.7%. The formula discounts expensive and slow systems, which by itself favors small models.

Karpo Decision 27B Act: a System 1 trained for the web

Karpo Decision 27B Act is karpo-browser's System 1. It follows Karpo Decision 31B[14], released on November 13: the same single forward pass that reads probabilities directly over the given options without generating text, and the same request interface and readout head. Training starts from Karpo Decision 27B, Karpo's general decision model, which was not released separately; it was trained with the same version of general decision data and the same method as Karpo Decision 31B, and is post-trained from Qwen3.8-27B, a multimodal model with its own vision encoder.

Action space: each step is a multiple-choice question

Options are generated from the numbered elements in the page list and written as "operation number", for example CLICK 5.

Actions on a single element:

Action

What it does

Which elements have it

CLICK n

Click

Clickable elements, inputs included

FILL n

Fill a value into an input

Editable inputs, not read-only ones

PRESS_ENTER n

Press Enter in an input

Inputs

SELECT n

Choose an item in a dropdown

Native <select>, which gets only this action and is never clicked open

SCROLL_DOWN n / SCROLL_UP n

Scroll

Scrollable areas, in whichever directions they can scroll; n = 1 means the whole page

Actions on the whole page:

Action

What it does

WAIT

Wait for the page to finish loading or changing

ESCAPE

Press Esc to close a pop-up or menu

REFRESH

Refresh the page

SOLVE_CAPTCHA

Hand over to the verification handler the caller configured

BACK

Go back to the previous page, or to the page that opened this tab

Then come the 7 endings above. Waiting and refreshing are actions, not endings. On KBB, the median number of page actions to choose from at each step is 99. In one of the runs, of the 2,653 steps System 1 chose, clicks were 46%, scrolling down 18%, done 9%, waiting 9% and filling 7%.

FILL and SELECT are each split into two questions: the step only decides which input or dropdown to act on, and a follow-up question decides which value to fill in or which item to choose. For filling, the options are the numbers of the values System 2 gave; for a dropdown, its own items; both also offer "none fits". The follow-up used to be a separate call. A switch, off by default, merges the follow-ups of up to 3 inputs or dropdowns on screen into the same request, answered in one forward pass, which saves 2 to 3 calls per task; this form is not in the training data, so the switch is off by default, and the Browserbase Benchmark v2 runs turned it on.

Training data: recordings on real websites

The training data comes from recordings on real websites: people and models complete tasks together on real consumer websites, recording each step's request (the page text, the values to type, the previous actions), the screenshot and the choice made. Only pages that pass a re-check are kept: the saved page data is read again, and the request built from it must be identical to the one recorded.

The labels are aligned with real human operations: human operations on real websites are the baseline, and the more numerous synthetic labels come from a larger general LLM and are aligned with the human data. The model learns to make the same choice on the same page with the same options.

  • The web data has 290,998 sequences (205,909 with screenshots) and 551,102 multiple-choice questions.

  • The websites cover consumer categories such as travel, shopping, dining, tickets and attractions, mostly in English, with sites from Japan, Korea, Germany, France and elsewhere.

  • General decision questions are mixed in, 20% of all questions, to keep the general decision ability.

  • Sites of the same brand form a brand family; 59 families are held out of training as a whole and used only for periodic validation during training.

KBB's tasks, instructions and pages do not enter the training data. The brand families of the websites KBB uses are in the training set, so KBB tests new tasks on these websites, not unseen websites.

Training, KBB evaluation and karpo-browser at run time all give the model the same kind of request. KBB, too, tests requests recorded on real websites, with only two extra time fields. The harness code that builds requests was ported from the program used during recording. After porting, we replayed 15,005 pages saved from 899 recordings, and the 15,379 requests the harness built were identical to the recorded ones, character for character. That is why KBB scores represent how the model performs inside karpo-browser.

Training method: full fine-tuning with screenshots

  • Full fine-tuning, no text generation: the objective is the cross-entropy over each question's own options. Training ran 1 epoch, 8,395 steps, at a learning rate of 2e-6, in about 17 hours on a single 8-GPU machine.

  • Training with screenshots: the general LLM that made the choices during recording looked at the page text plus a screenshot, and KBB's judging principles also say the screenshot wins when it disagrees with the text. The starting model had been trained on text only, and this was its first training with images, so the vision encoder was frozen and the projection layer was trained together with the language model.

  • Random screenshot formats: screenshots are scaled randomly to 768 to 1,440 pixels wide, with JPEG quality random between 50 and 95.

  • About a fifth of the step requests drop the screenshot: follow-up questions carry no image anyway. The same model thus works with and without images, and does not break when screenshots stop after a secret value is typed. During recording, screenshots likewise stopped after a secret value was typed, so 4,122 of KBB's 23,075 steps have no screenshot, and the judging principles say explicitly "with no screenshot, judge by the text".

Multi-GPU training hit one problem: when one GPU got only image-free sequences at a step, the vision projection layer had no gradient on that GPU, and the multi-GPU gradient synchronization hung. The fix was to pass image-free batches through the projection layer once with zero weight.

Model Fusion: merging checkpoints from different stages

Model Fusion is the merging step after training: an even average of the weights of three checkpoints, mixed 0.7 to 0.3 with the starting model Karpo Decision 27B, the averaged weights taking 0.7. The three checkpoints are steps 7,500 and 8,000 and the last step. The learning rate did not decay during training, the model wavered over the last stretch, and the best on the held-out set was step 7,556, not the last. When the learning rate does not decay, averaging several points along the training trajectory usually generalizes better than taking the last point alone[15]. Mixing back the starting model is what WiSE-FT[16] does: it interpolates the fine-tuned weights linearly with the starting weights, which is reported to give clearly higher accuracy under distribution shift while keeping the accuracy on the original task. The fused model is still an ordinary model, with the same architecture, parameter count and inference cost.

Model

KBB completion rate

Step accuracy

Held-out loss

General decision questions (sample)

Starting point, Karpo Decision 27B

27.6%

67.0%

0.792

76.21

Checkpoint at the last training step

34.1%

77.6%

0.328

73.47

Model Fusion

35.3%

78.3%

0.282

74.49

KBB is computed over all 2,748 scored instructions; the 35.5% and 78.5% earlier count only the private part. The held-out set is the same 11,740 questions throughout, and the general decision questions are a 15% sample. The Model Fusion row is the Karpo Decision 27B Act released now.

  • The KBB completion rate is 1.2 points above the last checkpoint, and the held-out loss drops from 0.328 to 0.282, the lowest of the 8 candidates.

  • Most of the gain comes from mixing back the starting model: mixing back alone, without averaging, raises the KBB completion rate by 1.3 points (paired test p = 0.004). Averaging the three checkpoints on top makes no further difference on KBB (p = 0.69) and lowers the held-out loss from 0.299 to 0.282.

  • Mixing back the starting model also keeps part of the general decision ability: on the sample of general decision questions, the last checkpoint is 2.7 points below the starting point, and Model Fusion 1.7 points below. For general decision tasks, please use Karpo Decision 31B.

Detailed results

Key metrics

Metric

karpo-browser

Comparison

Notes

Browserbase Benchmark v2 accuracy, all 102 tasks

89%

The leaderboard publishes results on the 38 core tasks only; see the comparison below

Previously reported scores for the three runs were 91%, 84% and 83%.

Browserbase Benchmark v2 time per task

201 s

—

Only the agent's own running time, average of 3 runs

Browserbase Benchmark v2 model cost per task

$0.096

—

Basis below; $0.11 to $0.14 by actual GPU hours

Online-Mind2Web accuracy, 300 tasks

72.3%

Best on the leaderboard 84.1%, second 72.3%

1 run, graded by WebJudge

Online-Mind2Web time per task

125 s

Records with 70% or more: 180 to 285 s

Only the agent's own running time

Online-Mind2Web model cost per task

$0.048

Cheapest record on the leaderboard: $0.058

Same cost basis as Browserbase Benchmark v2

System 1 time per decision (median)

0.95 s

One System 2 turn in the same runs: about 3.4 s

RTX PRO 6000 with inference optimizations; see below

Page actions done by System 1

About 78%

—

Browserbase Benchmark v2; not counting opening addresses, 14% of each task's time and 13% of its model cost

Browserbase Benchmark v2 results

Browserbase Benchmark v2 measures whether the whole job gets done: given a one-sentence task, complete it on real websites and give an answer, which a grader judges against a rubric written in advance. How we tested, and how it differs from the leaderboard:

  • Task set: the official task set has 102 tasks, 38 of them core[17], and each leaderboard record is one run of the core set.

  • Grading: Stagehand's open-source evaluation program (the November 10, 2026 version, plus a patch that connects karpo-browser), judging by each task's preset rubric (with the v1.2 clarifications), with the program's default grading model, gemini-3.5-flash.

  • Timing: only the agent's own running time, not grading; each task is limited to 450 s. We did not find how the leaderboard defines its timing.

  • Inference: System 1 runs on RTX PRO 6000 GPUs, with fp8 inference and CUDA graphs. Over the 8,089 decision calls of the 3 runs, the median time per decision is about 0.95 s, timed at the harness, network round trip included.

  • Cost: model cost = System 2 at OpenRouter's public price of the day for the tokens actually used + System 1 at $0.06 per million input tokens. The price for Karpo Decision 27B Act is the input price of the same-size base model on OpenRouter, for comparison only. Browserbase's browser fees (about $0.33 to $0.38 per task) and grading are not included.

The upper part of the table is taken from the public leaderboard (updated November 18, 2026, 91 records): the most accurate records, the ones closest to ours in accuracy or cost, and the one with the same model as our System 2, named as on the leaderboard, each record one run of the 38 core tasks. The last row is ours.

Model + harness

Accuracy

Time

Cost per task

Tasks and runs

claude-fable-5-2 + claude code

92%

378 s

$1.242

38 core tasks, 1 run

gemini-4-carbon + antigravity

90%

937 s

$1.130 (introductory price through November 30, $2.261 after)

38 core tasks, 1 run

gpt-6.2-astra + codex

89%

223 s

$1.457

38 core tasks, 1 run

claude-opus-5-5 + claude code

89%

326 s

$0.510

38 core tasks, 1 run (measured in September)

gpt-6.2-sol + codex

87%

267 s

$0.386

38 core tasks, 1 run

gpt-6.1-terra + codex

79%

312 s

$0.117

38 core tasks, 1 run

deepseek-v4.1-flash + eve

63%

713 s

$0.304

38 core tasks, 1 run (measured in September)

DeepSeek V4.1 Flash × Karpo Decision 27B Act + karpo-browser

84%

218 s

$0.096

38 core tasks, average of 8 runs (ours)

  • About a quarter of the cost of records with 80% or more: among the 12 leaderboard records with 80% or more, the cheapest is gpt-6.2-sol + codex, at 87%, 267 s and $0.386. In the adjacent 79% tier the cheapest is gpt-6.1-terra + codex, at $0.117 and 312 s; by actual GPU hours we are at $0.11 to $0.14, about the same.

  • Every faster record is less accurate than ours, and so is every cheaper one: 6 records are faster than 218 s, with at most 66% accuracy; 11 are cheaper than $0.096, with at most 74% (gpt-6-luna + codex, $0.023 and 311 s; claude-haiku-5-5 + claude code, $0.068 and 307 s). Among records with 75% or more, the fastest is gpt-6.2-astra + codex at 223 s, and the cheapest costs $0.117.

Online-Mind2Web results

Online-Mind2Web[4] is another benchmark that evaluates whole jobs on live websites, with 300 tasks on 136 websites. We used the new version of its tasks on Hugging Face (31 tasks reworded, 3 of them going to amazon.com), the same two models as on Browserbase Benchmark v2 with similar harness settings, ran each task once (November 19, 2026), and graded with WebJudge, the grader the paper recommends, using GPT-6 Luna as the grading model.

Browserbase also publishes an Online-Mind2Web leaderboard[11] (updated November 18, 2026, 10 records, all of them Stagehand's own agent in hybrid or cua mode). The leaderboard notes that these results were collected on Stagehand v3 and are kept for reference, with current numbers on Browserbase Benchmark v2. With our result added:

Model + harness

Accuracy

Time

Model cost per task

claude-opus-4-8 (hybrid)

84.1%

260 s

$1.156

DeepSeek V4.1 Flash × Karpo Decision 27B Act + karpo-browser

72.3%

125 s

$0.048

gpt-5.5 (hybrid)

72.3%

285 s

$1.730

gemini-2.5-computer-use (cua)

71.3%

180 s

$0.273

claude-opus-4-7 (cua)

66.3%

142 s

$1.221

gemini-3-flash-preview (cua)

65.3%

209 s

$0.063

claude-sonnet-4-6 (cua)

62.7%

387 s

$1.850

claude-sonnet-4-5 (cua)

57.3%

339 s

$1.498

claude-haiku-4-5 (cua)

54.0%

213 s

$0.468

gpt-5.4 (cua)

40.7%

108 s

$0.474

gpt-5.4-mini (cua)

24.3%

62 s

$0.058

  • Tied for second in accuracy: 72.3%, the same as gpt-5.5 (hybrid), behind only claude-opus-4-8 (hybrid) at 84.1%.

  • Lowest cost: $0.048 per task, below every record on the leaderboard; gemini-2.5-computer-use, close in accuracy, costs $0.273 per task, about 6 times ours, and gpt-5.5 costs $1.730, about 36 times.

  • Fast: 125 s per task; the only faster records are gpt-5.4 (108 s, 40.7%) and gpt-5.4-mini (62 s, 24.3%).

This comparison is for reference only, as the conditions differ: we graded all 300 tasks with WebJudge, while the leaderboard does not say how it grades or how many tasks it runs (by default the Stagehand evaluation program samples 25 tasks and runs each 3 times, with its own grading model); the cost counts model fees only, not the cloud browser (about $0.21 per task); time is averaged over the 296 tasks that actually ran, since the 4 theweathernetwork.com tasks kept redirecting through Browserbase's US proxy and never opened, and they count as failures in the accuracy. All 3 amazon.com tasks passed.

Try it

References

  1. Browserbase. Browser Agent Evals: Browserbase Benchmark v2. 2026.

  2. Browser Use. browser-use. GitHub, 2026.

  3. Browserbase. Stagehand. GitHub, 2026.

  4. Xue, T. et al. An Illusion of Progress? Assessing the Current State of Web Agents. arXiv:2504.01382, 2025.

  5. He, H. et al. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. ACL 2024.

  6. Deng, X. et al. Mind2Web: Towards a Generalist Agent for the Web. NeurIPS 2023.

  7. Zhou, S. et al. WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR 2024.

  8. Koh, J. Y. et al. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. ACL 2024.

  9. Princeton HAL. Online Mind2Web leaderboard. 2026.

  10. Steel. Online-Mind2Web leaderboard. 2026.

  11. Browserbase. Browser Agent Evals: Online-Mind2Web. 2026.

  12. Browser Use. Benchmarks. 2026.

  13. TIGER-AI-Lab. ClawBench leaderboard. 2026.

  14. Karpo. Karpo Decision 31B: A Fast, Accurate Multimodal Decision Model. 2026.

  15. Izmailov, P. et al. Averaging Weights Leads to Wider Optima and Better Generalization. UAI 2018.

  16. Wortsman, M. et al. Robust fine-tuning of zero-shot models. CVPR 2022.

  17. Browserbase. HardBench task set. GitHub, 2026.

Keep reading

Blog →
  • Research

    Karpo Decision 31B: A Fast, Accurate Multimodal Decision Model

    Today, we are releasing Karpo Decision 31B, a decision model built on Gemma 4 31B: given a state and typed questions, it answers them all in one inference. Decision Index Full 73.9, first on the board; JevBench Capability 87.2, third among 243 Jev-class open-weight models; median latency 95 ms.