We are open-sourcing karpo-browser, a browser agent that grew out of Karpo's production work. It divides the work the way psychology distinguishes "fast thinking" (System 1) from "slow thinking" (System 2): a general LLM is System 2, which understands the task, splits it into goals, writes the text to type and reads the results; a decision model is System 1, which chooses the actions on the page step by step, one forward pass per step, about 1 second each. This is how Karpo's Personal Agent operates the browser when it books tickets, places orders and looks things up for its users.
With DeepSeek V4.1 Flash as System 2 and Karpo Decision 27B Act as System 1, karpo-browser was evaluated on all 102 tasks of Browserbase Benchmark v2[1].
89%: task accuracy, the share of tasks done in full.
201 s: average time per task, counting only the agent's own running time.
$0.096: model cost per task.
The public leaderboard[1] uses only the 38 official core tasks. On those 38, karpo-browser averages 84%, 218 s and $0.096 over 8 runs; none of the 91 records on the leaderboard reaches 80% accuracy while staying within 250 s and $0.15 per task. Against the 12 of them above 80% accuracy, its accuracy is in the same tier (theirs run from 84% to 92%); it is slightly faster than the fastest of them (223 s) and takes about 40% of their median time (539 s); and it gets about 4 times as much accuracy per dollar as the most economical of them (gpt-6.2-sol + codex) and about 18 times their median, with Gemini at its list price after the introductory rate. See Detailed results at the end for the method and the full numbers. The framework's code, the decision model's weights and the benchmark are all open source, and we invite you to try them. Released together with the framework are two more parts:
Karpo Decision 27B Act, the model weights of System 1 in the framework, a version of Karpo's decision model post-trained for browser tasks
KBB (Karpo BrowserBench), a benchmark that measures web action decisions only, covering 584 real websites and 23,075 steps, with answers aligned with real human operations and its own evaluation tools.
Accurate, fast and economical, built to scale
When a Personal Agent books tickets, places orders or looks things up for its users, the work ends up on web pages: read the page, decide where to click and what to type, judge whether the job is done and whether to ask the user first. A task takes twenty-odd steps, and every step is a decision. Today's browser agents hand every step to a general LLM: a cheap model makes many mistakes per step, and they accumulate over twenty-odd steps; a strong model is expensive and slow at every step. To solve these problems and let browser agents scale in production, we built karpo-browser: which element to choose and what to do with it at each step goes to a specially trained, fast and cheap decision model; understanding the task, splitting it into goals, giving the values to fill in and reading the results go to a general LLM.
We evaluated karpo-browser on Browserbase Benchmark v2[1], Browserbase's public leaderboard. Its tasks run on real websites, and besides accuracy it measures each task's time and cost. Most other browser agent benchmarks look at accuracy alone, but in production the other two matter just as much.
Accurate: 89% accuracy on all 102 tasks.
Low cost: $0.096 per task on average, at public API prices. Among the 12 leaderboard records with 80% accuracy or more, the cheapest costs $0.39 per task (gpt-6.2-sol + Codex); karpo-browser costs about a quarter of that.
Fast: 218 s per task on average; the 12 leaderboard records above 80% accuracy have a median of 539 s, and the fastest takes 223 s.
Built for real web pages: it reopens a page after a network error, waits for pages that have not finished loading, does not click by mistake when the page changes during a decision, does not pay twice after a command times out, and recognizes going in circles and hands control back to the general LLM. These rules and thresholds come from evaluations on a large number of real websites around the world.
Today's browser agents are slow and expensive
Current frameworks generate every step with an LLM
Mainstream browser agent frameworks work in different ways, but "what to do next" on a web page is always decided by what a general LLM generates:
browser-use[2] (1.0): the LLM writes code in a persistent Python environment that calls the Chrome debugging interface and browser helper functions (
Accessibility.getFullAXTree,click_at_xy,type_text,screenshotand others), writing a piece per turn, looking at the result, then writing the next. The 0.x versions show the LLM a numbered list of elements and a screenshot, and at each step it outputs the next actions (up to 5 by default).Stagehand[3] (the 3.x agent): the LLM picks one tool per step (
ariaTree,act,fillForm,goto,scroll,extractand others);actthen hands a natural-language action and the accessibility tree to an LLM to locate the element, so one page action often takes two calls.Computer-use models: they look at screenshots and click at coordinates, one multimodal inference per step.
General coding agents (Claude Code, Codex and others): they connect to browser tools and operate the web step by step, the way they write code.
Every step needs the LLM to read the page and generate some text, so cost and time grow linearly with the number of steps. In our evaluation, one general LLM turn takes about 3.4 s, reading about 19.5k input tokens and writing 476 output tokens on average.
LLMs are slow and expensive, and paying more does not buy accuracy
The 91 records on the public leaderboard (updated November 18, 2026), grouped by accuracy:
Accuracy | Records | Median (time / cost) | Fastest record (time / cost) | Cheapest record (cost / time) |
|---|---|---|---|---|
85% or more | 10 | 452 s / $1.19 | 223 s / $1.46 | $0.39 / 267 s |
75% to 85% | 9 | 552 s / $0.85 | 268 s / $0.41 | $0.12 / 312 s |
60% to 75% | 38 | 469 s / $1.38 | 200 s / $0.05 | $0.02 / 311 s |
Below 60% | 34 | 469 s / $1.05 | 20 s / $0.09 | $0.01 / 764 s |
The median time of every tier is around 450 to 550 s, and the median cost around $1. The rank correlation of accuracy with time, and of accuracy with cost, is close to 0, while time and cost correlate at about 0.55: spending more time and money does not buy accuracy, and slow records tend to be expensive too. Records with 60% accuracy or more take at least 200 s; among the 10 records with 85% or more, the slowest takes 937 s per task (gemini-4-carbon + antigravity) and the most expensive costs $3.07 (gpt-6-astra + codex). Costs are as the leaderboard shows them now; for 7 Gemini 4 records they are introductory prices valid through November 30, after which they double.
The strongest models only trade time and money for accuracy
Changing the framework around the same model changes little: deepseek-v4.1-flash with three agent frameworks, Eve, Mastra and Deep Agents (records from September 12), scores 61% to 63% at 713 to 982 s and $0.22 to $0.56 per task. With the strongest models, accuracy goes up, and so do time and money: the two most accurate records are claude-fable-5-2 + claude code at 92%, 378 s and $1.24 per task, and gpt-6.2-astra + codex at 89%, 223 s and $1.46.
Looking at single steps, even the strongest LLMs often choose wrong. On KBB (a page-by-page replay of operations on real websites, described at the end), Claude Fable 5.2 chooses right at 82.1% of steps; an instruction averages 7.6 steps, and only 40.8% of instructions are right at every step. The four general LLMs on the KBB leaderboard complete 38.1% to 41.0% of instructions, and each page decision takes 2.6 to 5.0 s and costs $0.009 to $0.083. Hand every step to them, and errors and costs pile up with the number of steps.
How karpo-browser works: System 1 + System 2
Since the problem is handing every step to an LLM, karpo-browser has a specially trained decision model (System 1) carry out the steps on the page, and the general LLM (System 2) takes care only of goals and results. Each model does what it is good at, which is how karpo-browser manages to be accurate, fast and economical at once.
System 2 plans, System 1 decides
System 2 plans the goals and gets the results. It splits the user's task into goals, gives the values to fill in, and hands them to System 1. A goal states the result to reach and the limits to keep, not where the controls are or how to scroll, and it stops where a judgment is needed: for example, System 1 is told to stop at a list of results, System 2 reads the list and names the item to choose in its next goal. Reading lists, comparing, summarizing, writing the final answer and remembering information across goals are also System 2's job.
System 1 decides each step on the page. From the current state of the page, that is, the numbered page list, a screenshot, a summary of the goal so far and the page changes the last few actions caused, it takes the most probable of the listed options: one forward pass, about 1 s, no sampling and no generated text. Its options include no "open an address", no free typing and no clicking at coordinates; it does not read long lists or compare, and the text it types can only come from values System 2 gave. This is a deliberate trade-off: the more determined the options, the faster the model and the less likely an unexpected action; the cost is that someone has to give it goals and values.
Job | Who does it |
|---|---|
Split the task into goals | System 2 |
Write the text to type (search words, form values) | System 2 |
Open an address | System 2 |
Choose the element and the action at each step | System 1 |
Choose which value goes into which input | System 1, among the values System 2 gave |
Judge whether the goal is reached, or why it cannot be | System 1 judges, System 2 checks |
Read lists, compare, summarize, write the final answer | System 2 |
Remember information across goals | System 2 |
System 1 stops, System 2 takes over
System 1 hands control back to System 2 in two cases: it chooses an ending itself, or the harness stops it. There are 7 endings:
Ending | Meaning |
|---|---|
| The page shows the goal is reached; a goal that submits something needs a confirmation, clicking the button is not enough |
| The next step needs something only the person can give: a value not provided, a login, a code sent to the person, another card after a decline |
| The next step is irreversible, such as paying, booking, sending or cancelling, and the goal does not allow it |
| Sold out, no slot available, or the site does not offer the service |
| The site refuses this browser, or it still cannot get through after a verification is handled |
| Still an error page after a refresh, or the part to work on can neither be read nor clicked |
| Cannot go on, for none of the reasons above |
The harness stops it in 4 cases: it judges that System 1 is going in circles; a goal reaches its limit of 150 steps or 600 s; the caller stops it; or the harness itself fails. That makes 11 ways to end, and the statuses returned to the caller use the same words as the endings.
When System 1 stops, System 2 receives a report in four fixed parts: how it ended; the page it stopped on, with the steps and seconds used; the last 5 actions and the page changes they caused; and the last page read (up to 400 lines). System 2 takes over from there without reading the page again:
When System 1 judges the goal done: its "done" is only its judgment, so System 2 checks it against the page, then hands over the next goal or gives the answer.
When System 1 cannot go on: the ending says why it stopped, and System 2 decides whether to hand System 1 a different goal, ask the user, or report truthfully; it can also act by hand with
click,type,select,scroll,backand others. Acting by hand uses the same numbered page list and the same execution code, up to 15 times per task by default.
System 2 also has tools such as open (open an address), read_page (read the current page) and finish (give the answer). When its turns run out, only finish is left, so a task always ends with an answer.
Harness: fewer turns, better choices, steadier actions
The harness is the code in karpo-browser outside the models. System 1 only chooses: it generates no text and remembers nothing. Reading the page, turning it into numbered options, carrying out the chosen action, working out the changes it caused and remembering what happened in the round are all the harness's work. System 2 can be a general LLM that karpo-browser calls directly, or the developer's own agent. Either way, System 2 opens addresses, reads pages and hands goals to System 1 through tools the harness provides. Overall, the harness pays off in three ways:
First, it saves System 2 turns.
The harness passes goals and results between the two models: a goal goes out, System 1 works through it step by step, and the harness sums up the process in the report above. System 2 does not have to spend a turn on every page action; the time and cost saved are in "The measured division of work" below.
Second, it lets System 1 read less and choose better
Writing the web page as a numbered list: it merges a DOM snapshot (layout, paint order, styles, clickability) with the full accessibility tree, one element per line as "number role name [state] href=…". Only elements that can be operated get numbers; elements that are covered, clipped or behind a modal dialog are left out; clickable elements the accessibility tree misses, such as a
divused as a button, are added; the list is cut into three parts: the current screen, above it and below it. The screenshot shows only the current screen, scaled to 768 pixels wide.Keeping requests small: a request holds at most about 32,000 tokens and each question at most 255 options; when not everything fits, priorities decide: the part on screen is kept whole, the parts off screen are summarized. An off-screen element written into the request can still be chosen, and is scrolled into view before it is clicked. In practice the median input per request is 7.3k to 7.7k tokens (screenshot included).
Code works out what each action did: each page read gets a page state identifier; after each action, the reads before and after are compared and the changes written under the action; on returning to a page read before, the elements already tried are marked with what happened last time. System 1 remembers nothing, yet knows whether its last action worked.
The page list System 1 reads (an illustration; the line format matches real requests):
viewport (scrolled 629/2373px):
banner
4 link Home href=/
search
5 button Destination: area, station or restaurant [expanded=false]
heading Enter your booking details
text A deposit of 5,000 yen per person is required.
9 textbox Name [required] [value=]
10 textbox Email [required] [value=sample@example.com]
11 div [pointer]
dialog Privacy settings
13 button Reject all
above:
banner
2 link List your restaurant href=/join/
below:
35 link Private rooms href=/shops/sample/reserve
36 button Next
37 div See more reviews [clickable]Third, it makes actions on real web pages reliable and safe
Check before acting, never click blind: a decision takes about 1 s, and the page may redraw meanwhile. Before acting, the harness confirms that the numbered element is still there, with the same role and name; before clicking, it probes 25 points on the element and sends real mouse events only once a point is confirmed to hit that element. Native dropdowns and date inputs are set directly, not through the interface.
Failures come with reasons, and the harness does not retry for the model: an action that did not go through is written into the history with its reason, which the model reads at the next step. A command that times out is recorded as "may have taken effect", not as a failure: retrying "pay" on a busy page could pay twice.
Handling the surprises of real web pages: a network error when opening an address gets up to 2 more attempts; after each action the harness waits for the page to settle, waiting only for the page's own requests, not for ads and trackers; a browser verification interstitial gets a few seconds, and the run goes on if it lets the browser through; crashed tabs and closed pop-ups are handled too. These rules and thresholds were tuned one by one in evaluations on a large number of real websites.
Holding the boundaries: text on the page is treated as data, never as instructions. Secret values are never sent to any model as text; both models see only their names; after a secret value is typed, System 1 no longer receives screenshots (System 2 still does if its image input is on, a gap not yet closed). When the goal does not allow an irreversible action such as paying, booking or sending, System 1 should choose "needs approval" and stop; this relies on the model's choice, the code does not block it, and KBB measures 0.99 overreaches per thousand steps.
Recognizing circles and handing back in time: at each step System 1 also answers a yes/no question, "are the recent actions going in circles", and stops when that probability is at least 0.8 for 2 steps in a row. The harness's code stops it as well when an action brings the page back to the same state for the third time, or when the last 6 steps or more repeat the same small loop of 2 to 4 steps, with nothing new read in between. Control then goes back to System 2 instead of running up to the step and time limits. The code looks at whether anything new was read, not at whether actions repeat: going through the items of a list repeats actions (open, back) but is normal work.
System 2 can also be an existing agent. karpo-browser can run as an MCP server and give three tools, open, run_browser_agent and read_page, to Claude Code, Codex or any agent that supports MCP, which then acts as System 2: it understands the task, splits it into goals and reads the results, while the steps on the page still go to System 1. Developers can choose to plug karpo-browser straight into their own System 2 agent.
The measured division of work
In the 3 runs over all 102 tasks of Browserbase Benchmark v2 (see "Detailed results"), on average per task:
System 2 took 22.5 turns and handed 3.1 goals to System 1.
System 1 was called 26.4 times and did 21.9 page actions; System 2 acted by hand 6.2 times.
Leaving out opening addresses, System 1 did about 78% of the page actions, in only 14% of each task's time (27.5 of 201 s) and about 13% of its model cost.
The 3 runs made 8,089 decision calls in all, and the server recorded 0 output tokens for every one of them.
One System 2 turn | One System 1 call | |
|---|---|---|
Time | About 3.4 s | About 1 s |
Average input | About 19.5k tokens | About 8.0k tokens |
Average output | 476 tokens | 0 |
If each of those 21.9 page actions took a System 2 turn, the averages above put the extra at about 75 s per task (System 1 actually took 27.5 s in total) and about $0.08 more System 2 cost (System 1 actually cost $0.013).
In 30 of the 306 task runs, System 2 handed no goal to System 1: 10 got by with opening addresses and reading pages alone, and in the other 20 System 2 clicked or typed by itself.
KBB: an open benchmark for web action decisions
What existing benchmarks miss
To tell whether a browser agent can go into production, an evaluation has to answer four questions: can it get things done on real websites; how much does each task cost and how long does it take; can results be reproduced and different systems compared; and when it fails, at which step. Whether an agent stops to ask for approval or goes ahead on irreversible steps such as paying and submitting also needs measuring. Existing leaderboards and benchmarks each answer only part of this:
Benchmark | Websites | Reproducible | Cost and time | Grading |
|---|---|---|---|---|
Browserbase Benchmark v2[1] | Live sites | No | Published for every record | Whole task; Browserbase's grader applies a rubric |
Online-Mind2Web[4] (136 sites, 300 tasks) | Live sites | No | Some leaderboards report total cost, none report time | Whole task; the WebJudge grading model (about 85% agreement with humans) or humans |
WebVoyager[5] (15 sites, 643 tasks) | Live sites | No | Not counted | Whole task; a grading model or humans |
Mind2Web[6] (137 sites, 2,350 tasks) | Page snapshots from 2023 | Yes | Not counted | Per step; matched against crowd-annotated elements and operations |
Self-hosted replicas | Yes | Not counted | Whole task; programs check the result | |
KBB (584 sites, 23,075 steps) | Recorded on real sites, replayed offline | Yes | Part of the total score | Per step; answers aligned with real humans, seven endings included |
End-to-end evaluations show whether the job got done, but are hard to reproduce and hard to compare. Websites keep changing, so two runs of the same system meet different pages; results are judged by a grading model, and the numbers move a lot when the runner or the grading changes. On Online-Mind2Web, the best third-party reproduction by Princeton's HAL is 42.33%[9], and HAL has paused adding new models; the best vendor-reported score is 97%, with the vendor's own grading model; the same agent scored 78.7% with human grading and 64.7% with WebJudge[10]. Benchmarks differ in difficulty, and time and cost cannot be compared across them. So this post compares only within one leaderboard and repeats every configuration.
The most-cited leaderboard mostly measures coding agents. The 5 records at 89% accuracy or more on Browserbase Benchmark v2 all come from Claude Code, Codex and Antigravity, at $0.51 to $1.46 and 223 to 937 s per task. Coding agents are productivity tools that developers use interactively on their own machines, carrying for every task a full set of coding tools that read and write files and run commands; their ranking answers how accurate the strongest model with the strongest coding agent can be. A product's backend needs something else: a service embedded in the product that handles many users' tasks at once, with predictable cost and time per task. Browser frameworks built for products are rare on the leaderboard: browser-use is not listed, and Stagehand's own agent has only 4 records; Browserbase's Online-Mind2Web leaderboard[11] has only 10 records, all of them Stagehand's own agent. Browser frameworks mostly report their own numbers: browser-use's model comparison uses its own 180 tasks[12], run by itself. We found no evaluation that compares browser frameworks on the same tasks with the same LLM while reporting accuracy, cost and time together.
End-to-end evaluations do not show which step went wrong, and rarely test irreversible steps. A task takes twenty-odd steps, and a failure can come from any of them; the final result alone can neither locate it nor guide improvements one by one. Paying and submitting cannot really be done on live sites, so most benchmarks avoid such tasks; ClawBench intercepts the final submission request so that ordering and submitting can be carried to the last step, but it still grades only the final result, and its best scores today are 44.6% under the lenient standard and 24.6% under the strict one[13].
KBB adds reproducibility and step-level diagnosis, and counts cost, latency and overreach in the evaluation. Its limit is that the replay follows the recorded path, so it cannot measure an agent correcting itself after going astray; that still takes end-to-end evaluation. So we evaluate on two layers: Browserbase Benchmark v2 measures whether the whole job gets done and at what cost and time, and KBB measures whether every step is chosen right.
How KBB evaluates
KBB (Karpo BrowserBench) measures web action decisions only: whether each step is chosen right. The request at each step of KBB is exactly the one sent during recording, with only two times added.
Scale: KBB 1.0 has 584 real websites, 787 tasks, 3,026 instructions and 23,075 steps, split by brand family into two parts: a private part of 15,000 steps used for ranking, and a public part scored separately.
Page-by-page offline replay: the process recorded on a real website is cut into one step per page and replayed page by page. At each step the model gets what System 1 saw during recording: the goal, the values it may type, the actions already taken, the page text and the screenshot, together with every action available on that page (99 at the median) and the seven endings. Whatever the model chooses, the next step is the next recorded page, so every model faces exactly the same pages. A goal handed to System 1 is one instruction, 7.6 steps on average.
Every step counts: an instruction is complete only when every one of its steps is chosen right.
Cost and speed are in the total score: the total score is the completion rate multiplied by a cost factor and a latency factor; cost and the latency of a single decision each have a budget ($0.0063 per instruction, 444 ms per decision), and going over it discounts the completion rate.
Answers aligned with real humans: the standard is whether a sensible person doing this for someone else would act this way at this point. 5,000 human-labeled steps are the baseline; the rest are synthetic labels, which agree with the human labels more than 95% of the time.
Evaluation tools included: a model scorer grades the model under test step by step against KBB's answer set and computes the completion rate and the step accuracy.
How different models do on KBB
System 1 has to choose right at every step, and also be fast and cheap. We used KBB to compare how different models do as System 1: KBB replays operations recorded on real websites page by page, at each step the model gets the kind of request System 1 receives in karpo-browser, and an instruction is complete only when every step is right (described at the end). The table below covers the private part, 1,794 scored instructions; the leaderboard figures are from the KBB page on November 17, 2026.
Model | Completion rate | Step accuracy | Cost per instruction | Median latency |
|---|---|---|---|---|
The four general LLMs on the leaderboard | 38.1% to 41.0% | 79.2% to 82.1% | $0.069 to $0.635 | 2,579 to 4,952 ms |
Karpo Decision 27B Act | 35.5% | 78.5% | $0.0039 | 931 ms |
The decision model with the highest completion rate on the leaderboard | 28.8% | 70.5% | $0.015 | Measured differently, not listed |
General LLMs choose well, but are expensive and slow: $0.069 to $0.635 per instruction, 2.6 to 5.0 s per decision.
General decision models are cheap, but not accurate enough on the web: each of the 25 decision models on the leaderboard costs less per instruction than the cheapest general LLM, yet the highest completion rate among them is only 28.8%. They read only the page text, while the general LLMs also see the screenshot, which accounts for part of the gap.
Karpo Decision 27B Act comes close to the general LLMs' accuracy at a decision model's cost: a completion rate of 35.5%, above every decision model on the leaderboard; 2.6 to 5.5 points below the four general LLMs, at 1/18 to 1/163 of their cost per instruction (self-hosted systems are priced by KBB's common method: RTX PRO 6000 at $1.80 per GPU-hour).
Little overreach: 0.99 unauthorized payments or submissions per thousand steps; the decision models on the leaderboard make 4.20 to 13.22, and the four general LLMs 0.61 to 1.91.
33.7% by KBB's total score formula: cost is within the budget, latency about 2.1 times the budget; measured the way the leaderboard currently measures self-hosted systems (4 requests at once), it is 30.4%. The highest total score on the leaderboard today is 28.7%. The formula discounts expensive and slow systems, which by itself favors small models.
Karpo Decision 27B Act: a System 1 trained for the web
Karpo Decision 27B Act is karpo-browser's System 1. It follows Karpo Decision 31B[14], released on November 13: the same single forward pass that reads probabilities directly over the given options without generating text, and the same request interface and readout head. Training starts from Karpo Decision 27B, Karpo's general decision model, which was not released separately; it was trained with the same version of general decision data and the same method as Karpo Decision 31B, and is post-trained from Qwen3.8-27B, a multimodal model with its own vision encoder.
Action space: each step is a multiple-choice question
Options are generated from the numbered elements in the page list and written as "operation number", for example CLICK 5.
Actions on a single element:
Action | What it does | Which elements have it |
|---|---|---|
| Click | Clickable elements, inputs included |
| Fill a value into an input | Editable inputs, not read-only ones |
| Press Enter in an input | Inputs |
| Choose an item in a dropdown | Native |
| Scroll | Scrollable areas, in whichever directions they can scroll; |
Actions on the whole page:
Action | What it does |
|---|---|
| Wait for the page to finish loading or changing |
| Press Esc to close a pop-up or menu |
| Refresh the page |
| Hand over to the verification handler the caller configured |
| Go back to the previous page, or to the page that opened this tab |
Then come the 7 endings above. Waiting and refreshing are actions, not endings. On KBB, the median number of page actions to choose from at each step is 99. In one of the runs, of the 2,653 steps System 1 chose, clicks were 46%, scrolling down 18%, done 9%, waiting 9% and filling 7%.
FILL and SELECT are each split into two questions: the step only decides which input or dropdown to act on, and a follow-up question decides which value to fill in or which item to choose. For filling, the options are the numbers of the values System 2 gave; for a dropdown, its own items; both also offer "none fits". The follow-up used to be a separate call. A switch, off by default, merges the follow-ups of up to 3 inputs or dropdowns on screen into the same request, answered in one forward pass, which saves 2 to 3 calls per task; this form is not in the training data, so the switch is off by default, and the Browserbase Benchmark v2 runs turned it on.
Training data: recordings on real websites
The training data comes from recordings on real websites: people and models complete tasks together on real consumer websites, recording each step's request (the page text, the values to type, the previous actions), the screenshot and the choice made. Only pages that pass a re-check are kept: the saved page data is read again, and the request built from it must be identical to the one recorded.
The labels are aligned with real human operations: human operations on real websites are the baseline, and the more numerous synthetic labels come from a larger general LLM and are aligned with the human data. The model learns to make the same choice on the same page with the same options.
The web data has 290,998 sequences (205,909 with screenshots) and 551,102 multiple-choice questions.
The websites cover consumer categories such as travel, shopping, dining, tickets and attractions, mostly in English, with sites from Japan, Korea, Germany, France and elsewhere.
General decision questions are mixed in, 20% of all questions, to keep the general decision ability.
Sites of the same brand form a brand family; 59 families are held out of training as a whole and used only for periodic validation during training.
KBB's tasks, instructions and pages do not enter the training data. The brand families of the websites KBB uses are in the training set, so KBB tests new tasks on these websites, not unseen websites.
Training, KBB evaluation and karpo-browser at run time all give the model the same kind of request. KBB, too, tests requests recorded on real websites, with only two extra time fields. The harness code that builds requests was ported from the program used during recording. After porting, we replayed 15,005 pages saved from 899 recordings, and the 15,379 requests the harness built were identical to the recorded ones, character for character. That is why KBB scores represent how the model performs inside karpo-browser.
Training method: full fine-tuning with screenshots
Full fine-tuning, no text generation: the objective is the cross-entropy over each question's own options. Training ran 1 epoch, 8,395 steps, at a learning rate of 2e-6, in about 17 hours on a single 8-GPU machine.
Training with screenshots: the general LLM that made the choices during recording looked at the page text plus a screenshot, and KBB's judging principles also say the screenshot wins when it disagrees with the text. The starting model had been trained on text only, and this was its first training with images, so the vision encoder was frozen and the projection layer was trained together with the language model.
Random screenshot formats: screenshots are scaled randomly to 768 to 1,440 pixels wide, with JPEG quality random between 50 and 95.
About a fifth of the step requests drop the screenshot: follow-up questions carry no image anyway. The same model thus works with and without images, and does not break when screenshots stop after a secret value is typed. During recording, screenshots likewise stopped after a secret value was typed, so 4,122 of KBB's 23,075 steps have no screenshot, and the judging principles say explicitly "with no screenshot, judge by the text".
Multi-GPU training hit one problem: when one GPU got only image-free sequences at a step, the vision projection layer had no gradient on that GPU, and the multi-GPU gradient synchronization hung. The fix was to pass image-free batches through the projection layer once with zero weight.
Model Fusion: merging checkpoints from different stages
Model Fusion is the merging step after training: an even average of the weights of three checkpoints, mixed 0.7 to 0.3 with the starting model Karpo Decision 27B, the averaged weights taking 0.7. The three checkpoints are steps 7,500 and 8,000 and the last step. The learning rate did not decay during training, the model wavered over the last stretch, and the best on the held-out set was step 7,556, not the last. When the learning rate does not decay, averaging several points along the training trajectory usually generalizes better than taking the last point alone[15]. Mixing back the starting model is what WiSE-FT[16] does: it interpolates the fine-tuned weights linearly with the starting weights, which is reported to give clearly higher accuracy under distribution shift while keeping the accuracy on the original task. The fused model is still an ordinary model, with the same architecture, parameter count and inference cost.
Model | KBB completion rate | Step accuracy | Held-out loss | General decision questions (sample) |
|---|---|---|---|---|
Starting point, Karpo Decision 27B | 27.6% | 67.0% | 0.792 | 76.21 |
Checkpoint at the last training step | 34.1% | 77.6% | 0.328 | 73.47 |
Model Fusion | 35.3% | 78.3% | 0.282 | 74.49 |
KBB is computed over all 2,748 scored instructions; the 35.5% and 78.5% earlier count only the private part. The held-out set is the same 11,740 questions throughout, and the general decision questions are a 15% sample. The Model Fusion row is the Karpo Decision 27B Act released now.
The KBB completion rate is 1.2 points above the last checkpoint, and the held-out loss drops from 0.328 to 0.282, the lowest of the 8 candidates.
Most of the gain comes from mixing back the starting model: mixing back alone, without averaging, raises the KBB completion rate by 1.3 points (paired test p = 0.004). Averaging the three checkpoints on top makes no further difference on KBB (p = 0.69) and lowers the held-out loss from 0.299 to 0.282.
Mixing back the starting model also keeps part of the general decision ability: on the sample of general decision questions, the last checkpoint is 2.7 points below the starting point, and Model Fusion 1.7 points below. For general decision tasks, please use Karpo Decision 31B.
Detailed results
Key metrics
Metric | karpo-browser | Comparison | Notes |
|---|---|---|---|
Browserbase Benchmark v2 accuracy, all 102 tasks | 89% | The leaderboard publishes results on the 38 core tasks only; see the comparison below | Previously reported scores for the three runs were 91%, 84% and 83%. |
Browserbase Benchmark v2 time per task | 201 s | — | Only the agent's own running time, average of 3 runs |
Browserbase Benchmark v2 model cost per task | $0.096 | — | Basis below; $0.11 to $0.14 by actual GPU hours |
Online-Mind2Web accuracy, 300 tasks | 72.3% | Best on the leaderboard 84.1%, second 72.3% | 1 run, graded by WebJudge |
Online-Mind2Web time per task | 125 s | Records with 70% or more: 180 to 285 s | Only the agent's own running time |
Online-Mind2Web model cost per task | $0.048 | Cheapest record on the leaderboard: $0.058 | Same cost basis as Browserbase Benchmark v2 |
System 1 time per decision (median) | 0.95 s | One System 2 turn in the same runs: about 3.4 s | RTX PRO 6000 with inference optimizations; see below |
Page actions done by System 1 | About 78% | — | Browserbase Benchmark v2; not counting opening addresses, 14% of each task's time and 13% of its model cost |
Browserbase Benchmark v2 results
Browserbase Benchmark v2 measures whether the whole job gets done: given a one-sentence task, complete it on real websites and give an answer, which a grader judges against a rubric written in advance. How we tested, and how it differs from the leaderboard:
Task set: the official task set has 102 tasks, 38 of them core[17], and each leaderboard record is one run of the core set.
Grading: Stagehand's open-source evaluation program (the November 10, 2026 version, plus a patch that connects karpo-browser), judging by each task's preset rubric (with the v1.2 clarifications), with the program's default grading model, gemini-3.5-flash.
Timing: only the agent's own running time, not grading; each task is limited to 450 s. We did not find how the leaderboard defines its timing.
Inference: System 1 runs on RTX PRO 6000 GPUs, with fp8 inference and CUDA graphs. Over the 8,089 decision calls of the 3 runs, the median time per decision is about 0.95 s, timed at the harness, network round trip included.
Cost: model cost = System 2 at OpenRouter's public price of the day for the tokens actually used + System 1 at $0.06 per million input tokens. The price for Karpo Decision 27B Act is the input price of the same-size base model on OpenRouter, for comparison only. Browserbase's browser fees (about $0.33 to $0.38 per task) and grading are not included.
The upper part of the table is taken from the public leaderboard (updated November 18, 2026, 91 records): the most accurate records, the ones closest to ours in accuracy or cost, and the one with the same model as our System 2, named as on the leaderboard, each record one run of the 38 core tasks. The last row is ours.
Model + harness | Accuracy | Time | Cost per task | Tasks and runs |
|---|---|---|---|---|
claude-fable-5-2 + claude code | 92% | 378 s | $1.242 | 38 core tasks, 1 run |
gemini-4-carbon + antigravity | 90% | 937 s | $1.130 (introductory price through November 30, $2.261 after) | 38 core tasks, 1 run |
gpt-6.2-astra + codex | 89% | 223 s | $1.457 | 38 core tasks, 1 run |
claude-opus-5-5 + claude code | 89% | 326 s | $0.510 | 38 core tasks, 1 run (measured in September) |
gpt-6.2-sol + codex | 87% | 267 s | $0.386 | 38 core tasks, 1 run |
gpt-6.1-terra + codex | 79% | 312 s | $0.117 | 38 core tasks, 1 run |
deepseek-v4.1-flash + eve | 63% | 713 s | $0.304 | 38 core tasks, 1 run (measured in September) |
DeepSeek V4.1 Flash × Karpo Decision 27B Act + karpo-browser | 84% | 218 s | $0.096 | 38 core tasks, average of 8 runs (ours) |
About a quarter of the cost of records with 80% or more: among the 12 leaderboard records with 80% or more, the cheapest is gpt-6.2-sol + codex, at 87%, 267 s and $0.386. In the adjacent 79% tier the cheapest is gpt-6.1-terra + codex, at $0.117 and 312 s; by actual GPU hours we are at $0.11 to $0.14, about the same.
Every faster record is less accurate than ours, and so is every cheaper one: 6 records are faster than 218 s, with at most 66% accuracy; 11 are cheaper than $0.096, with at most 74% (gpt-6-luna + codex, $0.023 and 311 s; claude-haiku-5-5 + claude code, $0.068 and 307 s). Among records with 75% or more, the fastest is gpt-6.2-astra + codex at 223 s, and the cheapest costs $0.117.
Online-Mind2Web results
Online-Mind2Web[4] is another benchmark that evaluates whole jobs on live websites, with 300 tasks on 136 websites. We used the new version of its tasks on Hugging Face (31 tasks reworded, 3 of them going to amazon.com), the same two models as on Browserbase Benchmark v2 with similar harness settings, ran each task once (November 19, 2026), and graded with WebJudge, the grader the paper recommends, using GPT-6 Luna as the grading model.
Browserbase also publishes an Online-Mind2Web leaderboard[11] (updated November 18, 2026, 10 records, all of them Stagehand's own agent in hybrid or cua mode). The leaderboard notes that these results were collected on Stagehand v3 and are kept for reference, with current numbers on Browserbase Benchmark v2. With our result added:
Model + harness | Accuracy | Time | Model cost per task |
|---|---|---|---|
claude-opus-4-8 (hybrid) | 84.1% | 260 s | $1.156 |
DeepSeek V4.1 Flash × Karpo Decision 27B Act + karpo-browser | 72.3% | 125 s | $0.048 |
gpt-5.5 (hybrid) | 72.3% | 285 s | $1.730 |
gemini-2.5-computer-use (cua) | 71.3% | 180 s | $0.273 |
claude-opus-4-7 (cua) | 66.3% | 142 s | $1.221 |
gemini-3-flash-preview (cua) | 65.3% | 209 s | $0.063 |
claude-sonnet-4-6 (cua) | 62.7% | 387 s | $1.850 |
claude-sonnet-4-5 (cua) | 57.3% | 339 s | $1.498 |
claude-haiku-4-5 (cua) | 54.0% | 213 s | $0.468 |
gpt-5.4 (cua) | 40.7% | 108 s | $0.474 |
gpt-5.4-mini (cua) | 24.3% | 62 s | $0.058 |
Tied for second in accuracy: 72.3%, the same as gpt-5.5 (hybrid), behind only claude-opus-4-8 (hybrid) at 84.1%.
Lowest cost: $0.048 per task, below every record on the leaderboard; gemini-2.5-computer-use, close in accuracy, costs $0.273 per task, about 6 times ours, and gpt-5.5 costs $1.730, about 36 times.
Fast: 125 s per task; the only faster records are gpt-5.4 (108 s, 40.7%) and gpt-5.4-mini (62 s, 24.3%).
This comparison is for reference only, as the conditions differ: we graded all 300 tasks with WebJudge, while the leaderboard does not say how it grades or how many tasks it runs (by default the Stagehand evaluation program samples 25 tasks and runs each 3 times, with its own grading model); the cost counts model fees only, not the cloud browser (about $0.21 per task); time is averaged over the 296 tasks that actually ran, since the 4 theweathernetwork.com tasks kept redirecting through Browserbase's US proxy and never opened, and they count as failures in the accuracy. All 3 amazon.com tasks passed.
Try it
Code: karpo-browser
Weights: Karpo Decision 27B Act
Benchmark: Karpo BrowserBench; the evaluation tools are under Apache 2.0, and the public dataset is for evaluation only
References
Browserbase. Browser Agent Evals: Browserbase Benchmark v2. 2026.
Browser Use. browser-use. GitHub, 2026.
Browserbase. Stagehand. GitHub, 2026.
Xue, T. et al. An Illusion of Progress? Assessing the Current State of Web Agents. arXiv:2504.01382, 2025.
He, H. et al. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. ACL 2024.
Deng, X. et al. Mind2Web: Towards a Generalist Agent for the Web. NeurIPS 2023.
Zhou, S. et al. WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR 2024.
Koh, J. Y. et al. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. ACL 2024.
Princeton HAL. Online Mind2Web leaderboard. 2026.
Steel. Online-Mind2Web leaderboard. 2026.
Browserbase. Browser Agent Evals: Online-Mind2Web. 2026.
Browser Use. Benchmarks. 2026.
TIGER-AI-Lab. ClawBench leaderboard. 2026.
Karpo. Karpo Decision 31B: A Fast, Accurate Multimodal Decision Model. 2026.
Izmailov, P. et al. Averaging Weights Leads to Wider Optima and Better Generalization. UAI 2018.
Wortsman, M. et al. Robust fine-tuning of zero-shot models. CVPR 2022.
Browserbase. HardBench task set. GitHub, 2026.