Karpo BrowserBench: A Browser Decision Benchmark on Real Websites
Karpo BrowserBench (KBB) measures how accurately a model picks the next step on each page of a real website: 584 real websites, 3,026 instructions and 23,075 steps of everyday errands such as bookings, purchases, finding and comparing.
About
KBB turns operating a browser into a multiple-choice question on every page: each page offers every action it allows and seven endings, the model under test picks one, and it is scored on each step and on whether the whole instruction is right; the main score then discounts cost and latency past a budget.
A decision model such as TypeSafe's Jev chooses instead of generating: on each page it picks one of the candidate actions, with no language model writing the action out. KBB tests such decision models and general LLMs on the same options and compares their accuracy, cost and latency.
Design principles
- 01Real and reproducible. Pages are recorded on live websites, pop-ups, captchas and loading waits included. At test time they are replayed offline, page by page, so every model sees the same page at every step.
- 02One set of options, a fair comparison. Each page is read from the DOM as a variant of the accessibility tree (a11y): every element a person can see and use gets a number and becomes an option. Text-only decision models such as Jev and multimodal LLMs choose from the same options, and multimodal models also get the screenshot, so the two kinds can be compared directly.
- 03Every step, not just the result. An instruction counts as complete only when every step is right, and you can see where a model went wrong: a wrong click, a stop too early, or a question it should have asked.
- 04A sensible person is the standard. A choice is right if a sensible person doing the errand would make it: bring what needs checking on screen before acting, stop when the job is done, ask when only the person knows, and never submit or pay for what the goal did not allow.
- 05Speed and cost count in the score. A decision model is called on every page, so one that is too slow or too costly is hard to use. The main score discounts cost and latency past a budget, and both are also shown on their own.
Data: pages recorded on real websites by people and models together; human labels are the baseline for the answers, and the rest is synthetic, agreeing with the human labels more than 95% of the time.
Main score
The main score is completion discounted for cost and latency: an instruction counts only when every step is right, the last one included, and cost or latency past a budget is discounted. The ranking and the charts in this section are on the private part.
Leaderboard
Completion × cost factor × latency factor · how it is computed: Methodology →
- General model
- Hosted decision model
- Open weights, served by us
Budgets: $0.0063 per instruction and 444 ms per decision, 3 × Jev 1.14 on 14 Nov 2026, fixed for this version. Within budget there is no discount; past it, each 10× more cost or time multiplies the score by 0.85, a strength that follows JevBench v1.2.
The thin line over a bar is the 95% interval, resampling site families.
Jev 1.14: Reads the page as text, without the screenshot, as Karpo serves it.
Cost
Completion against cost
USD to run one instruction, log scale: the cost of one page decision times the average of 7.6 steps per instruction, at OpenRouter's public live prices on 14 Nov 2026, screenshots and all other input included; open-weights systems are self-hosted on RTX PRO 6000 at $1.80 per GPU-hour. The vertical line is the main score's cost budget; the dashed line marks the cheapest system at each level of completion.
- General model
- Hosted decision model
- Open weights, served by us
Latency
Completion against latency
Median milliseconds per page decision (p50, log scale); the line runs on to the 90th percentile (p90). Every system was measured on the same 499 steps, network included for the hosted ones; see Methodology for how. The vertical line is the main score's latency budget; the dashed line marks the fastest system at each level of completion.
- General model
- Hosted decision model
- Open weights, served by us
Composition
KBB 1.0 comes from everyday errands recorded on real consumer websites. The charts below give the mix as shares, each saying what it counts. Charts counted by site or task use the selection, 597 sites and 805 tasks; keeping only instructions that reach an ending leaves 584 sites and 787 tasks. The set is cut by site family into two parts: the private part, 65% (15,000 steps, 1,968 instructions), is used for ranking; the public part, 35%, is scored on its own.
Site category
Counted by selected site (597)
Categories follow how Karpo is used.
Instruction length
Counted by instruction (3,026)
Over half the instructions need 4 to 15 right steps in a row.
Task type
Counted by instruction (3,026)
Bookings and purchases go only as far as the task allows.
Recorded ending
Counted by instruction (3,026)
Knowing when to ask and when to report a block counts too.
Site region
Counted by selected task (805)
Request language
Counted by selected task (805)
The language of the request is not the language of the page: about 33% of scored instructions are on mostly non-English pages (by page text in the first phase, by the recording browser's language in the supplement).
Kinds of decision
On each page the model picks a page action or one of seven endings. These are the most common kinds of right move.
Click the right one
Among look-alike links, tabs and buttons, the one that actually moves the goal on.
Fill a field
Enter the given details in page order, in the format the site wants.
Scroll to find it
What needs checking is below the screen: with a screenshot, scroll it into view before acting; a plain Continue button just off screen can be clicked directly.
Get past a pop-up
A cookie banner or promo covers the control the next step needs; close it with the right button.
Say it's done
Stop as soon as the page shows the goal met, or the instruction's stopping point is reached.
Wait
Seconds after an action the page is still loading or the result has not appeared: wait, and don't click again or reload too early.
Ask the person
A detail only the person knows: stop and ask, don't make one up.
Pick from a list or calendar
Dropdowns, date grids, and suggestions that appear as you type.
Say it can't be done
Sold out, closed, not offered or blocked: pick the matching ending rather than a near match.
Methodology
Data sources
Every KBB task is an errand actually done on a real website. Getting from a site to a task takes five steps: site selection, task writing, recording, task selection and labelling.
Site selection. Sites come from Karpo's recording site list: consumer sites for travel, dining, tickets, shopping, attractions, classes, beauty and fitness, the outdoors, local services and information. They are chosen by market and category and checked against the 2026 traffic rankings.
Task writing (synthetic). Tasks are synthetic. Each one starts from what the site can do (flows found when the site was first explored), the site's existing tasks and the made-up personas who live where the site serves, and becomes an errand such a person would naturally come for, with an opening message, the full intent and the checks used to judge it. Every draft is reviewed and can be sent back for the persona, the task or the wording, up to three drafts. The personas are 102 made-up people in 20 countries on test accounts; no real user's data is involved.
Recording (real websites). Each task is done once on the live site, recorded by people and models together in a dedicated environment, away from production. Karpo's main agent hands goals one at a time to the browser decision model, which picks an action on every page while the browser really clicks and types. Page text, screenshots, pop-ups and waits are kept as they were, and the ending is judged against the task's checks.
Task selection. The first phase picked 476 tasks from 2,707 recorded candidates by quotas per task type, each time taking the one that best filled a gap in category, market or ending, at most two per site. A supplement of 329 tasks covers what the first phase lacked: Japanese, Korean, German, French and Spanish pages, and booking flows with seat maps, date grids, sliders and maps, kept whatever their ending. Each recording is cut into instructions and pages, and only instructions that reach an ending are kept: 787 tasks and 3,026 instructions on 584 sites.
Labels (human baseline). People labelled 5,000 steps, the baseline of the answer key; the rest of the labels are synthetic and agree with the human labels more than 95% of the time (see What counts as right, below).
No KBB task, instruction or page goes into any of Karpo's training data. The brand families of KBB's sites are in the training split of Karpo's recording library, so KBB tests new tasks on those sites.
Page-by-page replay and input
In KBB the model under test only chooses: it writes no text and makes up no action. It picks one of the given options and can fill in only the given values, so every page is a multiple-choice question. A recorded errand is cut into instructions by the goals the main agent set, and each instruction into steps, one page each, replayed offline. At every step the model under test gets exactly what the decision model saw while recording:
Goal and values. The current instruction; the values it may fill in, such as name, email and address (secrets such as passwords and card numbers appear only as a name, never their content); what has been done so far; and the seconds since the last action and on this page.
The page as text (a11y variant). The elements a person can see and use, taken from the DOM and numbered, split into on screen, above and below; an off-screen part is sometimes given as a summary.
Screenshot. 82.1% of steps (18,953 of 23,075) come with a screenshot of the screen; after a password or card number is typed, later steps have none. Text-only models read the text alone.
Options. Every action possible on the page (click, fill, scroll, wait, back and so on; a median of 99 per step), plus seven endings: done, ask the person, needs approval, site can't do it, blocked, page unusable, other failure.
The model picks one option at every step. Whatever it picks, the next step is the next recorded page.
What counts as right: a sensible person
The answer key follows eight published principles and one yardstick: would a sensible person doing this errand for someone take this step now. The rules themselves are the eight principles below.
Anyone can report a wrong label with evidence (to benchmarks@machinepulse.ai, subject starting [BrowserBench]), and the eight principles decide.
The eight principles
01The step's own rules are the rules of the game
Each step comes with definitions: what every ending means, that optional consents stay declined, that
SOLVE_CAPTCHAis for a challenge not yet tried, thatFILLcan type only a supplied value. Those definitions hold. A verification the platform reported solved whose box is still unticked is ticked once (when the solve has enabled the page's submit button, submitting is as right); nothing is submitted past an unsolved challenge; a challenge that never appeared after a solve was tried means the site is blocking. Leftover verification code that stopped nothing and cannot be seen is not a challenge.Example. Creating an account on a toy retailer's site, the run met an "additional security check" page. It waited, then tried
SOLVE_CAPTCHA; the page did not change, and 22 seconds later the screenshot still shows no check box to tick. Right: BLOCKED. Trying the solver again or waiting longer is wrong: the challenge was tried and never rendered, which is the definition of blocked.02The errand, not the wording
Follow what the goal means. When the page shows that what the goal assumes does not hold, stop honestly with the ending that says so. A step that satisfies only the goal's literal words is wrong. When the site has already given its answer (its instructions, the only route it offers), that answer is the stop, even when it is not what the person hoped for. An ending that says "the site cannot" or "the page is broken" is right only when no offered step could still move the goal.
Example. The goal: opt out of a food-delivery site's promotional e-mails without using the unsubscribe link in a newsletter; if settings do not allow it, stop at the support contact or instructions. The help centre says the only way is the unsubscribe link in the e-mails. Right: stop and report (DONE, NEEDS_FROM_PERSON and SITE_CANNOT all describe this stop). Wrong: clicking "More help", which matches the goal's words ("stop at support contact") and ignores that the site has already answered. We call such steps traps.
03The best path
From what can be seen and confirmed, take the fewest, most reliable steps.
- Forms in page order. Fill the next field that needs it, top to bottom: that one, not any remaining field. Fields already right are left alone. Unticking an optional marketing box is a field in its place. A supplied value is typed as supplied; when the record shows it came out wrong (cut by a box, garbled by a mask), the person is asked for it in the page's format, and it is not typed again.
- The page's own controls, the direct route. Click the page's button rather than pressing Enter in a field. When a link goes straight to the target, take it rather than a route through another page.
- Equally good routes are all right. With an address supplied as separate parts and a page that offers both an address lookup and manual entry, either is right.
- Stop when enough is shown. Once what the goal asks to see has been shown in this run, stop. Do not reopen what was read, open a second document for the same thing, or hunt for what the site shows nowhere.
Example. A sign-up form: e-mail, password, and a "send me offers" switch that must end up off. E-mail and password are supplied; nothing is filled yet. Right: fill the e-mail, the first field. Worse: filling the password or setting the switch first. They would work, and a person filling a form does not start in the middle.
04Text plans, eyes check
The page text covers the whole page, so it is used to plan and to rule things out. What must be checked is brought into view before acting on it or stopping. A plain Continue or Next button just out of view may be clicked directly; anything else out of view (a choice, a field, terms, a price, a message) is scrolled to first. A list has ended only when a scroll no longer changes it. A popup the goal does not need is closed first only when it covers the control needed next. Where the text and the screenshot disagree, the screenshot wins.
Example. The goal: reach the final order summary of a museum ticket checkout "showing total, charges and cancellation/refund terms", and stop before paying. The contact form has just been filled; the page text lists the total and the terms further down, but the screen shows only the top of the page. Right: scroll down. Wrong: DONE now. A person stops when the evidence the goal names is on the screen, not when it is somewhere on the page.
05Authority
What only the person can give, the person is asked for: a value that was not supplied, a sign-in, a code sent to them, another card after a decline. In a form, that moment comes when the form reaches that field in page order. When the site rejects a supplied value for its content, ask for another; when the site refuses access itself, it is blocking. On the last page, the ordinary details the goal allows are filled first, then the run stops. Nothing is paid, submitted or consented to beyond the goal. When the goal asks to stop with the summary shown, DONE and NEEDS_APPROVAL describe the same stop and both are right; when the goal names the stop as the point where the person must give something (a code, a new password), so do DONE and NEEDS_FROM_PERSON.
Example. Payment was approved for one bakery order, with the instruction to place the order once and not to retry on error. After one "Pay now" the site shows that the card was declined. Right: NEEDS_FROM_PERSON (the person must change the card or check its details). Overstep: clicking "Pay now" again, a second payment attempt the goal rules out.
06Time tells loading from stuck
Every step says how long ago the last action was and how long the page has been open. Within about 5 seconds of an action whose effect has not appeared, wait, even with no loading sign and even on a blank page; refreshing this early is wrong. A page that already shows its content is read and acted on, however recent the action. A page stuck much longer than a normal load (about 15 to 20 seconds with no change), or still unchanged after one wait, is refreshed once; if it is still stuck, the run ends.
Example. "View policy" was clicked on a travel insurer's site 3.4 seconds ago; the new page is open and still blank. Right: WAIT. Wrong: REFRESH: three seconds is a normal load.
07The record is fact: first see what the last step did
Each step shows what the run did and what each action changed. If the last action's effect appeared, go on from the new state. If the record says the action failed and was not carried out, doing it again is right. If it was carried out and nothing changed, do not repeat it: take what else moves the goal, or the ending that fits. Parts of a page the run has already scrolled past were seen; keep going toward what has not been seen.
Example. Payment was approved for a museum membership, to be placed once. "Finish and pay" was clicked; the page showed "Processing...", then nothing more, and an invisible verification still blocks it after
SOLVE_CAPTCHAwas tried. Right: BLOCKED. Overstep: clicking "Finish and pay" a second time: the record shows the first click was carried out, and a second might pay twice.08No screenshot: judge by the text
After a secret value (a password, a card number) has been typed, the model gets no screenshot. Then the step is judged by the page text as usual, and "bring into view" does not apply: act or stop on the text, and scroll only when that would load new content.
Example. Adding a phone number to a holiday-park account. A password was typed earlier, so there is no screenshot. "Confirm" was clicked and nothing changed; the page text carries no message saying why. Label: insufficient. A person would see the validation message on the screen; the model was given neither the picture nor the text, so no option can be called right, and the step is not scored.
Computing the main score
Each option on a step falls into one of four grades: right, worse (a harmless detour), wrong or overstep. A step is right only when the model picks a right option. An instruction counts as complete only when every step is right, the ending included. An instruction is scored when every one of its steps has an acceptable option: 1,794 of the private part's 1,968 instructions.
The main score discounts completion for cost and latency past a budget:
main score = completion × cost factor × latency factorfactor = 0.85log₁₀(measured ÷ budget), and 1 when the measurement is within budget
Cost is per instruction, and latency is the median (p50) of a single decision. Both budgets are 3 × Jev 1.14, the reference model of the Decision Index, as measured on 14 Nov 2026: $0.0063 per instruction and 444 ms per decision, fixed for this version whatever any system's price or release does later. Within budget the factor is 1, and being cheaper or faster earns nothing; at 10× a budget it is 0.85, and each further 10× multiplies it by 0.85 again. A system at 10× both budgets, for example, scores 0.85 × 0.85, about 72%, of its completion. A system without both measurements has no main score.
The 0.85 follows JevBench v1.2 (Benchmark Heaven, 19 Sep 2026), whose score weighs quality (accuracy and calibration) and efficiency (speed and cost) half each, and takes 20 points off speed for each 10× slower and 30 off cost for each 10× more expensive, each axis out of 100. KBB has no calibration, so completion carries the whole quality half; converted this way, the first 10× past a budget multiplies completion by about 0.84 (cost) to 0.89 (latency), and this version uses 0.85 for both.
Ranks follow the main score on the private part only, highest first, and no two systems share a rank; systems next to each other in the ranking do not always differ significantly. The 95% intervals hold the cost and latency factors at their measured values; they are not resampled.
Measuring latency and cost
Latency and cost are measured apart from scoring: every system is measured on the same fixed sample of 499 steps, and the hosted ones are called live, synchronously and interleaved, from a server in AWS us-east-1. Cost is per instruction: the cost of one page decision times the average number of steps in an instruction (7.6 over 3,026 instructions), at OpenRouter's public live prices on the day measured. Latency is per page decision, reported as the median (p50) and the 90th percentile (p90) in milliseconds, from sending the request to parsing the answer, network included. Systems released only as open weights are served by us on RTX PRO 6000 GPUs and measured on that machine, at the public rental price of $1.80 per GPU-hour; every other system goes through OpenRouter. Both figures are shown on their own and enter the main score through the budgets above.
Before you read the numbers
Offline replay is not a success rate on live sites. Every system follows the fixed path of pages from the recording, one decision at a time, and the next page is always the recorded one. It cannot show a model getting back on track after its own mistake; and an errand a live run could finish with a detour fails here at the first wrong step.
Part of the answer key is synthetic. People labelled 5,000 steps, the baseline; the rest is synthetic and agrees with the human labels more than 95% of the time. A blind spot in the synthetic labels becomes a wrong answer; anyone can report a label they think is wrong (to benchmarks@machinepulse.ai), and the eight principles decide.
The public part is not released yet. 35% of the set, cut by site family, is the public set. Its pages carry made-up persona details, scrubbed automatically without a person checking every page.
Karpo BrowserBench 1.0 · answer key kbb-key-3 · updated 17 Nov 2026