How we benchmark AI agents in a browser: 50 tasks, open harness, raw results

We benchmark AI agents in a browser by giving the same agent the same 50 public web tasks twice: once through the Browz MCP server and once through a plain browser MCP, with the same model, prompt and limits. A program scores every answer. In the run recorded on 2 November 2026 the agent completed 46 of 50 tasks through Browz and 38 of 50 through the plain browser. The task list, prompts, runner, scorer and raw logs are public, so anyone can check the numbers or record their own.

This post explains why we measure, how the tasks and rules were chosen, what each metric means, the results, and how to rerun it.

Why measure at all

Claims about agents and browsers are easy to make and hard to check. What matters to someone building an agent is simple: when the agent is sent to a real site, does it come back with the right answer, or does it stop at a block page? A benchmark turns that into a number with a date on it, and an open harness turns the number into something you can verify rather than trust.

We had a second reason. Browz adds safety controls around an agent: its own profile, its own key, allowed sites, sign-ins it cannot read. Controls only help if the agent still gets its work done, so we wanted to see completion measured on ordinary tasks.

The tasks

The 50 tasks are fixed, short and read-only, in the style of public web-agent benchmarks such as WebVoyager: look something up, search a site, read a price, a title or a headline.

• 32 tasks are on sites that rarely challenge automated browsers: an encyclopedia, a paper archive, code hosting, developer documentation, dictionaries, course catalogues and a news site.
• 18 tasks are on well-known sites that do challenge automated browsers: search engines, online shops, travel, jobs, recipes, sports, film and housing sites.

Every task reads public pages. No task signs in, signs up, buys anything, or submits a form other than a site's own search, and no account is used. Each task has a known answer or an expected format.

The two setups

• Browz MCP: the MCP server that ships in Browz. Each task gets a new throwaway session, its own profile with a consistent device fingerprint, wiped when the task ends.
• Plain browser MCP: an open-source MCP server that drives a standard browser, with a new isolated browser context for each task. It is the setup most agents start with, which makes it the fairest baseline. The kit names the exact server and version.

Both ran headless, from the same computer and internet connection, side by side.

The agent and the rules

The agent was Claude Sonnet, run through Claude Code in non-interactive print mode. For every task in both setups:

1. The same prompt. Only the first two lines differ, naming the setup's tools. The agent must end with ANSWER: … and BLOCKED: yes|no.
2. The same limits. At most 30 turns and 5 minutes per task.
3. One way to the web. Only that setup's MCP tools are allowed. The agent's own web search, fetch, shell and file tools are switched off, so it cannot reach a site any other way.
4. Stop at a block. The agent is told to stop if a site blocks it, not to work around it. Browz does not solve CAPTCHAs, and in this benchmark no person steps in, so a run that a site stopped counts as blocked.

How each run is scored

A program scores each run, not a person. A task succeeds when all of these hold:

• the answer passes the task's check: exact, one of a list, a pattern, a number range, a price, or non-empty text;
• the browser actually visited the task's site;
• the agent did not report a block;
• the run did not time out.

A run counts as blocked when the agent reports a block page, CAPTCHA or bot check, or when a block-page signature appears in what the browser showed and the answer is wrong. Harness problems, such as a browser that did not start or a usage limit, are not task results: they are set aside and retried.

The metrics

Medians are taken over all 50 tasks, finished or not.

The results

Recorded on 2 November 2026 with Claude Sonnet:

The full table is on the agent benchmark page. The difference sits almost entirely on the sites that challenge automated browsers: the same agent, with the same instructions, reached the content far more often through Browz, where each task had its own profile with a consistent device fingerprint.

A second test: bot-defence checks

On 10 November 2026 we ran a separate set of 12 checks on pages protected by Cloudflare Turnstile and challenge pages, DataDome, PerimeterX (HUMAN), Akamai Bot Manager, reCAPTCHA v3 scoring, CreepJS and Fingerprint bot detection. The same Claude Sonnet agent ran each check headless, with the same prompt and limits, in a fixed order. An agent in Browz passed 12 of 12. The table is in the bot detection section.

Reading any single run

These are one run of each setup, on one day, from one computer and one connection. Sites change their pages and their defences, bot protection varies with the IP address and the time of day, and models take different paths on different days. A run is a measurement of that run. That is exactly why the harness is public: your network, your computer and your day are a second measurement, and we would like to see it.

Rerun it yourself

Everything is in the MIT-licensed kit, browz-agent-benchmark on GitHub.

Check our numbers without a browser

Re-score the raw logs of the published run, with no browser and no account:

It reads every run record and prints the results table, plus a per-site table.

Run both setups

You need Node.js 22 or later, Claude Code, your own Claude credential in the environment, Google Chrome for the plain browser setup, and Browz (the free plan is enough):

That runs both setups on all 50 tasks side by side and prints the score. The kit starts Browz with a new, empty data folder, so your own profiles are never read or changed. For a quick trial, pass a run folder and a few task ids after the app path with --tasks, and the runner's --dry-run option prints the settings and prompts without running anything.

Send us your result

Open an issue with the "Benchmark result" template in the kit's repository and attach your run folder. The template asks for your region and kind of network, never an IP address. The kit redacts keys, tokens, email addresses, paths and addresses from every record before writing it.

Frequently asked questions