Browser agent testing: measure verified feedback, not fast clicks

• Published: 2026-10-27
• Author: Sarthak Agrawal
• Updated: 2026-10-28
• Reading time: 18 min read
• Canonical HTML: https://sassmaker.com/learnings/browser-agent-testing

This experiment compared UI-testing drivers and agent strategies without confusing fast clicks with a correct result. The retained runner runs bounded commands, records warm and cold timing, and requires a separate product-specific verifier before counting a run as passed.

The runner does not replace Playwright, Maestro, AXe, Jev, Browser Use, or your application tests. It gives those tools one measurement and evidence contract.

The measurements below were last run on 20 September 2026. They cover seeded local web and iOS Simulator journeys on one Mac, not production traffic or a universal ranking. This article brings the completed experiment into SaaS Maker; the standalone site is retired.

The retained evidence contains 26 measured journey arms, 118 verified completions across 129 attempts, seven additional probes and five seeded defect types. Eleven tools were repeatedly benchmarked, ten received bounded screens, and four encountered dated setup blockers. The current map contains 80 tools, including 55 source-reviewed entries without runtime rankings. The Pomerado source review was added on 27 October 2026; the historical measurements were not rerun.

Jump to the journey timings, other probes, experiment records, version pins, or Pomerado discussion.

Current benchmark snapshot

The first adapter tested Vaultwealth web and native iOS journeys. These are screening results, not universal tool rankings:

• Repeatable web regression — saved Playwright tests. Reliable baseline; no tested replacement met the 2× adoption target.
• Exploratory web navigation — Jev, with an independent verifier. Search passed 5/5 and detected the seeded stale-result fault, but login is unsupported because password inputs are excluded.
• Repeated semantic web action — Stagehand cached replay remains promising. Search replay passed 4/4 at a 2.201 s median with no repeat model calls after 25.544 s of authoring. This was promising screening evidence, not a 2× improvement.
• Repeatable iOS regression — saved Maestro flows. Slower in the tested login runs, but the more dependable retained baseline.
• Targeted iOS inspection and actions — AXe or XcodeBuildMCP. AXe login was faster in screening, but readiness, configuration and text-entry failures prevented general adoption.

Read the full screening report, round-two comparison, and the expanded web and native readiness screens before using the numbers in a decision. No tool has yet qualified as the overall winner.

The complete measured results

These are the original September measurements. A verified pass requires the independent verifier to accept the outcome. A fast unsuccessful attempt remains a failure, and an unavailable measurement is shown as —, not zero.

On a phone or narrow window, scroll each table horizontally to read every column. Tables can also receive keyboard focus for scrolling.

Coverage and cost

Journey timings, verified results and model usage

Medians and observed p95 use the timing basis printed under each mode. For the batched Codex arm, timings cover all attempts despite zero qualified completions; Browser Use timings cover successful verified passes. At these sample sizes, observed p95 is usually the maximum and is not a reliable population tail estimate. Stagehand authoring is separate from replay.

No candidate completed the planned 20 warm and three cold qualification runs across every journey. The five-run and four-replay results remain screening evidence.

Web: Login

Web: Edit + reload

Web: Search

Web: Adaptive login

iOS: Login

iOS: Search

Other measured probes

These seven probes cover text generation, diagnostics, individual actions and simulator readiness. They are separate from the 26 end-to-end journey arms; a readiness result does not establish saved-flow reliability.

What each experiment established

The ten retained experiment records include the numeric findings and their original qualification limits.

1. Saved web workflows

Status: complete screening. Arms: Playwright · agent-browser direct · agent-browser batch.

Playwright remained the baseline. agent-browser batching improved its own login time, but no candidate met the 2x adoption threshold.

• Login: Playwright 1.178 s; direct 2.714 s; batch 1.323 s medians; all 5/5.
• Edit: Playwright 1.898 s and direct 1.762 s medians; batch failed 0/5.
• Search: Playwright 2.768 s; direct 2.826 s; batch 2.649 s medians; all 5/5.

Original evidence.

2. Seeded web defects

Status: complete screening. Arms: Playwright oracle · agent-browser direct · agent-browser batch where viable.

The viable Playwright and direct arms detected all five seeded fault types with no clean-run false failure in the selected trials.

• Detected save failure, wrong persisted value, duplicate write, stale search and clipped control.
• The visual oracle detected 53.14% changed control pixels against a 2% limit.
• This validates fixed oracles; it is not a blinded AI discovery rate.

Original evidence.

3. Coding-agent stepwise versus batched interaction

Status: complete screening. Arms: Stepwise Codex · Batched Codex.

Stepwise passed 5/5. Batched qualified 0/5 and reduced total reported input mostly through cache, not uncached reasoning.

• Stepwise verified median 155.971 s; 5/5.
• Batched median attempt 123.684 s; only 2/5 functional completions and 0/5 within the recovery gate.
• Total input fell 56.6%, while uncached input fell 8.6%.

Original evidence.

4. Jev specialist with local Bonsai text helper

Status: complete screening. Arms: Jev + TypeSafe selection + Bonsai text · Fixed Playwright/backend oracle.

Authenticated search passed 5/5 and found the hidden stale-search fault. Login is unsupported because Jev excludes password inputs.

• Clean search median 11.101 s; observed p95 11.391 s.
• Seven TypeSafe calls plus three local Bonsai calls per run.
• Estimated TypeSafe spend $0.00313047 across five clean runs; about $0.00063 per run.

Original evidence.

5. iOS saved flows: Maestro versus AXe

Status: screening incomplete. Arms: Maestro saved flow · AXe batch.

AXe was much faster on warm login and search, but cold accessibility readiness and inconsistent text entry prevented replacement.

• Login: Maestro 32.964 s versus AXe 12.612 s medians; both 5/5.
• Search: Maestro 34.673 s versus AXe 15.279 s medians; both 5/5 with journey-specific AXe typing configuration.
• Three app-process relaunch probes returned an empty AXe tree until a Maestro hierarchy request.

Original evidence.

6. Local Bonsai 27B text-helper compatibility

Status: complete synthetic screening. Arms: Reasoning budget only · Explicit no-thinking template.

Both profiles passed 25/25 exact responses. Explicit no-thinking was 4.54x faster and used less sampled memory.

• Budget-only median 4.907 s; observed p95 8.943 s; sampled peak RSS 9.252 GB.
• No-thinking median 1.082 s; observed p95 1.193 s; sampled peak RSS 7.305 GB.
• This was a text-helper probe, not a browser-agent or defect-detection result.

Original evidence.

7. Round-two local browser tools

Status: complete screening. Arms: Stagehand uncached · Stagehand cached · Browser Use · Playwright CLI · Chrome DevTools MCP.

Stagehand cache was the most promising replay result, but missed the 2x gate. Browser Use was slow and 4/5. Chrome DevTools MCP was useful for diagnostics only.

• Stagehand uncached 26.951 s median; 5/5. Cached 2.201 s median; 4/4 after 25.544 s authoring.
• Browser Use 90.730 s median among successes; 4/5.
• Chrome DevTools MCP diagnostic bundle 0.826 s median; Lighthouse snapshot 1.243 s.

Original evidence.

8. Additional native candidates

Status: bounded probe. Arms: XcodeBuildMCP · XCUITest feasibility · Detox feasibility.

XcodeBuildMCP replaced the formatted cash field in 5/5 probes but no save/relaunch journey was qualified. XCUITest and Detox were setup-blocked by checkout structure.

• XcodeBuildMCP action median 3.834 s; semantic observation median 1.343 s.
• Maestro/debug setup still took about 35 s.
• No claim is made that XCUITest or Detox failed as tools.

Original evidence.

9. Expanded saved-web driver screen

Status: complete screening. Arms: Puppeteer Core · Selenium WebDriver · WebdriverIO · Nightwatch · Taiko.

All five candidates completed five independently verified clean searches and detected the stale-search fault. Puppeteer was fastest, but did not meet the 2x adoption gate against Playwright.

• Workflow medians: Puppeteer 2.081 s; WebdriverIO 2.958 s; Selenium 3.633 s; Nightwatch 5.425 s; Taiko 23.850 s.
• Every candidate passed 5/5 clean runs and detected the stale-search mismatch through the backend/UI oracle.
• TestCafe 3.7.6 hit the bounded 60 s setup timeout; Cypress 16.1.0 had no isolated application binary. Neither is counted as a tool failure.

Original evidence.

10. Expanded native agent-tool readiness

Status: complete bounded screen. Arms: ios-simulator-mcp · Mobile MCP · fb-idb.

Both MCP servers started and found the dedicated simulator. ios-simulator-mcp returned the accessibility tree through isolated idb; Mobile MCP listed 27 elements and its temporary device agent was removed afterward.

• ios-simulator-mcp 2.1.0 exposed 17 tools and returned a 6,940-character accessibility description.
• Mobile MCP 1.0.4 exposed 32 tools and listed 27 elements with device agent 0.0.26.
• These are readiness probes, not saved-flow latency or defect-detection results.

Original evidence.

Exact observed versions and environment

Machine: Apple M5 Pro, 48 GiB RAM, Darwin 27.0.0. Browser: Dedicated Chromium at 1280x800. Simulator: iPhone 16 Pro, iOS 26.5.

These pins describe the historical runs. They do not establish results for later releases. The model and runtime hashes remain visible so the retained evidence can be traced precisely.

When the agent builds a reusable tool

Website-to-API and MCP generators belong in the comparison too. Pomerado takes a website task, explores it with a browser and generates a source-code integration served through MCP. Its architecture documentation separates the agent that builds the integration, Guardian’s authoring-time review, and the runtime that executes it.

The open-source core is MIT-licensed. Authoring uses OpenAI models and requires a paid model API; running a generated integration makes no model calls. That makes it relevant to repeated workflows, but a generated tool is still an executor. It does not independently prove that a saved change persisted or that every returned row satisfies a filter.

For this kind of tool, measure the first successful build separately from repeat execution. Vary the inputs, test empty results and pagination, verify persisted outcomes independently, and record what happens when the site changes. Count repair work, retries and human intervention alongside execution time.

Pomerado is source-reviewed only here. It has no local execution, speed, reliability or defect-detection result in this experiment. Its upstream claims are not benchmark findings. Stagehand’s measured cached replay is a related example of separating authoring from repeat execution; its results do not establish Pomerado’s performance.

Clone and prove the runner works

The core has no package dependencies. It needs Node.js 22 or newer.

gh repo clone sarthakagrawal927/agent-testing
cd agent-testing
pnpm test

node bin/agent-testing.mjs validate \
  --manifest fixtures/good.manifest.json
node bin/agent-testing.mjs run \
  --manifest fixtures/good.manifest.json \
  --out artifacts/quickstart
node bin/agent-testing.mjs summarize \
  --receipt artifacts/quickstart/receipt.json

The generated artifacts/ directory is ignored. A run never overwrites a non-empty output directory.

Add your product

Keep the retained runner beside the application rather than copying its code into the application. Add a directory such as adapters/my-product/ containing:

• one manifest per journey and candidate;
• bounded commands for reset, readiness, workflow, and verification;
• any product-specific orchestration scripts;
• a README describing local startup, seed data, tool pins, and known limits;
• an opt-in patch only if the app needs debug hooks that cannot live in normal test code.

A minimal manifest looks like this:

{
  "schemaVersion": "fleet.agent-testing.manifest.v1",
  "id": "my-product.web.search.playwright",
  "platform": "web",
  "journey": "dynamic-search",
  "candidate": "playwright-saved",
  "sample": "screening",
  "workingDirectory": "../../../my-product-worktree",
  "runs": { "warm": 5, "cold": 0 },
  "phases": {
    "reset": {
      "command": ["node", "scripts/reset-benchmark-data.mjs"],
      "timeoutMs": 10000
    },
    "readiness": {
      "command": ["node", "scripts/check-local-readiness.mjs"],
      "timeoutMs": 10000
    },
    "workflow": {
      "command": ["pnpm", "exec", "playwright", "test", "tests/search.spec.ts"],
      "timeoutMs": 120000
    },
    "verification": {
      "command": ["node", "scripts/verify-search-result.mjs"],
      "timeoutMs": 10000
    }
  }
}

workingDirectory is resolved relative to the manifest. Use a clean isolated product worktree and a seeded non-production backend. Commands must be arrays, must not invoke a shell, and must have bounded timeouts.

The verifier is intentionally separate from the workflow. It should check the displayed result, persisted backend state where relevant, and explicit visual evidence. A driver exiting successfully is not proof that the product behaved correctly.

Run a useful comparison

1. Choose one real journey and give every candidate the same starting state, interactions, assertions, and screenshot checkpoints.
2. Start with five warm screening runs. Measure cold startup separately.
3. Run clean and individually broken variants. Hide each defect from an exploring agent and let the independent verifier classify the outcome.
4. Record initial authoring cost separately from repeat execution.
5. Promote a candidate to 20 warm and three cold runs only when screening is promising.
6. Adopt only when it is at least 2x faster, or cuts model usage by at least 50% without becoming slower, with no missed seeded defects or observed reliability regression.

Report median and observed p95 alongside verified completions, model calls, spend, retries, interventions, setup effort, and workflow repair time. At these sample sizes, tail latency and reliability estimates remain preliminary.

Vaultwealth example

The retained Vaultwealth adapter applies only to a clean, non-main worktree:

node adapters/vaultwealth/prepare.mjs check --target /path/to/vaultwealth-worktree
node adapters/vaultwealth/prepare.mjs apply --target /path/to/vaultwealth-worktree

After preparation, follow the generated local-agent-eval/fast-testing/README.md in that worktree. Do not merge the experiment patch into the application merely to run the benchmark.

Retained evidence and tool map

The full datasets now live alongside this article in SaaS Maker:

• 80-tool catalogue: source links, evidence labels, costs, version pins or concrete applicability boundaries, across seven categories.
• Experiment records: all 26 measured journey arms, verified pass denominators, median and observed p95, model usage, fault outcomes, seven probes and ten experiment write-ups.
• Observed version pins: the exact tool and environment versions recorded for the historical runs.

The categories separate saved web automation, adaptive browser agents, agent evaluation suites, remote browser infrastructure, visual verification, iOS and mobile tools, and local model helpers. Remote infrastructure and source-reviewed alternatives have fit boundaries, not invented local scores.

“Benchmarked” means repeated local execution. “Screened” means a bounded local probe. “Setup-blocked” records a named prerequisite or bounded failure without judging tool quality. “Source-reviewed” records official-source research without a runtime ranking.

The original repository retains the complete source, reports and reproduction commands. This article and the linked snapshots are a retained research publication. Run a fresh comparison when a concrete application journey or material tool change justifies it; the existence of another tool is not a reason to restart the entire benchmark catalogue.