Walkcast LabUpdated · by Kunj Shah
Walkcast Lab. Why the browser agent failed, and the test that proves it.
Short answer: Walkcast Lab is browser agent testing for teams that build AI agents which drive a web browser. When a run fails, Lab turns the failure into a cause, the evidence behind it and a replayable regression fixture, then compares the base agent against a candidate over repeated trials using 95% Wilson score intervals. Nothing is scored by an LLM judge: causes come from the browser's own trace, and a cause the rules cannot explain is left marked unexplained instead of being guessed.
The problem with testing a browser agent
A browser agent is a program that cannot be unit tested. It reads a page, decides what to click, and the result depends on what the site returned, what the model decided, and what the page looked like three steps ago. When a run fails, the agent usually cannot explain itself, and the log is a wall of DOM snapshots with no clear cause.
The usual response is to watch the recording and squint. That does not scale, it does not survive the next release, and it tells you what happened without telling you what changed. Lab closes that gap: it captures the run as evidence, classifies the cause from that evidence, and writes the failure down as a test.
The vocabulary
A run
- One attempt by a browser agent at a task, with everything it did recorded as evidence.
A fixture
- A failed run frozen into something replayable: the task, the pages, and the point where it went wrong.
An experiment
- The same task run repeatedly against a base agent and a candidate, scored with 95% Wilson score intervals.
A cause
- Why the run failed, classified from the evidence. A cause the rules cannot explain stays marked unexplained rather than being guessed.
A comparison
- Base against candidate over repeated trials, so a fix that broke something else shows up as a regression.
Fault replay
- Re-running a recorded failure through the simulator to check whether a candidate agent still trips over it.
Browser agent evals, in one page
If you are building a browser agent, you already know the shape of the problem. You write an eval for a task, you run it, and one day it goes red. The hard part is never the red — it is working out why, proving the fix, and noticing that the fix broke a different task last week.
That is what Walkcast Lab is: the evals half of browser agent testing, built on the recorder rather than on an agent's own logs. A run that fails becomes a cause, the evidence behind it, and a fixture you can replay. When you compare two versions of the agent, you get the same task run repeatedly against both, with 95% Wilson score intervals, so a real improvement is distinguishable from a lucky run.
How a failure becomes a regression test
1. Run the agent
The agent drives a real browser against a real site. Everything it does is recorded: the pages, the elements, the requests, the responses.
2. Classify the cause
When the run fails, the cause is derived from the trace, not from the agent's own account of what it did. An agent that claims it clicked the right button is not evidence; the browser's trace is.
3. Freeze it as a fixture
The failing run becomes a fixture: the task, the starting state and the point of failure. It is replayable, so it can be run again against the next version of the agent without re-creating the conditions by hand.
4. Compare base and candidate
An experiment runs the task repeatedly against both agents and reports 95% Wilson score intervals, so a real improvement is distinguishable from a lucky run — and a fix that broke something else shows up as a regression rather than hiding in the average.
5. Replay faults in the simulator
Recorded failures can be pushed back through the fault replay simulator to check a candidate agent still trips over the same thing.
How this compares to Stagehand, Playwright and browser-use
Those are ways of running an agent. Stagehand is an open-source browser automation framework built on Playwright, driven by a mix of code and natural-language instructions; browser-use is an agent that drives a browser from Python. Both are excellent at getting an agent to act. Neither is about what happens after the run fails.
Walkcast Lab sits one step downstream. It works with any agent that drives a real browser, because it reads the recorder's own output rather than an agent's logs. If you build on Stagehand, Playwright or browser-use, Lab is the part that tells you whether your change fixed anything — it is the Stagehand alternative for the testing half of the job, not for the driving half.
Questions people ask
What is browser agent testing?
- It is testing an AI agent that drives a web browser: you give the agent a task such as 'sign in and open the invoices page', and you check whether it completed it, and if not, why. The hard part is the 'why'. An agent that fails usually cannot explain itself, so the failure has to be captured as evidence and classified.
What is browser agent regression testing?
- It is running the same browser agent task again after a change, and finding out whether the change fixed the task, did nothing, or broke a task that used to pass. It needs a recorded failure to replay against the new version — that record is the fixture, and classifying the cause is what turns a bug report into a test.
What is a regression fixture for a browser agent?
- A fixture is a failed agent run kept in a form you can replay: the task, the pages it visited, the state of the browser and the point where it went wrong. Replay it against the next version of the agent and you find out whether your fix worked, and whether it broke something that used to work.
How does Walkcast Lab compare two agent versions?
- With experiments. The same task is run repeatedly against a base agent and a candidate, and the results are compared using 95% Wilson score intervals, so a difference you can act on is separated from a difference that is only noise.
Does Walkcast Lab use an LLM to judge the agent's work?
- No. Every cause comes from the browser's own trace: which page was opened, what was typed, what was clicked and what the site answered. A cause the rules cannot explain is left marked unexplained instead of being guessed.
What is Stagehand and how does it compare?
- Stagehand is an open-source browser automation framework built on Playwright, used to drive web pages with a mix of code and natural-language instructions. It is a way to run an agent. Walkcast Lab is what you use after a run fails: it turns that failure into a cause, evidence and a replayable fixture, and compares agent versions over repeated trials.
Can Walkcast Lab test an agent that uses browser automation tools?
- Yes. The Lab reads from the recorder, so it works with any agent that drives a real browser, including agents built on Playwright, Stagehand or browser-use.
Read the Lab documentation
The Lab chapter of the README walks through runs, fixtures, experiments, compare and the fault replay simulator, with the commands to run them: Walkcast Lab on GitHub.