AI Browser Automation That Survives a Redesign

TL;DR: AI browser automation is sold as an agent that reads a page and decides what to click. Measured on 300 realistic tasks across 136 live sites, the best of those agents finished 61.3% of them and the most popular open source one finished 30.0%, against the roughly 90% quoted from an easier benchmark. Plain Playwright scripts are the opposite: perfect until a class name changes, then zero. The design that actually survives is a hybrid, where fixed steps do the work and a model is called for exactly three jobs: writing the steps once, repairing a locator that stopped matching, and reading a page that has no structure. This post explains that split and shows it as configuration in Heym's Playwright node.

Key Takeaways:

• Benchmark success rates for browser agents are measured on tasks far easier than the ones you would automate. The same agents drop from about 90% to 30% when the tasks get realistic
• Determinism is a feature, not a limitation. A step list that cannot improvise is also a step list that a hostile page cannot redirect
• Self-healing belongs in business automation more than it belongs in testing. A healed test can hide the regression it existed to catch; a healed automation just keeps the process running
• Pay for the model once, not per run. Generated steps written back into the workflow turn an AI-authored flow into a fixed one from the second run onward
• Session reuse beats logging in again. Restoring a saved storage state with a visible check and a fallback login is the pattern the Playwright docs and experienced practitioners both land on

Table of Contents

• What Is AI Browser Automation?
• Why Browser Agents Break More Often Than the Benchmarks Suggest
• Why Plain Scripts Break Too
• The Three Jobs AI Should Actually Do in a Browser
• How Selector Healing Works
• Healing a Test Is Not the Same as Healing an Automation
• What Each Approach Costs Per Run
• Why Fixed Steps Are Harder to Hijack
• Staying Logged In Without Retyping a Password
• Building a Page Watcher That Alerts You
• When You Should Use a Full Browser Agent Instead
• Getting Started
• FAQ
• References

What Is AI Browser Automation?

Definition: AI browser automation is any system in which a language model participates in driving a real web browser, whether by deciding each action at runtime, assisting inside the browser you are using, or filling specific gaps in a script you wrote. It is distinct from plain web scraping, which fetches and parses pages without a browser session, and from browser testing, which drives a browser in order to assert that a product behaves correctly.

That definition is broad on purpose, because the term currently covers three designs that behave nothing alike.

Most articles about AI browser automation are about the first row. Most production work belongs in the third. The rest of this post is about why, and what the third row looks like when you build it properly.

Heym, the open source AI workflow automation platform this blog belongs to, ships the third design as a node. I co-founded Heym and work on its workflow executor, which is where browser steps, credentials, and sandboxing actually meet, so the design calls described below are ones we argued about while building the Playwright node rather than features I am reviewing from the outside. That is also a commercial interest, so I have kept every claim narrow enough for you to check: each one is either in the node documentation, in the source, or cited to a third party. Everything here reflects the product and the research as of July 2026.

What is Heym? Heym is an open source, self-hosted AI workflow automation platform with a visual canvas. Workflows are built from nodes such as AI Agent, LLM, RAG, HTTP, Condition, and Playwright, and run on your own infrastructure, so credentials, page content, and browser sessions never leave your environment.

Why Browser Agents Break More Often Than the Benchmarks Suggest

The marketing numbers for browser agents come from WebVoyager, where scores near 90% are common. In 2025, a research team from Ohio State University and UC Berkeley argued those numbers were an illusion and built a harder benchmark to prove it. Online-Mind2Web uses 300 realistic tasks across 136 live websites and grades them with human evaluation.

The results, from Table 2 of the paper:

Notable fact: Browser Use reports roughly 90% on WebVoyager and scores 30.0% on Online-Mind2Web. The authors also found that several recent agents do not beat SeeAct, a baseline released in early 2024.

The paper's own summary is blunt: the results "depict a very different picture of the competency of current agents, suggesting over-optimism in previously reported results."

Now turn a percentage into an operations problem. A price check that runs twice an hour is about 1,460 runs a month. At 61% that is roughly 570 failed runs a month, and unlike a failed test, each one is a business event that silently did not happen. At 30% it is over a thousand. No amount of retry logic fixes a system whose base rate is a coin flip, because the retries are just as likely to wander somewhere new.

This is not an argument that browser agents are bad. They are genuinely good at the thing benchmarks measure: a novel task, described once, on a site nobody prepared for. That is exploration. Exploration and unattended repetition are different jobs.

Why Plain Scripts Break Too

The honest counterpoint is that traditional automation is not reliable either, it just fails on a different schedule.

A CSS selector describes how a page is built. Front-end teams change how pages are built constantly, and none of those changes are announced to you. A renamed utility class, an extra wrapper div, a component library upgrade, or an A/B test variant is enough to make .btn-primary--large match nothing. The step throws, the run stops, and every step after it is skipped.

So the practical failure modes look like this:

The third row is not a compromise between the first two. It is a different claim about where model calls earn their cost.

The Three Jobs AI Should Actually Do in a Browser

Here is the framework the rest of this post defends. In a production browser automation, a language model should be doing exactly three jobs, and nothing else.

Job one: write the steps, once

Describing a flow in plain language is faster than reading a DOM and hand-picking selectors. Heym's Playwright node exposes this as an aiStep: you give it instructions such as "dismiss the cookie banner, then open the pricing tab" and an LLM credential, and it returns ordinary Playwright actions. With saveStepsForFuture enabled, those generated actions are written back into the workflow node as savedSteps after the run, so every later run replays them directly. The model authored the automation; it is not running it.

Job two: repair a locator when one stops matching

This is where a model has real leverage, because the information needed to fix a broken locator is on the page right in front of it. Detail in the next section.

Job three: read a page that has no structure

Extracting a delivery date from a paragraph of prose is a language problem, not a browser problem. The right shape is to have the browser grab the text with getVisibleTextOnPage and hand it to an LLM node for structured extraction, so the model never touches navigation at all.

Everything else, which is to say the clicking, typing, waiting, and navigating, should be fixed instructions. This is the same principle behind agentic workflows generally: give the model authority over the decisions that genuinely need judgment and none over the ones that do not.

The framework also answers the adjacent question of how to expose browser automation for AI agents rather than as one. In Heym you attach the browser workflow to an AI Agent node as a callable tool, so the agent decides whether the browser step is needed and the step itself still runs as fixed instructions. The agent gets the judgment, the browser keeps the determinism.

How Selector Healing Works

Auto heal is the concrete version of job two. In Heym's Playwright node the mechanism is deliberately narrow:

1. A selector-based step fails. Heym retries it. It fails a second time.
2. Heym captures the current page HTML, truncated to 50,000 characters, plus a screenshot when you enabled sendScreenshot.
3. That context, the failed step, and the original instructions go to your chosen model with a system prompt that asks for one alternative locator.
4. The model is constrained by a strict JSON schema to return a single replacement step, and is told to prefer robust locators such as role=button[name='Login'] or text=Exact button text, with CSS as the fallback.
5. The step retries with the new locator.

The threshold is two failures rather than one on purpose. A first failure is usually a timing problem: the element exists but the page had not finished rendering it. Waking a model for that would spend a model call to solve a problem an ordinary retry already solves, so the heal path only opens once a plain retry has been ruled out.

Two further limits matter more than the feature itself.

The first is that healing only applies to click, type, fill, hover, and selectOption. Those are the actions where substituting a different locator is a safe repair: you wanted to press the submit button, and the submit button moved. Heal is deliberately not offered for navigation or data extraction, because an "alternative" there is not a repair, it is a different result.

The second is that the preferred output is a role-based or text-based locator, not another CSS selector. That is not a stylistic preference, and it is not our opinion either. Playwright's own locator guidance says to prioritize role locators because they are "the closest way to how users and assistive technology perceive the page," and states plainly that "CSS and XPath are not recommended as the DOM can often change leading to non resilient tests."

The reason holds outside testing too. The label on a button is a product decision, reviewed by designers and translated by writers. The class name around it is an implementation detail that a component library upgrade can rewrite without anyone noticing. Healing toward the first and away from the second means the automation comes out of a failure more durable than it went in, which is the opposite of how brittle systems normally age.

Three things that only show up once you run this

Documentation describes the happy path. These are the details that decide whether a browser automation survives its first month, and all three are visible in the node's behavior rather than in its feature list.

An AI step needs to know whose credential it is spending. In Heym, a step that calls a model runs against a short-lived execution token minted for the workflow's user, which is what stops an unattended run from quietly borrowing someone else's LLM key. A scheduled run is fine, because the scheduler executes as the workflow owner. A genuinely ownerless run, such as an anonymous public entry point, is not, and the node fails with an explicit error instead of silently skipping the step. If you plan to expose a browser automation publicly, put the AI step behind an owned workflow and keep the public surface deterministic.

Headless is not always your choice. In a container with no display server, the node forces headless mode regardless of the setting, because the alternative is a confusing X server error at runtime rather than a clear one at design time. Practically, this means the headed mode people use to debug flaky flows locally is a local-only tool. Debug headed on your machine, ship headless.

The heal sees a slice of the page, not the page. The HTML sent for repair is capped at 50,000 characters. On a heavy application page that is not the whole document, so a heal on an element buried far down a huge DOM is less reliable than a heal near the top. The practical mitigation is the same thing that makes the rest of the flow reliable: navigate to a focused page or view before the fragile interaction, rather than driving a giant single-page application from its root.

Healing a Test Is Not the Same as Healing an Automation

Search for self-healing browser automation and nearly everything you find is about test automation: cloud test platforms, test frameworks, test agents. The QA community has also been arguing about whether it is a good idea at all, and the objection is a serious one. A test exists to fail when the product changes. An AI that quietly rewrites a broken locator until the suite goes green can erase exactly the signal the suite was built to produce.

That objection is correct, and it does not transfer to business automation, because the two systems have opposite goals.

Key principle: In a test, healing risks lying to you about the product. In an automation, the page was never the point. The invoice still needed downloading.

This is why auto heal deserves to be a default consideration in a workflow tool and a contested one in a test runner. It also explains the design choice to keep the healed run visible: the step is repaired, the run continues, and the execution record shows the heal happened, so a person can still notice that a site is drifting. Pair it with observability and a heal becomes a maintenance signal rather than a hidden patch.

What Each Approach Costs Per Run

Cost is the argument most often skipped in browser agent write-ups, and it is where the hybrid separates itself hardest.

The mechanism that makes the third row possible in Heym is small and specific. After a run, generated aiStep output is persisted into the workflow definition as savedSteps on that step. Run two does not ask the model what to do; the actions are already there. You paid for authoring once, and you pay again only on the days a page changes.

Put arithmetic on it. Take a flow of twenty browser actions running four times an hour, which is roughly 2,900 runs a month. An autonomous agent reasons about each action, so that is on the order of 58,000 model calls a month, every month, forever. The same flow built as fixed steps with one AI step costs one authoring call, then nothing, until the page changes. Even at a generous one breakage per week, that is about four model calls a month against 58,000.

Those two numbers do not need a price attached to make the point, because the ratio is four orders of magnitude and it holds at any token price. It also compounds in the wrong direction for agents: page complexity raises the per-run call count, so the sites most worth automating are the ones an agent is most expensive on. The broader discipline of keeping model spend proportional to the value of the decision is covered in AI agent cost optimization; browser work is simply an unusually clear case, because most of the actions in a browser flow require no intelligence whatsoever.

Why Fixed Steps Are Harder to Hijack

There is a security argument for determinism that is stronger than the cost one, and 2025 and 2026 supplied the evidence.

In October 2025, Brave demonstrated prompt injections carried inside screenshots: text that is effectively invisible to a human, read by the assistant through OCR, and executed as instructions. It followed their earlier disclosure of indirect prompt injection in Perplexity Comet. Their conclusion generalizes past any single browser: the attacks "boil down to a failure to maintain clear boundaries between trusted user input and untrusted Web content."

In early 2026, researchers at the University of Washington's Allen School tested seven agentic browsers and reported a successful cross-origin data theft attack against ChatGPT Atlas in Agent Mode, with the preconditions present in others. Their framing is the sentence to remember:

"The same-origin policy is reduced to the strength of an agent's defenses against prompt injections."

The structural reason is simple. If the model reads the page and then decides the next action, page content is control flow. Hidden text becomes an instruction. This is not an exotic edge case: prompt injection is ranked LLM01, the top entry in the OWASP Top 10 for LLM Applications, and a browser agent is the version of it with the widest blast radius, because the browser is already authenticated as you. We covered the attack class itself in depth in prompt injection.

A fixed step list changes the shape of the problem. The page cannot append a step that is not in the list, because nothing consults the page about what to do next. Two honest caveats belong next to that claim. First, extracted content is still untrusted data, so if you feed page text into a downstream model that can act, you have reopened the door and need guardrails there. Second, the heal path does read the page, which is exactly why it is constrained to returning one locator for one of five interaction actions under a strict schema, rather than a free-form plan.

We also learned this one the hard way on our own product. Heym's Playwright node has a Run Code mode for arbitrary Python, and a researcher reported that it amounted to remote code execution for any authenticated user. The fix was not a patch to the code path but a change to the default: custom code is now off unless an operator explicitly sets HEYM_PLAYWRIGHT_CUSTOM_CODE_ENABLED=true, and when enabled it runs in a throwaway hardened container with no Docker socket, no backend secrets, dropped capabilities, and resource limits, failing closed if that container is unavailable. Step-based automations were never affected, which is the third reason we tell people to stay in Steps mode. The same containment philosophy shows up in our background coding agent design, where the credential that can publish never enters the environment that runs generated code.

Staying Logged In Without Retyping a Password

Most useful browser automation happens behind a login, and this is where naive implementations quietly become both fragile and dangerous. Typing credentials into a form on every run means the password exists in the automation, the run is slow, and you will eventually trip a bot check or a multi-factor prompt.

The pattern practitioners converge on is session reuse. It is the documented Playwright approach, and it is also what the highest-voted answer in a 2026 r/AI_Agents thread on the best browser agent recommends: save the storage state after a real login, reuse it, and put strict waits and a hard cutoff around the flaky parts.

Heym's node implements that as four settings rather than four scripts:

1. playwrightAuthEnabled restores cookies or a full Playwright storageState before any step runs
2. playwrightAuthStateExpression says where that state comes from, typically a global variable such as $global.authState written by an earlier run
3. playwrightAuthCheckSelector names an element that only exists when logged in, checked right after the first navigation
4. playwrightAuthFallbackSteps run only when that check fails, performing a real login and handing control back to the main flow

The result is a login that happens rarely instead of constantly, and a run that fails with a clear auth error rather than silently scraping a logged-out page. When you also enable network capture, the node returns networkRequests, cookies, localStorage, and sessionStorage from the run, which answers a question people ask repeatedly in automation forums: how to get at the JSON a page fetched rather than the HTML it rendered.

Building a Page Watcher That Alerts You

Here is the whole framework as one small workflow. What it does is ordinary: on a schedule it opens a pricing page, pulls the plan prices out of it, and posts to Slack only when something looks wrong. What matters is where the model sits inside it.

Be clear about what you are looking at. This is not a demo of self-healing, because self-healing has no visible output when it works. It is a page watcher, and auto heal is the property that keeps it alive between redesigns rather than the thing it produces. You would only notice the heal by reading the execution history.

{
  "heym": true,
  "nodes": [
    {
      "id": "watcher_setup_note",
      "type": "sticky",
      "position": {
        "x": 380,
        "y": 540
      },
      "data": {
        "label": "SetupNote",
        "note": "### Before you run\n- Open CheckPage and pick your own LLM credential for the AI step.\n- Replace the navigate URL with a page you control or are allowed to automate.\n- Pick a vision capable model if you turn on sendScreenshot.\n- Add a Slack credential on AlertTeam or swap it for email.\n- Auto heal only repairs click, type, fill, hover, and selectOption steps."
      }
    },
    {
      "id": "watcher_cron",
      "type": "cron",
      "position": {
        "x": 60,
        "y": 240
      },
      "data": {
        "label": "EverySixHours",
        "cronExpression": "0 */6 * * *"
      }
    },
    {
      "id": "watcher_browser",
      "type": "playwright",
      "position": {
        "x": 380,
        "y": 240
      },
      "data": {
        "label": "CheckPage",
        "playwrightMode": "steps",
        "playwrightHeadless": true,
        "playwrightTimeout": 45000,
        "playwrightSteps": [
          {
            "action": "navigate",
            "url": "https://example.com/pricing"
          },
          {
            "action": "aiStep",
            "instructions": "Dismiss the cookie banner if one is visible, then open the section that lists plan prices.",
            "credentialId": "",
            "model": "gpt-5.5",
            "saveStepsForFuture": true,
            "autoHealMode": true,
            "sendScreenshot": false,
            "aiStepTimeout": 30000
          },
          {
            "action": "getVisibleTextOnPage",
            "outputKey": "pageText",
            "timeout": 3000
          }
        ]
      }
    },
    {
      "id": "watcher_extract",
      "type": "llm",
      "position": {
        "x": 720,
        "y": 240
      },
      "data": {
        "label": "ExtractPlans",
        "model": "gpt-5.5",
        "systemInstruction": "You extract pricing data. Return JSON with a plans array of name and monthlyPrice, and a boolean changedFromExpected set to true only when a price is missing or clearly wrong.",
        "userMessage": "$CheckPage.results.pageText",
        "jsonOutputEnabled": true
      }
    },
    {
      "id": "watcher_gate",
      "type": "condition",
      "position": {
        "x": 1060,
        "y": 240
      },
      "data": {
        "label": "NeedsAttention",
        "condition": "$ExtractPlans.json.changedFromExpected === true"
      }
    },
    {
      "id": "watcher_alert",
      "type": "slack",
      "position": {
        "x": 1400,
        "y": 120
      },
      "data": {
        "label": "AlertTeam",
        "channel": "#site-watch",
        "message": "Pricing page looks off: $ExtractPlans.json.plans"
      }
    },
    {
      "id": "watcher_ok",
      "type": "output",
      "position": {
        "x": 1400,
        "y": 380
      },
      "data": {
        "label": "AllGood",
        "outputSchema": [
          {
            "key": "plans",
            "value": "$ExtractPlans.json.plans"
          },
          {
            "key": "checkedAt",
            "value": "$executionId"
          }
        ]
      }
    }
  ],
  "edges": [
    {
      "id": "watcher_e1",
      "source": "watcher_cron",
      "target": "watcher_browser"
    },
    {
      "id": "watcher_e2",
      "source": "watcher_browser",
      "target": "watcher_extract"
    },
    {
      "id": "watcher_e3",
      "source": "watcher_extract",
      "target": "watcher_gate"
    },
    {
      "id": "watcher_e4",
      "source": "watcher_gate",
      "target": "watcher_alert",
      "sourceHandle": "true"
    },
    {
      "id": "watcher_e5",
      "source": "watcher_gate",
      "target": "watcher_ok",
      "sourceHandle": "false"
    }
  ]
}

Read it against the three jobs. The navigate and getVisibleTextOnPage steps are fixed, so they cost nothing and cannot be redirected. The single aiStep covers the part of the page that keeps moving, with saveStepsForFuture so it stops calling the model after the first run and autoHealMode so it repairs itself when the button moves again. The extraction is a separate LLM node reading text, with no browser access at all. The Condition node decides whether a human hears about it.

So there are two different alerts in play, and it is worth keeping them apart. The Slack message fires when the content is wrong, which is the job. The heal fires when the page changed, which is maintenance, and it shows up in the execution record rather than in your Slack channel. A workflow that pinged you every time a class name moved would be the thing this design exists to avoid.

If you would rather start from something already published, the Playwright Visual AI Monitor template takes the screenshot route instead: it captures the page on a schedule and has a vision model look for layout breakage, which catches the class of problems that text extraction cannot see. Both belong to the same family of jobs in our AI workflow automation examples catalog.

When You Should Use a Full Browser Agent Instead

The hybrid is not the answer to everything, and pretending otherwise would be the same overclaiming this post opened by criticizing.

Reach for an autonomous browser agent when:

• The task runs once. Authoring steps for a flow you will never repeat is wasted effort. Let an agent improvise.
• You do not know the site. Research across dozens of unfamiliar pages is exploration, and exploration is what agents are actually good at.
• The path genuinely varies per run. If the correct sequence depends on what is found halfway through, a fixed list is the wrong abstraction, and you want an AI agent with the browser as one of its tools.
• A human is watching. Interactive use changes the arithmetic completely. A 61% success rate is fine when a person can retry, and unacceptable when nobody is awake.

Stay with fixed steps plus targeted AI when the flow repeats, runs unattended, touches an authenticated account, or has a consequence when it silently does the wrong thing. Most business automation is in that second group, which is the whole reason this design deserves more attention than it gets.

Getting Started

If you want to try the hybrid design rather than read about it:

1. Pick one flow you already automate or do by hand weekly, ideally one that has broken at least once.
2. Build it in Steps mode first, with no AI at all, and confirm it passes on a good day.
3. Add a single aiStep for the one part that keeps changing, with save steps and auto heal enabled.
4. Attach a cron trigger and let it run for two weeks before touching it again.
5. Read the execution history. Every heal is a page telling you it drifted.

Heym is open source and self-hosted, so the browser runs on your infrastructure and the page data never leaves it. That property matters more here than in most automation categories, because browser automation by definition handles session cookies and authenticated pages. If workflow automation is new to you, start with what is AI workflow automation, or browse the template library for a running start.

FAQ

What is AI browser automation? AI browser automation is any system where a language model participates in driving a real browser. It covers three different designs: an autonomous agent that decides every click from a goal, an AI-assisted browser where a sidebar assistant acts on the current page, and a scripted automation that calls a model only for specific jobs such as writing the steps or repairing a selector that stopped matching. The three fail in very different ways, so the label alone tells you almost nothing about reliability.

Are AI browser agents reliable enough for production? Not on their own for repeating business processes. On Online-Mind2Web, a 300-task benchmark run on 136 live websites, the best agent tested finished 61.3% of tasks and Browser Use finished 30.0%, against the roughly 90% those agents report on the older WebVoyager benchmark. A process that runs 200 times a day at 61% leaves about 78 failures a day for a human to clean up. Agents are strong for one-off exploration and weak as an unattended production runtime.

Why does browser automation keep breaking? Because CSS selectors describe how a page is built, not what it means, and front-end teams change how pages are built constantly. A class rename, a wrapper div, or an A/B test variant is enough to make a selector match nothing. The automation does not degrade gracefully; it fails at the first missing element and every downstream step is skipped.

What is self-healing browser automation? Self-healing means the automation repairs its own locator when an element cannot be found. In Heym, after a selector-based step fails twice, the page HTML and an optional screenshot go to a language model, which returns one alternative locator, preferring role-based and text-based locators such as role=button[name='Submit'] over CSS. The step retries with that locator. Healing covers click, type, fill, hover, and selectOption, the actions where an alternative locator is a safe substitution.

Is AI browser automation safe to point at untrusted pages? It depends entirely on who decides the next action. If a model reads the page and then chooses what to click, hidden text on that page can rewrite the plan. Brave documented this against Perplexity Comet and Fellou in October 2025, and University of Washington researchers reproduced cross-origin data theft against ChatGPT Atlas in Agent Mode in early 2026. A fixed step list is not immune to bad data, but the page cannot add a step that was never in the list.

Does AI browser automation cost money on every run? Only if you designed it that way. An autonomous agent calls the model on every step of every run, so cost scales with runs times steps. In a hybrid design the model call happens when you author the steps and again only when a selector breaks. Heym writes generated steps back into the workflow as savedSteps, so the second run and every run after it replays fixed actions with no model call at all.

References

• Xue et al., An Illusion of Progress? Assessing the Current State of Web Agents, arXiv:2504.01382, 2025. Introduces Online-Mind2Web, 300 tasks across 136 websites; dataset and code
• Brave, Unseeable prompt injections in screenshots, October 2025
• Brave, Agentic Browser Security: Indirect Prompt Injection in Perplexity Comet, August 2025
• Roesner and Kohlbrenner, Agentic Browsers and the Same-Origin Policy, Paul G. Allen School, University of Washington, 2026
• OWASP, Top 10 for LLM Applications, prompt injection ranked LLM01
• Playwright, Locators, on preferring role locators over CSS and XPath
• Playwright, Authentication and storage state
• Heym documentation, Playwright node