AgentWorks documentation

Browser Automation: Durable Selectors & Discovery

This doc describes the browser-specific rules injected into execution, learning, harden, and review prompts when a workflow has a browser MCP available (playwright, agent-browser skill, or a CDP port).

The problem

Browser automation agents interact with elements through two channels:

1. Accessibility tree — via browser_snapshot, which returns a YAML tree of roles + accessible names + current state (disabled, expanded, selected). Each element gets an ephemeral ref (@e1, e68) usable only within that snapshot.
2. Direct DOM — via browser_evaluate for reads, and the usual browser_click / browser_type / browser_navigate for writes.

Two failure modes this doc addresses:

• Refs bleed into saved scripts. If a learn-code step's main.py bakes click @e3, the next run's snapshot assigns e3 to a different element and the script fails. Refs are session-local.
• The a11y tree misses elements. Custom <div> buttons with onclick, dropdowns inside React portals, autocomplete options, form inputs lacking role/label — none show up in the snapshot. Agents flying blind either guess selectors from memory or give up.

The core rule

Selectors persisted to main.py must be DETERMINISTIC across future runs. A deterministic selector resolves to the same element on every replay — across browser restarts, page rebuilds, deploys that rename auto-generated classes (css-8xy3zb), React key changes, re-hydration. Anything that depends on session state (refs) or on unstable DOM shape (nth-child chains, auto-gen class names) is NOT deterministic and will silently match the wrong element tomorrow.

Refs (@e1, e68, "ref": "abc123") are session-local identifiers the browser tools assign per snapshot. They are valid only for the immediate next tool call in the same session. Any ref hardcoded into main.py is a bug.

Two equally valid paths to a deterministic selector:

• Path A — snapshot + act. browser_snapshot gives you role + accessible name + widget state. Pick a locator from the priority list, then act via individual tool calls (browser_click, browser_type, browser_select, browser_navigate) OR via Playwright code (browser_run_code with page.getByRole(...), page.locator(...)). Tool-call style is a debugability preference, not a durability dimension.
• Path B — DOM probe via eval. Run the canonical read-only probe below via browser_evaluate (Playwright) or agent-browser eval to get a structured inventory. Use when the a11y snapshot misses elements (custom <div> buttons, portal/popover children, form inputs the tree skips).

Both paths terminate in a durable locator expressed in the persisted script. browser_run_code using Playwright's locator API (page.getByRole('button', { name: 'Continue' }).click()) is durable and is the right shape for chained multi-step interactions. browser_evaluate for inspection is allowed; browser_evaluate that hand-rolls document.querySelector for an action and then writes that selector into the script is fine IF the selector is in the durability tier below — avoid structural CSS chains that break on DOM rearrangement.

Durable-selector priority

When writing locators, agents must pick the highest-priority hook that uniquely identifies the element:

1. data-testid / data-test / data-cy / data-qa — ideal, rare on production sites.
2. Hand-written semantic id or name — e.g. #panAdhaarUserId, #loginPasswordField. Skip auto-generated ids: radix-_rN_, mat-mdc-*, React :rNN:, UUID-shaped 8-4-4-4-12 — all rotate across rebuilds.
3. aria-label — very durable when present.
4. Role + accessible name — page.get_by_role("button", name="Sign in").
5. get_by_label(...) / get_by_placeholder(...) / get_by_text(...).
6. Structural CSS / XPath with nth-child chains — last resort; flag in learnings.

This priority mirrors Playwright's own recommendation but is explicitly enforced in prompts so code-gen doesn't default to fragile nth-child paths.

The DOM probe

When the a11y snapshot is insufficient (custom divs, portal dropdowns, missing form inputs), the agent runs a read-only DOM probe via browser_evaluate. The probe is a canonical JS snippet embedded in the prompt — agents copy it verbatim so results stay comparable across runs.

What the probe returns

{
  "url": "https://example.com/page",
  "framework": "angular-material" | "radix" | "headlessui" | "react" | "unknown",
  "stableHookInventory": {
    "data-testid": 0, "aria-label": 38, "id": 80, "name": 5, "role": 73, ...
  },
  "popoverItems": [
    // Visible children of floating/portal containers, auto-detected
    { "source": "popover", "tag": "div", "text": "Option A",
      "attrs": { "role": "option", "aria-label": "..." },
      "cssPath": "[aria-label=\"Option A\"]" }
  ],
  "actionableItems": [
    // Cursor-pointer / onclick / interactive-role elements body-wide
    { "source": "actionable", "tag": "button", "text": "Continue",
      "attrs": { "aria-label": "Continue", "type": "submit" },
      "cssPath": "[aria-label=\"Continue\"]" }
  ],
  "counts": { "popover": 3, "actionable": 42 }
}

Why each field matters

• stableHookInventory tells the agent which hook strategy applies site-wide: "38 aria-labels, 0 testids → use aria-label + role+name across this workflow". Record this once per site in learnings.
• framework drives known-bad-id filters (Radix → skip _rN_, Angular Material → skip mat-mdc-*).
• popoverItems captures floating/portal content (React portals, Radix Popover, Headless UI menus) that the a11y tree often misses.
• actionableItems catches custom <div> buttons and form inputs the a11y tree skips because the tag itself implies interactive.
• cssPath is pre-filtered against auto-generated id patterns. If non-null, use it directly in main.py. If null, fall back to role+name from the a11y snapshot.

When to run the probe

• After first navigation to a new site (for the stable-hook inventory).
• When a specific element isn't in browser_snapshot output but should be clickable.
• When debugging a selector failure — the probe's current DOM view is ground truth.

When NOT to run the probe

• As a substitute for browser_snapshot. The snapshot is faster, lighter, and carries widget state (disabled/expanded). Probe the DOM only when the snapshot is insufficient.
• For actions. The probe is strictly read-only. Writes go through browser_click, browser_type, browser_select, browser_navigate.

Multi-backend invocation

The probe JS body is identical across backends. Only the wrapper differs:

Shared CDP tab contract

When a workflow uses agent_browser in CDP mode, it is controlling the user's real Chrome through a shared CDP port. Chrome tabs are global to that browser, so the workflow must not rely on agent-browser's "latest active tab" behavior. The project wrapper enforces an explicit tab on every page action.

CDP is not a background-isolated mode. Because it drives a visible, user-profile Chrome window, tab selection, navigation, clicks, fills, uploads, and sometimes snapshots may bring Chrome to the foreground and interrupt typing in Codex or another app. Use CDP only when the workflow needs the user's real logged-in browser or a site blocks headless automation. For schedules and unattended work, prefer headless mode or a dedicated automation Chrome profile/port that the user does not use interactively.

Required flow

1. Try the real tab list first. The wrapper gives it 15 seconds, then falls back to remembered selected-tab state if Chrome/CDP is stuck:

   browser("tab", [])
   

2. Reuse a listed or selected tab if it matches the workflow.
3. Create one stable labeled tab when no tab is selected or the selected tab is unrelated:

   browser("tab", ["new", "--label", "upwork-profile", "https://www.upwork.com/"])
   

4. Navigate with URL-only open, then include the tab inline on later page actions:

   browser("open", ["https://www.upwork.com/freelancers/example"])
   browser("snapshot", ["tab", "upwork-profile", "-i"])
   browser("click", ["tab", "upwork-profile", "@e1"])
   browser("fill", ["tab", "upwork-profile", "@e2", "search text"])
   browser("eval", ["tab", "upwork-profile", "document.title"])
   

["--tab", "<tab-id-or-label>", ...] is also accepted.

Enforcement

The MCP wrapper rejects CDP page actions that omit a tab. The error includes the selected-tab hint and tells the agent to retry with either ["tab", "<tab-id-or-label>", ...] or ["--tab", "<tab-id-or-label>", ...]. tab and reset commands are exempt because they are used to inspect or repair tab state.

The wrapper translates the inline tab into native agent-browser behavior by selecting the tab immediately before the command, then running the command under a per-CDP-port mutex. This keeps select tab -> action atomic enough to avoid interleaving with another workflow command on the same CDP port. Commands against different CDP ports do not share that lock.

Important distinction

Native agent-browser uses a session-level active tab (agent-browser --session ... tab t2, then agent-browser --session ... snapshot). The inline tab form is this project's MCP wrapper API; it is intentionally stricter so the LLM chooses the tab on every browser action instead of inheriting whichever tab happened to be active.

Direct CDP scripts

Raw CDP scripts can bypass the wrapper, so they must follow the same tab discipline manually:

• Read existing targets from /json/list.
• Reuse a matching target by URL/title when possible.
• Prefer agent_browser eval with an inline tab for complex JavaScript because it uses the shared CDP command lock.
• Treat direct CDP as read-only diagnostics by default. It bypasses the wrapper lock, so do not use it for navigation, clicking, filling, scrolling, uploads, or multi-page loops.
• Do not call window.location, element.click(), Target.createTarget, /json/new, Target.closeTarget, or /json/close unless the task explicitly requires disposable raw-CDP control and the user accepts that it bypasses shared-browser locking.
• If direct CDP is used for diagnostics, connect to the chosen target's WebSocket and close only the WebSocket client, not the Chrome tab.

Site-access resilience

Production sites often block Playwright-launched browsers. When browser_navigate returns "Permission Denied" / a blank page / a native alert() freeze, the agent should:

1. Stop launching a fresh browser — the Playwright fingerprint is the problem.
2. CDP-attach to an already-running Chrome with a real user profile:
   • agent-browser --cdp <port> / --auto-connect
   • Playwright's connect_over_cdp("http://localhost:9222") in main.py.
3. Register a dialog handler before interacting if the page shows native alerts.
4. Document the access preamble in learnings so future steps detect-and-switch automatically.

Cloudflare-style interstitials may also be bypassable by using an alternate subdomain (e.g. prod.gcp.example.com vs prod.example.com). Record these in the site-profile learning.

Ephemeral refs — explicit ban

Refs like @e1, e68, "ref": "abc123" appear in snapshot output and are usable only for the immediate next tool call in the same session. They are:

• Not stable across snapshots — a fresh snapshot reassigns them.
• Not stable across sessions — every new browser spawn rebuilds the ref map.
• Never safe to save to main.py, learnings, or any other artifact.

The validator reviewMainPyScript at controller_learn_code.go runs a regex check (Check 10) that rejects any saved main.py containing ['"]@e\d+['"] or {"ref": "abc..."} when the script calls browser tools. This catches the failure mode at save time before it costs you a broken run.

What learnings must capture for browser steps

When learnings_access="read-write" and the step used a browser tool, the post-step learning agent MUST produce the following content in learnings/_global/SKILL.md (typically split across references/site-profile.md and references/selectors.md):

1. Site access preconditions — CDP required, Cloudflare interstitial on apex, dialog handler needed, failure signatures so future steps can detect-and-switch.
2. Stable-hook inventory — once per site: counts of testids / aria-labels / ids, framework, recommended locator strategy.
3. Per-action intents, not raw selectors — record semantic identity plus 1–2 alternates:

   Step [login.fill_user_id]
     intent: {by: "id", value: "panAdhaarUserId"}
     alt:    {by: "placeholder", value: "PAN/ Aadhaar/ Other User ID"}
     alt:    {by: "role+name_contains", role: "textbox", name: "User ID"}
     notes:  Continue button stays disabled until input has a value.
   

4. Behavioral quirks — multi-step flows (User ID → Continue → Password), cross-domain redirects (e-Filing → TRACES), disabled-until-valid gates, OTP/captcha branches, phantom controls (e.g. a #btn that looks like Proceed but does nothing — the real action is a link below).
5. Known-bad selector patterns — explicit per-site "do NOT use" list (Radix auto-ids, mat-mdc-*, whatever bit this workflow).

Intents (not raw selectors) are the core contract: future fix loops and harden runs re-derive the selector from the intent against the current snapshot/probe output. This is why intents are 1–2 alternates per action rather than a single locator.

Which agents see these rules

The browser-authoring block (BuildBrowserAuthoringRules) is injected into every prompt that authors, patches, or reviews main.py — provided the workflow has a browser MCP (HasBrowserCapability() returns true):

Non-browser workflows (workflow config has no browser server/skill/CDP) skip the entire block — saves ~60 lines of prompt tokens per step.

Code locations

• prompt_sections.go:BuildBrowserAuthoringRules() — the full browser-rules block including the canonical DOM probe JS. Single source of truth.
• prompt_sections.go:browserAuthoringRulesIfBrowserEnabled() — helper for call sites that have direct access to the orchestrator (workshop manager).
• prompt_sections.go:BrowserAuthoringRulesFromTemplateVars() — helper for call sites that only have templateVars (harden, review, execution-only).
• controller.go:HasBrowserCapability() — the gate. Checks explicit non-none browserMode, CDP port, registered servers (playwright), agent-browser skill. Never use GetBrowserMode() != "" as a proxy — empty means auto-detect, not disabled.
• execution_only_agent.go — template has {{.BrowserAuthoringRules}} slot; populated by BrowserAuthoringRulesFromTemplateVars(templateVars).
• interactive_workshop_manager.go — workshop/harden/review call sites set HasBrowserAccess from iwm.controller.HasBrowserCapability().
• learning_agent.go — {{if .HasBrowserAccess}}...{{end}} block describes the required learnings shape for browser steps.
• controller_learn_code.go:reviewMainPyScript Check 10 — regex-rejects ephemeral refs in saved main.py when the script imports browser tool calls.

What's intentionally NOT done

• No new browser_inspect MCP tool. The probe is JS the agent passes to existing browser_evaluate. Rationale: keeps tool surface lean; probe logic stays in one prompt file. Promote to a real tool only if prompt-embedded proves unreliable in practice.
• No self-healing runtime. When a saved main.py selector rots, the current flow is: run fails → execution agent sees error → patches main.py. A future "self-healing" pass could read the intent from learnings, re-derive the selector from the current snapshot, and patch automatically without invoking the execution agent. Designed but deferred until we have real failure-pattern data.
• No iframe probe traversal. The probe runs in the top document only. Sites with iframes (TRACES embedded forms, Stripe checkout) need per-frame probe invocation. Deferred.

Related docs

• learning_architecture.md — how learnings get written and manually locked.
• step_config_format_specification.md — learnings_access field governing read/write of global SKILL.md.
• learn_code_flow.md — the full learn-code lifecycle (fast path, fix loop, save-back).