Browser and desktop task automation with CUAWright plus Hermes agent skill

What they did

They set up CUAWright (formerly Webwright) as a terminal-driven harness so a coding model could launch Playwright browser sessions, inspect pages, and write scripts instead of predicting one click at a time. They used `cuawright web` for browser work and `cuawright desktop` for Ubuntu VM desktop tasks, keeping the workspace (code plus logs) as the persistent state rather than the browser session. They installed the shared `skills/cuawright-web/` folder as a plugin/skill so their existing agent (Hermes, and equivalently Claude Code or Codex) could call CUAWright for browser tasks. After a task succeeded, they ran the optional Skill Factory to distill the solve into a parameterized, verified script that reruns standalone without a model. They also experimented with Task2UI mode to render task results into a small HTML app for reuse.

How it went

The workflow produced rerunnable Playwright scripts that execute in roughly 40 seconds with zero tokens, and the project reports reuse lifting held-out WebArena accuracy from 55% to 70%. Browser benchmarks cited are 88.1% on Online-Mind2Web and 77.5% on Odysseys. Polish is good for a research-grade tool: ~1.5k LoC, OpenAI/Anthropic/OpenRouter backends, and backward-compatible `webwright` commands and imports.

Worth knowing

Desktop tasks require an Ubuntu VM and separate setup per docs/desktop.md, so it is not a drop-in on macOS or Windows hosts. Skill Factory is optional and its reuse gains depend on task similarity. Model backends need API keys, and long-horizon reliability still depends on the underlying coding model. Plugin install paths differ per host agent.

Wiring

Install CUAWright (Python >=3.10) and Playwright Chromium. Set a backend key for OpenAI, Anthropic, or OpenRouter. Use `cuawright web` for browser tasks and `cuawright desktop` for Ubuntu VM desktop tasks; legacy `webwright` commands and imports still resolve. Register the shared `skills/cuawright-web/` folder with the host agent (Hermes, Claude Code, or Codex via `/plugin install cuawright@cuawright`). The agent runs terminal commands, spawns Playwright sessions, writes scripts and logs into the workspace, and iterates on failures. Optionally enable Skill Factory to convert a successful solve into a verified parameterized script, and Task2UI to render results as an HTML app.

Try it yourself

Set up CUAWright in this repo as a browser automation harness for my coding agent. Install the package, configure an OpenAI or Anthropic backend key, and add the skills/cuawright-web folder as a skill for my agent. Then write a script under scripts/ that uses cuawright web to log into a demo site, fill a multi-field form with a parameterized date, wait for lazy-loaded content, and save a screenshot plus a JSON result. Run it, fix any selector or timing failures, and finally distill the working script into a reusable parameterized skill with a short README on how to rerun it without a model.