One browser per agent: web automation that does not trip over itself

Level AdvancedEstimated time 1 hA word you don't know? Glossary

Agents that use the web (to check a site, fill a form, read a page) often drive a browser through its remote-debugging port. The tempting setup is one browser for everyone, each agent in its own tab. We ran twenty agents and six scripts that way, and the results lied.

How these figures were obtained. Measured on our own machines (see the protocol), usually a single run, with speeds as reported by the server itself; we do not publish raw logs. Treat them as orders of magnitude on this hardware, not as guarantees on yours. Figures from outside are labelled with their source.

Twenty-six agents in one browser share one process, one memory and one DNS cache: one runaway tab freezes them all, and their bursts get them throttled. One browser each, with its own profile, port and encrypted DNS, removes all three.

What went wrong, measured

One runaway tab froze everyone: a renderer at 10.7 GB and 162 % CPU made every agent's connection hang for 180 seconds. Killed, the next connection took 0.4 s.
Navigations interrupted each other: 11 out of 20 page loads failed with "navigation interrupted", a race between tabs, not a refusal by the site.
Healthy domains declared dead: the shared browser's bursts of DNS lookups were throttled by our own DNS filter, so large sites came back as "does not resolve".

The setup that works

Start one browser per agent, each with its own profile folder, its own debugging port and its own process. Give each one encrypted DNS of its own, so its lookups no longer go through your home resolver:

chromium --headless=new --user-data-dir=/tmp/agent-1 \
  --remote-debugging-port=9301 \
  --dns-over-https-mode=secure

A clean profile also removes a subtle false positive: "I am logged in" can no longer come from a session that another run left open.

Traps

Tabs accumulate run after run. Close them between waves: closing 100 tabs took one machine from 28 GB used to 19.

Know your ceiling. On a 30 GB laptop, 16 browsers and 16 agents filled the memory; at 24 browsers the system started killing new ones, and each kill looked like an instant "site blocked": 47 false failures in 29 seconds. Move the browsers to a bigger machine before scaling further.

Browser profiles in a small memory-backed folder (such as $XDG_RUNTIME_DIR) can fill it, and other programs keep important things there. Put the profiles on disk.

Found this useful? Tell us what to measure next.