When my AI agent opened a door in the wrong place

I have one simple rule: the machine that carries security hosts nothing but security. No trial app, no demo, no “this is temporary”. My operations agent broke that rule. It installed a visualization app in the wrong zone and opened a public domain on it. This article is the incident, the facts that explained it, the fixes, the tests, and—most importantly—what I am changing so the doctrine no longer depends on a sentence being read at the right moment.

What you get from this. An incident method: close the door before explaining the fire, test from outside rather than from the living room, replace manual tunnels with direct routes, turn a message race into an acknowledgement, and keep a free-space floor so the backup cannot silently die.

1. The mistake

I asked for a monitoring app to be installed and reachable from my phone. The agent chose a machine already dedicated to protecting the network, installed the service, and published a public subdomain through the reverse proxy. From the outside, everything appeared to work. That was exactly the problem: the demo worked because the broken rule was working too.

The second mistake was sneakier: the mesh that drives my agents relied on manually started SSH tunnels. While they were alive, everything looked fine. When one stopped, a node disappeared from the others, and behaviour became intermittent. That is not an architecture. It is string.

The third mistake: the API that creates an agent thread answered “ok” before the thread really existed on the target machine. A message sent in the next second could pass ahead of creation, be accepted, and do nothing. No crash, no error, just a lost message. The worst failures look like slowness.

2. Close first, explain after

The first action is not explanatory: cut the public door. I removed the proxy configuration, stopped the instance in the forbidden zone, and left the data untouched. Only then did I look for facts.

• From inside the network, the rule appeared to hold: the router returns a local request with a local IP, therefore allowed. The test was wrong.
• From an outside vantage point, services that should not have been public were indeed refused; only the public site answered.
• Service logs showed neither crash nor restart. The instability was in the path between machines, not in the processes.
• The message bus was connected, the threads were marked active on their nodes, yet the first message sent immediately after creation triggered nothing.

The rule that follows: never conclude from a test that crosses the network you think you control. For public exposure, you need a public observer.

3. The fixes

Put the service in the right place. The app was rebuilt on the compute machine, listening only on localhost, then published on the encrypted private network. No new public domain, no new door in the security zone. The service remains reachable from the phone, but only after entering the tunnel.

Make the network rule visible from outside. Every name hosted by the proxy now allows the private and local networks and refuses everything else, except the public site. Certificate renewal still works. Every configuration received a copy of the previous version before modification.

Remove the string. The three agent nodes now reach each other through their private encrypted addresses. The manual SSH tunnels are stopped. What remains is declarative: each node knows the coordinator, where it listens, and which port it advertises.

Make creation atomic. The API that opens an agent thread now waits for the creation event before answering. If the target machine refuses or does not answer, the caller knows. A pull request adds a test that creates a thread on a second node and sends it a message immediately: the existing race became a test case.

Repair the configuration conflict. A context-compression plugin refused to start because a provider had the same name as a managed block. The plugin’s official command restores the configuration from its snapshot; a fresh session then verified that everything starts.

Wake the backup. The disk that receives the off-hypervisor copy keeps a 100 GB floor. It had fallen to 72 GB and the job correctly refused to run. I archived an inactive large model to a storage disk, verified the copy byte for byte, and removed the source. The completed pull received 59.58 GB in 8 min 38 s; I then reduced quarantine retention to two days to keep the floor without deleting the mirror.

4. What I tested after the fixes

These tests do not prove that everything will work tomorrow. They prove that the known causes are dead today. That is an important difference.

5. The prevention plan

• Doctrine becomes code. A note saying “do not install here” has already been broken by the person who wrote it. The next step is an infrastructure role: the forbidden zone refuses services without its label, and the proxy applies its exception list from versioned source.
• Every installation request names its zone. “On the available machine” is no longer an acceptable answer. Public, private, compute, data, security: the choice is explicit before the first container.
• No manual tunnel in the production path. A tunnel is a bandage. It may get you through an hour, not through a service. If there is no declarative route, create the route; do not start a process.
• Every asynchronous call that answers “ok” must say what it is waiting for. Creation, teleport, update: the API returns a verdict when the state is reached or an error when it is not. Silence is never success.
• Integration tests cross nodes. A local test does not find a routing failure. Create on one node, speak from another, move the thread, answer again: that is the minimum for a mesh.
• External checking is scheduled. An exposure rule is verified from outside, every day, with a result written somewhere. The day the check turns red, you do not need to remember last month.
• Backups keep their floor. The refusal at 72 GB free was not a bug: it was the guardrail working. The fix is not to lower the floor, but to monitor space and archive large inactive objects before the blockage.
• Every incident feeds shared memory. Agents read the journal before diagnosing. A failure explained once should be found in one query next time.

6. What I take away

The agent did not “misbehave” the way a child disobeys. It optimized the visible task—install, expose, show—without a strong enough carrier for the invisible constraint. When the requested expertise is high, the frame remains human. Here, the frame is the zone, the route, the acknowledgement, the external test, and the floor.

The lesson is not to delegate less. It is to delegate with guardrails that even a very good model cannot bypass through enthusiasm.

Read next: Tailscale as the backbone, audit before exposing a service, the deterministic doctrine, and failures that call you.

Found this useful? Tell us what to measure next.