Giving code work to a local agent without it breaking everything

Level AdvancedEstimated time 15 minRead first Homelab 8: your own code forge, where nothing reaches main without a green checkA word you don't know? Glossary

A local coding model writes well-commented, plausible code. Sometimes it does not run. These rules come from our own forge, where local agents propose changes every day; each one was paid for with a broken run or a defective merge.

The agent never touches main. It gets a short, closed request with the real data, works on its own branch in a sandbox, must make a test that runs the code pass, and opens a pull request that a human or a stronger model reads before the merge.

1. Small requests

The failure is stamina, not difficulty. Our short deliverables came out clean (13 and 31 lines, flawless); the long ones broke (83 and 342 lines, defective). Requests with one or two deliverables finished in 6 to 11 minutes, four in 7 minutes; eight ran 28 minutes and more, then expired with nothing. So: four deliverables at most, and split the rest. Our queue refuses wider requests by itself.

2. Give the data, not its description

One run burned 57 minutes searching a repository for JSON fields that another repository produced. Paste a real excerpt into the request and say explicitly not to look for it. Give the commit subject too, and a closed list of files it may touch.

3. A test that executes the code

Always ask for a test that runs what was written, however small, and never drop it when you split a task. The only two pull requests of ours that could not run were the only two without one. What slipped through without such a test: an authorization header built and never sent, a host name confused with a full URL, a path check comparing text instead of folders, a function called without its underscore. A test that simply ran the code would have caught each of them.

For anything with a strict format, give the expected output as a golden file and compare byte for byte. Then break the code on purpose and check that a test turns red: a test that never fails proves nothing.

4. Read every pull request

Two of five merged pull requests in one day carried a defect that would not run. The agent works on its own branch and never touches main; a human, or a stronger model, reads the diff before the merge. And check that a pull request is still open before pushing a fix to its branch: auto-merge may have closed it already.

The trap we found today

Qwen-family models use special tokens (<|im_start|>, <|im_end|>) to mark their own conversation turns. Asked to repeat a line containing them, our model stopped dead at the first one: it cannot write them as text. Three runs failed on a module that needed them. The first invented other tokens and bent the tests to accept them, the second looped for 314 turns, the third silently dropped them. The fix: build those tokens by concatenation in the code ("<|" + "im_start|>"), use plain-text markers in data files, and never type them in the request itself. If you fine-tune or prompt such models through raw templates, expect this.

Found this useful? Tell us what to measure next.