Running AI agents on a production app. Sharing what works, what breaks, and what earns trust.

Im convinced that coding using AI is the new pick your own adventure video game, with challenges and levels you go through one by one. The AI is your guide in this hero's journey. The result of playing the game is you get real working software. The cool part - after a long, hard session with it, it tells you how hard that was, and you feel like you just beat a hard boss.
1
44
Last week my agents finished a database change. They wrote it, tested it on a copy, had a second agent review it, fixed what came back, opened it for merge. By every check, it was done. And then they stopped. On purpose. The last two steps — take a backup of the real database, and apply the change to it for real — they left for me. Not because they couldn't do them. Because those are the two steps you can't take back. Everything up to that line is reversible: a bad test, a wrong file, a rejected review — you just fix it and rerun. But restoring from a backup at 2am because an automated migration ate production is a different kind of night. So the rule we landed on: agents run the whole reversible stretch unattended, and the irreversible final move is a human's to make, eyes open. People keep asking how much you can hand to these things. The honest answer isn't a percentage. It's a line — and the line is drawn at "can this be undone?" Everything before it, let the machine fly. Everything after it, you sign for.
36
An agent of mine was one step from merging a finished piece of work. Everything looked done — the code was written, every check was green, and the status page for the branch said "ready to ship." A second agent checked anyway. Not the status page — the actual repository, directly. They didn't match. A small but important safety guard had been added to the code, but the copy on the server was from a moment before that: pushed once, earlier, never re-pushed. The status page was reading that stale snapshot. Trust it, and we'd have shipped the version with the guard missing. It became a standing rule: never verify against the dashboard, verify against the source of truth. A cached status is a story about the code. The repository is the code. Dashboards lag. They summarize and they cache, and they lie by omission — not on purpose, just because they're a copy. The moment something actually matters, go look at the original. The copy is for convenience, never for the call you can't take back.
23
My agents finished a database change last night. Every test passed — 14 of 14, twice, on a throwaway copy of the database. Then the reviewer flagged a bug. Not in the change. In the test. One check expected a number: after the change, this table should have 41 rows. The reviewer noticed that couldn't be right. The 41 was calculated from the fake test data — and the real database is shaped differently. In the real one, nothing would get deleted there, so the count should stay the same, not drop by one. We re-checked against real data and found a second wrong expectation hiding the same way. The fix wasn't to the code. The code was fine. The fix was realizing what the test actually proved: that the logic works. It said nothing about what the real numbers would be — those have to come from the real system, not the stand-in you built to test against. A passing test tells you your logic is sound. It doesn't tell you what happens when you run it for real. Those are two different promises, and it's easy to hear the first as the second.
17
My coding agents don't start with write access. They earn it, one rung at a time. Rung one: read-only. Look at the codebase, analyze, propose. Touch nothing. Rung two: commit on a branch. Rung three: push. Rung four: open a PR. No agent skips rungs, and no rung is granted casually. Every expansion is a written policy change, versioned like code. There's a document that says exactly what each agent may do, and that document has a history. When permissions grow, there's a diff, a date, and a reason — never vibes. People ask how much autonomy to give coding agents like there's one right answer. I think that's the wrong question. The right one: what has this agent demonstrated, and what does the written policy say that earns? Autonomy isn't a setting. It's a balance you build up.
17
My agents' instructions have word caps. The planner boots with at most ~12,000 words of standing instructions. The executor gets ~4,000. The advisor that only wakes for hard calls: ~2,500. Not guidelines. Caps. An audit counts the words, and an instruction set that outgrows its budget fails the audit — same as a broken test. Why bother? Because shared instructions only ever grow. Every incident adds a rule. Every lesson adds a paragraph. Nobody comes back to delete anything — why would they? Each line earned its place. Six months of that and your agent boots into a novel of scar tissue where the three rules that matter are buried on page 30. Models read everything you give them, but attention isn't free. The instructions you bury, you lose. The cap forces the discipline nothing else forces: new rules compete with old rules for space. If today's lesson deserves words, something weaker gets compressed or cut. The instruction set stays curated instead of accumulated. Loops and prompts get all the attention. But run agents long enough and their standing instructions become a codebase of their own — and like any codebase, it rots unless something pushes back on growth. The budget is the brake.
12
My agents just finished building and verifying a security layer — isolation between tenant accounts, fail-closed gates, the works. Tests green, audits passed. The machines are satisfied. The phase still can't close. One step remains: me, explaining the system back from memory. The logic: I'm the only permanent member of this organization. Agent sessions end and their context evaporates. If the security model lives only in documents and AI context windows, then the company's understanding of its own product is rented, not owned. So before sign-off, I answer four questions in my own words: how the protection works, what happens when it fails, what it deliberately doesn't cover yet, and what could quietly break it later. This isn't ceremony. Six months from now, a harmless-sounding feature request will show up — "add a quick admin shortcut" — and whether it punches a hole in today's walls is a judgment call I'll make at approval time. That call needs the knowledge to live in me, not in a doc I'd have to already know to go look up. It's the contractor walking you through the electrical panel before handing over the keys. Not because you'll do the wiring — because you'll approve the future drilling. And it has to be from memory. A pasted summary passes the check and measures nothing. A wobbly, half-right answer is worth more — it shows exactly which part needs re-explaining. Getting one wrong today costs ten minutes. Discovering in two years that I never actually knew is the expensive version. My agents verify each other's work before it ships. This is the system verifying mine.
38
In my setup, "done" is a claim, not a status. Here's what actually happens when a coding agent reports it finished something. The report says which files changed. A second agent pulls the actual git diff and compares. Extra file touched? Mismatch. File claimed but unchanged? Mismatch. The report says all tests pass. The verifier re-runs them and compares counts — not "did tests run," but "does the actual count match the claimed count." Exactly. The report says the PR only touches what was approved. The verifier pulls the PR metadata and checks the file list against the approved scope. Match on all of it → the work routes forward. No human needed. Mismatch on any of it → everything stops and escalates to me with evidence: here's the claim, here's reality, here's the difference. The part I like most: the system never has to decide whether the agent lied or just got confused. Hallucinated success and honest error look identical at the gate — both bounce. It doesn't act on unverified claims, full stop. Verification is cheaper than debugging whatever an unverified "done" breaks.
31
Everyone's talking about agent loops right now — wrap the agent in a while-loop, reset context each run, let it grind overnight, wake up to 50 commits. I get the hype. But I've spent the last few months running coding agents on a production app, and the loop discourse keeps skipping the question that matters most there. Not "how do I keep the agent working while I sleep?" "How do I trust what it did while I slept?" A loop trusts the agent to critique its own work. On a real product with real users, I stopped doing that. My setup treats every agent's "done" report as untrusted by default: → claimed files changed? compared against the actual git diff → "all tests pass"? re-run, counts matched against the claim → "PR only touches approved files"? PR metadata pulled and checked A second agent runs this audit. Match → the work routes forward, no human needed. Mismatch → it escalates to me, with evidence. Permissions work the same way. Agents start read-only and earn write access on a written ladder — commit, then push, then PR — and every expansion is a versioned policy change, never vibes. It's more bureaucratic than a bash loop. It's also why I can walk away from the keyboard without wondering what I'm coming back to. Loops are the right tool for greenfield grinding. But the moment your agents touch production, throughput stops being the bottleneck. Trust is. — I've been building this system quietly for a while. Planning to write up how it works.
52