Your AI agent is either improving or decaying. And the dashboard you have does not show the difference. Observability shows what the agent did. Evals score it. Neither catches decay. The self-improvement layer catches it. It sits between your agent and production, turning observation into shipped fixes. Three things define it: • Production traces cluster into behavior patterns. • Evals get minted from real failures. • A person approves what ships. Each cycle makes the next cycle sharper. Find the full breakdown below.

A good agent works at launch. But a great agent improves after launch. That requires a complete self-improvement loop: • Capture traces: Record decisions, tool calls, outputs, and failures—not just success rates. • Map real behavior: Cluster traces by what the agent is actually doing across users, tasks, and edge cases. • Find recurring failures: Surface patterns such as wrong tool calls, hallucinated capabilities, tone misses, and incomplete execution. • Turn failures into evals: Create scoring functions, test candidate fixes, and measure which changes improve performance. • Keep humans in control: The system proposes improvements. A person decides what reaches production. Each step depends on the one before it. No traces means no patterns. No patterns means no reliable issues. No evals means no measurable fixes. Miss one step, and the team is back to debugging production failures and running manual evals at 2 a.m.

Read the full breakdown here go.adaline.ai/0kJYsbv

Using the same model family to generate and judge your outputs isn’t evaluation. It’s self-grading. Three biases that don’t show up in aggregate agreement scores but consistently show up in practice: 1. Position bias: Give the model two responses, and it favors whichever appears first, regardless of quality. 2. Verbosity bias: Longer outputs score higher, not more accurate ones. 3. Self-enhancement bias: When a model judges its own outputs against a competitor, it rates itself higher even when the outputs are identical. The research case for LLM-as-a-judge is solid. The case for running it without calibration is not. Which of these has burned your eval pipeline?

Monitoring tells you that an agent failed, but observability tells you which step in the sequence caused it. For a single LLM call, the distinction barely matters. For a multi-step agent with tool calls, branching logic, and intermediate states, the distinction is the difference between a two-hour fix and a two-week investigation. The four things production agent observability actually requires: • Input traces per step tell you what the agent received at each stage, not just the final prompt. Without them, you can only see the end state. • Tool call logs capture which tool was called, with what parameters, and what it returned. This is the layer where silent failures hide. • Intermediate decision points show where the agent chose one path over another and on what signal. • Eval attachment links evaluations to specific execution traces so you can see what the eval found on the exact run that failed. Build this before the first failure you cannot reproduce. By then, the trace is gone.

The model stopped being the variable. Frontier models commoditized between mid-2024 and late 2025. The difference between top-tier closed models on production tasks is compressed into the noise floor. Same model, same task, dramatically different outcomes depending on its surroundings. John Yang and colleagues at Princeton showed this directly with the SWE-agent in 2024. The interface a coding agent used to read files, run tests, and edit code sharply affected performance, even with the model held constant. That flips where the compounding lives. It sits in the harness around the model, not inside the weights. The self-improving agent is not a smarter model. It is an agent embedded in a harness that runs a closed loop on its own behavior and learns from production traffic without retraining the model underneath. The pattern has a shape now. The shape has started to repeat.

Routing is not a dropdown menu. Routing is a policy system. It sits at the center of the agentic stack and asks the same set of questions on every request. Isaac Ong and coauthors at Berkeley showed that learned routers trained on preference data can cut cost by more than half while preserving quality. Sierra ships a multi-model router that keeps a task-specific ordered list and swaps when provider quality regresses. So a well-designed router asks: • Sensitivity: How regulated is the data on this request? • Complexity: How much reasoning depth is required? • Tools: Are tool calls or long context needed? • Budget: What is the cost and latency envelope? • Consequence: What breaks if the answer is wrong? • Verification: Does the output need an independent reviewer? • Escalation: Which model tier catches the request if the first one fails? Route down by default. Escalate on evidence.

The maturity model for loop engineering has five levels. Each level names a real capability, not a slogan. Teams that skip a level tend to ship pilots that never make it past the demo call. - Level 1 Scripted: The agent runs one prompt against one model, no branching, no retries. - Level 2 Structured: Tools, retries, and simple routing exist, but the loop is manually operated. - Level 3 Instrumented: Traces, evaluators, and a small labeled set close the review loop. - Level 4 Governed: Halt conditions, permissions, and approvals are policy, not custom. - Level 5 Compounding: Production traces feed a scheduled harness update on a real cadence. Two to three is the hardest transition. It is where the loop stops being a script and starts being an artifact with an owner. Most teams sit there for a quarter before they clear it.

A replayable trace is not a chat log. It is an essential record of how the agent is performing. Here are the eight fields that are non-negotiable. Miss one, and the run cannot be reconstructed the day someone asks. • Inputs: The exact user request, plus every retrieved artifact fed to the model. • Model Config: Weights, version, temperature, tool list, and system prompt used for the call. • Tool Calls: Every tool invocation with arguments, response, and latency. • State: Scratchpad, memory reads and writes, and checkpointed run state. • Decisions: The routing choice, the escalation trigger, the halt reason. • Verifier Outputs: Judge scores, schema results, policy check verdicts. • Approvals: Who approved what, at what time, with what evidence. • Result: Final action taken, or the honest failure message. Any product without those fields is operating in open loop.

Return on Tokens needs an operating model, not a slogan. Six phases move an agent run from token-max reflex to measured spend. The workflow: • Explore: Widen the search space before committing to a plan. • Prove: Verify the plan on a small slice with real tolerances. • Measure: Track completion, tokens, and cost per attempt. • Route: Send each subtask to the smallest capable model. • Budget: Cap tokens per task and enforce the cap in code. • Compress: Prune history, cache tools, and summarize context between steps. Each phase answers one question: is this token worth spending? The order matters because measurement precedes routing, and routing precedes budgets. Without that sequence, budget cuts land on the wrong steps. Run ROT on the whole loop, not on a single call.

The capture spec for agent replay is not “log everything.” Terabytes of noise nobody can navigate. Eight fields, per step, per run: • Prompt sent to the model: In full, system prompt, tool definitions, and message history. • Model output: In full, reasoning tokens where the model exposes them. • Tool call arguments: Exactly as passed to each tool. • Tool responses: Exactly as returned, including errors and timeouts. • Intermediate state: Whatever the loop carries between iterations. • Decision points: The paths the agent considered and did not take. • Time and cost: Per step, so the replay is also a budget replay. • Version pins: Which model, which prompt, which tool build. Design these in from day one. Retrofitting a trace layer means walking back through every wrapper and touching every tool interface.