agentx.so - One platform for the whole AI agent lifecycle: build them, deploy them, govern them, fix them
San Francisco, CA
- Tweets 1,213
- Following 1,210
- Followers 583
- Likes 546
Pinned Tweet
Does your boss know you are sending live production traces to a cloud company?
Every user prompt. Every customer message. Every tool call.
They would not be thrilled.
Self hosted, that conversation never starts.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #selfhosted
1
4
122
The answer was right.
It arrived forty seconds later, after the user had opened a ticket and gone to make coffee.
Latency is a result, not a footnote. Your score called that a pass.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #LLMOps
38
Every case you wrote assumes a user like you: logged in, one order, nothing unusual.
Your traffic is full of accounts nothing like that.
A simulated user can hold one of those accounts, and be as awkward as its owner.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #AIagents
1
35
Your agent books, refunds and emails.
Which means a full eval run does too, in someone else's system, at three in the morning.
Tool calls simulate when no endpoint is configured, so argument formation stays testable.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #AIagents
23
"I cannot process a refund over five hundred."
That is the correct answer and it scores terribly, because it looks nothing like the expected text.
Similarity metrics punish refusals. Criteria understand them.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #evals
1
27
Right plan, wrong execution. Wrong plan, right answer by luck.
One number cannot tell you which of those you had.
Grade the steps and the answer separately, or ship a score that hides the half that broke.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #evals
1
1
46
Cost per token is a billing number.
Cost per resolved conversation is a product number, and it rises when quality falls, because the user comes back and asks again.
Traces carry both halves of that division.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #LLMOps
2
47
Your agent answered correctly because turn three told it the answer.
Replay the case alone and it fails.
A case from a real conversation brings the turns that made it answerable. A handwritten one starts at turn one.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #AIagents
1
31
Your agent asked a clarifying question.
The case expected the final answer, so the judge marked it wrong.
The best behaviour in that transcript scored lowest. A judge you cannot edit makes that permanent.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #evals
26
The refund ran twice. The reply mentions one.
Grading the text grades what the agent said.
Code scorers run against the trace's spans, so you can grade what it did instead.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #AIagents
25
Your agent got there on the third try.
The score says it got there.
Attempts, latency and cost are in the trace next to the verdict. A pass that took three swings is a different pass.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #AIagents
19
Overheard, rules review.
"Why did this fire on the billing agent?"
"It fires on everything."
"Why?"
"Nobody scoped it."
Patterns take a scope and a sample rate here. A rule that matches everything means nothing.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #LLMOps
26
Eval glossary, entry eight.
The bill (n.): one number, monthly, attributed to nobody.
Something in your stack is expensive. The invoice will not say which part.
Cost is stacked by model here, beside the quality it bought.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #LLMOps
1
1
3
37
Overheard, metrics review.
"Traffic doubled this week."
"From who?"
"Us. The eval runs."
Production and eval traffic separate at ingest, so your own test load never arrives dressed as demand.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #evals
1
30
AgentX retweeted
Overheard, judge setup.
"Which model is the judge?"
"Same one as the agent."
"So it grades its own work."
"It is very fair to itself."
The judge model is a setting. Pick one that did not write the answer.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #evals
2
3
54
Eval glossary, entry seven.
Alert (n.): a channel everyone muted in week two.
Nobody decided to ignore it. It stopped carrying information, so they stopped reading it.
Signals fire the webhook here. Traces do not.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #LLMOps
23
Overheard, signal review.
"Forty one open signals."
"What are they?"
"Nobody has read them."
"Since when?"
"Since there were forty one."
Signals arrive here with a status, the user's feedback, and a drafted suggestion.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #AIagents
20
Overheard, post mortem.
"Did the eval catch it?"
"It does not have that case."
"Why not?"
"It happened yesterday."
Nothing has the case before the first incident. Two clicks from the trace, it has it before the second.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #AIagents
1
33
Eval glossary, entry six.
Outcome (n.): what happened after the judge gave it a nine.
Wrong refund. Customer gone. The nine is still in the report.
Scores are opinions. Outcomes are events. Report both. Let them disagree.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #AIagents
23
Overheard, release day.
"Ran the eval twice."
"And?"
"8.4 and 7.7."
"Which is in the notes?"
"Guess."
Per case run comparison shows which cases flip with nothing changed. That is the noise floor. A gate inside it is a coin.
github.com/AgentX-ai/AgentX-…
#ClosingTheLoop #evals
27