[new
@arxiv paper] There’s a huge gap between playing with AI skills in a demo and ⚙️ engineering them for production.
In a simple setup, semantic routing feels like enough: the user asks something, the LLM matches the text to a skill description, and executes it. But in real-world production at scale (like
@Wix Customer Care), semantic matching alone isn't enough. 🚩
Why? Because semantic matching only looks at what the user says, completely ignoring what the user is actually allowed or able to do.
A user asking to "connect a domain" is a 100% semantic match for the Domain Skill. But if they don't have an active premium plan, executing that skill is a dead end.
In a new arXiv paper by
@AshkenaziOrtal, Vitalii Kloz, and Mykhailo Ulianchenko, we present Helpmate’s (Wix’s Customer Care AI agent) architecture: introducing a deterministic executability gate that evaluates account state before candidate skills reach the LLM.
Tested across 756K+ user messages in production:
🎯 59.4% of semantically matched candidates were filtered out because the account state made them non-executable
📉 90.5% reduction in skill context size 228.8M tokens saved
🛡️ In a targeted replay of 1,000 risky conversations, removing the gate caused the model to select a non-executable skill in 7.8% of cases.
Read the full paper:
arxiv.org/html/2608.01050v1