Full results. 12 support messages, categories cancel/refund/bug/praise/other.
1 "please cancel my plan" -> cancel | rules cancel | Jev cancel (100%)
2 "i want to stop paying for this" -> cancel | rules other MISS | Jev cancel (100%)
3 "discontinue my membership" -> cancel | rules cancel | Jev cancel (100%)
4 "don't cancel my order, i changed my mind" -> other | rules other | Jev other (45%)
5 "can i get my money back" -> refund | rules refund | Jev refund (100%)
6 "i was charged twice this month" -> refund | rules refund | Jev refund (100%)
7 "this broke again, third time this week" -> bug | rules bug | Jev bug (100%)
8 "it's not working" -> bug | rules bug | Jev bug (100%)
9 "the app crashes when i upload a file" -> bug | rules bug | Jev bug (100%)
10 "you guys are genuinely amazing" -> praise | rules praise | Jev praise (100%)
11 "how do i change my email address" -> other | rules other | Jev other (100%)
12 "cancel" -> cancel | rules cancel | Jev cancel (100%)
Score: rules 11/12, Jev 12/12.
12 Jev calls ran at once: 1.0s wall, ~970ms each, ~$0.0001 total.
Things I'd rather not gloss over:
- #2 is the only miss, and it's the interesting one. There is no cancel/stop/end keyword in that sentence at all. You cannot write rules for phrasings you didn't anticipate.
- I gave the keyword rules a hand-written negation guard, and that's the only reason they got #4 right. Without it, rules fail #4 too and go 10/12.
- #12 is where Jev disappointed me. One bare word, genuinely ambiguous, and it returned 100%. Calibrated is not the same as infallible. Validate it on your data, don't trust the pitch.
- Rules are free, instant and deterministic. Jev is a network call. On a keyword-friendly task plain code is a genuinely strong baseline, which is the part most demos leave out.
- The real edge isn't the score, it's #4: rules said "other" and had no way to express doubt. Jev said "other" at 45%, so your code can decide to escalate instead of guessing.
Method: 12 messages through
@typesafeai jev-latest via the official SDK, keyword baseline written by hand. Code and page are local.