With model interactions in natural language full text search must become a first class citizen for every major database workload. OLTP, OLAP, logs, metrics, and traces.
Database industry never stops giving!
Quick database rant :)
There is still a lot of basic wood to chop when it comes to agent observability. It's a totally new database workload. OLAP systems are built for <5kb events with UUIDs and very repetitive dictionary-like strings. And the techniques that work to query those do not work for agent traces.
That's compounded by the crazy cost and scale problems. Agents have PMF now and so it's common for these workloads to generate GB/s per agent of tracing data. It's just not practical operationally to store this on attached disks. You have to use object storage.
This problem of fat rows + object storage throws the existing systems' query schedulers for a total loop. They're used to reading local data in microseconds and for that data to be very dense / high signal. If you're searching for random shit in agent traces, this falls apart completely, and you end up doing a terrible job of scheduling work on your CPUs. Which in turn means you need to create a lot of OS threads to handle the workload and make your kernel very unhappy.
We waddled our way around this by cobbling together Tantivy and our own I/O scheduler for a while, but ultimately realized we had to rebuild the execution engine from scratch. We tried using some off the shelf stuff (Datafusion is by far the most promising system here), but sadly to achieve optimal performance you need control flow while decoding postings from the inverted index so that you can short circuit. Tantivy, etc. do this but the logic is interleaved with the I/O itself, so you end up with terrible cold starts no matter what.
A simple way to think about this is a graph traversal on the inverted index. The columnstore-maxxing approach is to read the whole graph and then crunch on it very fast (and you can schedule that efficiently, of course). The classic inverted index approach is traverse and potentially short circuit as you do so, interleaving those operations.
We solved this by building a new engine called Nitro, which does async I/O scheduling, but with control flow / short circuiting built into the actual I/O layer, so that we can still saturate the network but avoid reading stupid amounts of data before actually executing a search. For example, if you search for something like "aha! you must be right about describe_all_queries() not working for Monday's cohort", you do not want to read all postings for "you", "must", "be", "right", "not", and "for".
I think the benchmarks speak for themselves.. We compared Nitro against two modern columnar systems used in agent observability.
We also still have a lot of work to do here. If this kind of stuff interests you, we are hiring!