Last year at @tryramp I laid out three predictions for how language models would evolve. I was trying to clarify which bets might actually be durable over time. A lot of it is now starting to take shape. Here’s an update. Thread 👇

😄

Was great to hangout with @GregorVand on @software_daily

the saddest part of AI writing my code is the depths to which I understand rust lifetimes to the point where I can construct lifetime cases that boggle the mind is now completely useless. but alas, this is good, it should be useless.

I see it as a point on the LLM pareto optimal curve in a regime that had a large revealed latent demand (no thinking, single token, low latency acceptable intelligence) that was under-invested into because of a race to higher intelligence.

company values should be falsifiable. There should be companies who would abhor what you value.

important trend

About to cure some diseases

a wise man once told me "sufficiently advanced latency is indistinguishable from downtime"

They’ll expect one of us in the training data

people underestimate the degree to which large companies open sourcing software is about labor fungibility

so utterly tired of consciousness in AI conversations! especially when it comes to multi agent systems. First, it’s not a binary! Go read @drmichaellevin Second, It’s a terrible engineering tool. Better to ask what cognitive capabilities we are witnessing, . plz stop pontificating and go be an engineer.

at-scale language model training is akin to semiconductor process development 1. small experiments lie, failures only appear at scale 2. you are fighting a yield problem 3. need world class observability into dynamics. how to do so is non obvious the developed recipe compounds

maybe eval awareness is not to be squashed but harnessed

- But zoom back even further, and a disagg WAL is simply an instance of Lamport's State Machine Replication, which inherently uses a shared, distributed log. (Our WAL-on-S3 design at Confluent heavily relied on this theory; as usual, it turns out Lamport got there first!)

morsel driven parallelism ftw

We invented a new concurrency control protocol, named Fission, for agents swarms to update shared data for Foundation. trychroma.com/engineering/tr… and includes a section on "why not Git"

Allow me to say that retrieval is the only ML problem in which we still cling to the assumption that an inert scoring function (a meager dot product!), which couples your “search-time compute” with the dimensionality of the representation, can be sensible. Who would have guessed this forces you to have exponentially larger embeddings just to do basic things. Instead, if this makes it sound more modern, you need “inference scaling for retrieval”, also known as late interaction from 2019/2020:

"are you memory bandwidth or compute bound?!?" "memory bandwidth bound" "so you *do* know the difference"

morsel driven parallelism is the Harvard bar scene of good will hunting for database engineers