Optimization story time! So in DBOS, every workflow is represented by a row in a Postgres workflow status table, which stores the workflow’s status. I was trying to figure out where to store the workflow’s inputs and outputs. The natural thing to do, and the thing we did originally, was to make them columns of the workflow status table. That way, they’re updated with the rest of the status. The issue is that workflow inputs and outputs can be large. This is tricky because Postgres does multi-version concurrency control (MVCC), meaning that every time you update any value in a row, Postgres rewrites the entire row. So storing inputs and outputs directly in the workflow status table is dangerous, because they could get rewritten many times by unrelated status changes. But this isn’t the end of the story, because Postgres has another feature specifically to manage large columns: aptly named “the oversized-attribute storage technique (TOAST)”. What this does is store large column values in their own separate “TOAST tables” both to give them more space and to make sure they aren’t rewritten every time the main row is updated. So storing inputs and outputs in the workflow status table is safe, because Postgres TOASTs them and they aren’t rewritten. But this also isn’t the end of the story, because we need to be able to efficiently delete workflows after they’re done processing. And if workflow inputs and outputs are TOASTed, they’ll be stored in random locations on disk far from their parent workflow status row. So that means when we do a batch delete of workflows, we have to seek all their TOASTed inputs and outputs, which causes cache thrashing and terrible performance. So storing inputs and outputs directly in the workflow status table is dangerous, because it makes deleting workflows too expensive. So what we finally ended up doing is normalizing the main workflow status table and pulling workflow inputs and outputs into their own tables. When deleting workflows, we delete from each table separately, so all data is deleted in insertion order, giving us good cache locality and performance. What’s the takeaway from all this? At enough scale, all abstractions are leaky! None of these details (MVCC, TOAST, cache locality) are in the SQL spec, but you need to understand them to make a Postgres-backed system fast. Ultimately, making any system work well requires deeply understanding how it interacts with everything below it.

I used to think durable execution required a server and workers. Now I think it's just a library and an S3 bucket. Change my mind.

DBOS now integrates with the Vercel AI SDK! This integration makes your AI SDK agents durable, backed by your Postgres database. It checkpoints every action (model and tool call) your agents take in Postgres, so if your agent is interrupted, it automatically uses those checkpoints to recover from where it left off. This lets you build long-running agents with a human in the loop, or agents performing sensitive or critical tasks, without worrying about what happens when models or infrastructure fail. Check it out! 👇

Reminder that your benchmark latencies don't mean anything if I don't know precisely where the client and server are located.

New blog post on another Postgres scaling war story 🐘 When trying to scale 20K+ deletes/sec, ran into two tricky issues: 1. Deletes cascading through foreign keys caused cache thrashing that cut throughput by 10x. 2. After a large batch of deletes finished, index lookups could take 20+ minutes navigating a wall of dead tuples. We talk about how deletes actually work in Postgres, what went wrong, and how we fixed it. Check it out! 👇

When you do boring shit, you tend to over engineer it When you do ambitious shit, you’re forced to keep it simple Great lessons in there

We’ve just released new major versions of DBOS Python (3.0) and TS (5.0)! The headline features are schema changes (namely, splitting workflow inputs and outputs into their own tables) that should substantially improve performance at scale, especially when workflow data volumes are large. We’re also releasing large optimizations to workflow retention policies. More on that later in future posts… As major versions, these remove some deprecated/legacy interfaces. We’ve published upgrade guides to help you migrate away from these if you’re still using them. Check out the release notes 👇