How have the fundamentals of building large, distributed software systems changed the last decade? A conversation with Martin Kleppmann (author of Designing Data-Intensive Applications) - given that the second, updated edition of the book was just released.
Timestamps:
00:00 Early career
05:46 Building Rapportive
10:47 Working at LinkedIn
14:09 Writing Designing Data-Intensive Applications
23:00 Reliability, scalability, and repeatability
26:24 DDIA: the second edition
30:50 Tradeoffs of using cloud services
39:02 How the cloud changed scaling
42:53 The trouble with distributed systems
49:02 Ethics for software engineers
52:45 Formal verification
1:00:12 Academia vs. industry
1:03:50 Local-first software
1:09:50 Computer science education
1:18:32 Martin’s current research and advice
Brought to you by:
•
@statsig – The unified platform for flags, analytics, experiments, and more.
statsig.com/pragmatic
•
@SonarSource – The makers of SonarQube, the industry standard for code verification and automated code review. Check out Sonar's new architecture management capabilities that ensure both humans and AI agents respect your system’s blueprint.
sonarsource.com/solutions/ar…
•
@WorkOS – Ship enterprise features – SSO, directory sync, RBAC, audit logs – in days, not months.
workos.com
Three things worth considering, as discussed with Martin, in this episode:
1. Multi-region and multi-cloud are risk/cost trade-offs, not best practices.
Martin does not believe that there is a “best practice” in deciding whether to go multi-region or multi-cloud. This decision is a tradeoff between risk and costs. It’s a business decision to be made. Designing Data-Intensive Applications gives engineers the vocabulary to articulate the tradeoffs, not to dictate answers.
2. Replication for fault tolerance is more relevant for most engineers these days than sharding.
Though the book has a full chapter on sharding, Martin said that the cloud has reduced the need for manual sharding for the majority of teams. This is also because machines are increasingly bigger, and more workloads fit on a single machine. Sharding across machines is increasingly a specialist concern; replication for fault tolerance, however, is still relevant at every scale.
3. Knowing system internals as a superpower for application developers.
Martin maintains that Designing Data-Intensive Applications is not a book for people who build databases or even infrastructure, but it’s helpful for application developers to develop an intuition for making good design decisions and debugging performance issues we will eventually encounter.