Principal @Atlassian | Helping engineers reach Staff/Principal | 1:1 Mentorship & Mock Interviews | 90+ System design fundamentals - puneetpatwari.in

Hyderabad
A month ago, I told everyone: "I've been a backend Engineer for 12+ years. Today, I'm a Principal Engineer at Atlassian. I've designed systems that handle millions of requests. Sat on both sides of system design interviews. Reviewed more architecture docs than I can count. Starting today, I'm breaking down the fundamentals of scaling for the next 25 days. If you're learning system design bookmark this thread, you're going to get a lot of learning from this." FYI, the series has concluded. Here are all the concepts, please bookmark, share, learn and the most important build from the learnings you get. Also, if you have any ideas on what you'd like to see from me, please let me know, any other series or concepts to be broken down.
35
105
1,356
122,947
The database is overloaded by reads. Candidate says: “Add five read replicas.” Then I reveal that one query scans 40% of the table and runs every second. Five replicas multiply capacity. They do not repair a bad access pattern. What do you fix first?
3
2
24
3,712
At Amazon, a strong L5 with 7 to 8 years of experience can cross ₹1 Cr+ CTC. At L6, compensation can go well beyond ₹1.5 Cr+ depending on team, location, stock, and level. But the interesting part is not just the money. Look at why recruiters reach out for these roles. It is because their background shows they understand hard systems, and they have enough proof of work that people can see it. Today I am a Principal Engineer, but if I were starting in distributed systems from scratch and wanted to turn that skill into career leverage, this is exactly how I would do it. [1] Build the fundamentals before touching big scale I would first get comfortable with the building blocks: → networking and request lifecycle → databases, indexes, transactions → caching → queues and async processing → replication → sharding → consistency → retries, idempotency, backpressure → observability and failure handling Basically, have the understanding to see what happens when one normal backend service gets slower, busier, or partially unavailable. [2] Build systems where the tradeoffs come into the picture. Do not only watch architecture videos. Build things. Start with: → URL shortener → rate limiter → notification service → job scheduler → file upload system → search service → payment workflow Then deliberately make them harder. – What happens at 10x traffic?” – What if Redis dies?” – What if the same event arrives twice?” – What if one shard becomes hot?” [3] Read engineering blogs and reverse-engineer decisions Uber, Netflix, Stripe, Discord, Cloudflare, LinkedIn, Meta, Amazon. There’s golden material out there. While reading, ask yourself: – Why did they choose this? – What was breaking before? – What tradeoff did they accept? – What new problem did their solution create? That is how engineers actually learn architecture. [4] Put your thinking in public This part is massively underrated. Write about what you learn. Publish: → design breakdowns → architecture diagrams → GitHub projects → incident analyses → open-source contributions → performance experiments Build enough public proof that when someone searches your name, they can tell what kind of engineer you are. That is when learning starts compounding. You study distributed systems to become better at engineering. Then that knowledge helps you design better systems. Those systems give you better stories. Those stories become public proof. And eventually, opportunities start finding you instead of you constantly chasing them. That is the leverage.
getting cold emailed by amazon is crazy
8
65
803
111,304
Uber's database layer serves over 40 million reads a second, and almost none of them hit the database. This is how they got every team off its own homemade cache. Before, any team that needed speed stuck its own Redis in front of Docstore, Uber's MySQL-based database. Every team wrote its own invalidation logic, and every team got it slightly wrong in its own way. So the storage team built caching into the database's own query layer and called it CacheFront. 1. Reads check Redis first. On a miss, the query layer reads MySQL and fills Redis on the way back. 2. Services keep using the same client as before. Caching comes with zero code changes. 3. Invalidation is driven by tailing the database's change stream, so when a row changes, the cache hears about it from the source of truth instead of relying on every app to remember. One use case doing 6 million reads a second would have needed around 60,000 CPU cores on the storage engine. With the cache in front, it ran on about 3,000 Redis cores. Caching became one team's job, done once. Everyone else stopped reinventing invalidation, which is where cache bugs are born.
4
4
48
3,198
Fabrice Bellard is the weirdest software engineer on Earth. > Wrote the video stack that powers YouTube, Netflix and TikTok. > Created the emulator that runs half the cloud. > Made a C compiler that compiles and boots Linux in 15 seconds. > Built a full PC that runs Linux inside your browser. > Wrote a JavaScript engine used in FIFA. > Set a world record calculating 2.7 trillion digits of pi on a cheap desktop. > Turned a VGA card into a TV transmitter. > Makes software 4G/5G base stations. > Invented a better-than-JPEG image format. > Built neural-net and LLM compressors that beat the official benchmarks. It’s like FFmpeg, QEMU, LLVM, V8, a math institute, Qualcomm, and a compression lab… All rolled into one guy who barely gives interviews. @ID_AA_Carmack has said Bellard is probably a better overall programmer than he is. That tracks. Truly a legend 🫡
1
6
129
6,593
If you're learning system design, check out my fundamentals series:
706
Figma's database grew almost 100x after 2020. That year they were running one Postgres instance on the biggest machine AWS sold. By 2024 they'd horizontally sharded it, and the way they de-risked it is worth knowing even if you never shard anything. Sharding is scary because the moment data moves, a routing bug means lost or misplaced writes, and backing out is painful. Figma split it into two steps. 1. Logical sharding first. They created Postgres views, one per shard, each filtering the same table by a hash range of the shard key. The app read and wrote through those views as if the database were already sharded, while every row still sat in one place. 2. Physical sharding later. Only once the app had been running happily against the views did they actually move data onto separate machines. So every query missing a shard key, every routing mistake, surfaced while the data was still in one database. They could roll out by percentage and back out at any point, because nothing had been copied yet. They also grouped tables that share a shard key into "colos," so joins and transactions within one colo still work, and put a Go proxy called DBProxy in front to parse each query and send it to the right shard. Before any big migration, look for a way to fake the end state first. Making the risky step boring is most of the job.
9
6
59
4,967
Python's requests library has no default timeout. If the server on the other end stops responding and you never passed one, your worker waits forever. A handful of those and your entire pool is stuck on sockets that will never answer.
2
3
47
3,482
How to actually get good at backend systems. It's slow. But it works. 1. Pick one thing you already use every day. Postgres, Redis, Kafka, whatever your job runs on. 2. Go one level deeper than you are right now. If you write queries, learn to read query plans. If you can read plans, learn how indexes actually sit on disk. Know that already? Go learn what VACUUM is doing. 3. Break it on your laptop. Fill up the disk. Kill a replica. Leave a transaction open and watch the table grow. Breaking it once beats reading about it ten times. 4. Explain it to one person on your team. If you can't explain it simply, you don't get it yet. Then go one more level down. Background loop you should keep running: whenever something weird happens at work, a slow query, a random spike, a flaky test, chase it till you know the real reason. Most people stop at "restarted it, it's fine now." The ones who keep digging are the ones who get good. Stuff that gets in the way: "I'll do another system design course" → courses teach you the words. They can't give you that moment where your own test fails in a way you didn't expect. That's where you learn. "I've read DDIA, I know distributed systems" → cool, now pick one chapter and make that failure happen on your machine. Then you know it. "We're on managed services, I don't need internals" → managed stuff breaks too, you just get less info when it does. The person who knows what's underneath fixes it while everyone else refreshes the AWS status page. "I'll get to it when work calms down" → it won't calm down. Thirty minutes a day beats the free weekend that never shows up.
4
16
197
10,366
Verifying a webhook signature after parsing the JSON is a classic trap. The signature is computed over the raw request bytes, and parsing then re-serialising the body can change whitespace or key order. Always verify against the raw body first.
1
26
1,759
Discord stores trillions of messages. When they wrote about leaving Cassandra in 2023, everyone talked about the database swap. The part I'd actually steal is a small layer they put in front of the database. Their problem was hot partitions. A big server posts an announcement, thousands of people open that channel at the same moment, and every one of them asks the database for the exact same rows. One node gets hammered, its latency spikes, and the slowness spreads to everything that depends on it. So they built a set of "data services" in Rust, sitting between the API and the database. 1. They hold almost no business logic. Roughly one endpoint per query. 2. They coalesce requests. If 5,000 people ask for the same row at the same instant, the service sends one query and hands the same answer back to all 5,000. 3. Requests for the same channel get routed to the same instance, which is what makes the coalescing actually catch the duplicates. Along with the move to ScyllaDB, the cluster went from 177 nodes to 72, and p99 reads dropped from anywhere between 40 and 125ms to a steady 15ms. When a thousand callers want the same thing at the same moment, the cheapest fix is to ask once. A bigger database won't save you from a thousand identical queries.
5
2
31
2,608
OFFSET pagination is a classic trap on large tables. To serve OFFSET 500000 LIMIT 50, the database still reads and throws away the first 500,000 rows, so every page is slower than the one before. Keyset pagination (WHERE id > last_seen_id) stays fast no matter how deep the user scrolls.
2
1
80
5,990
These genuine reviews make me wonder. Are these models sentient? Different models seem to mimic different behaviours of a living organism.
day 1 observations for grok 4.7 ignore the reports that say “it’s terrible” and the only thing they reference is a public benchmark. the same benchmarks told us opus 5 was better that fable - they are useless also ignore the reports that compare models with 3d games - that’s not real work. it's made for attention on social media i used grok 4.7 for a whole day as my firstmate, and it has been a really solid model with visible improvements over 4.5 (i'm ignoring 4.6 because 4.5 has been working better in my experience) key differences with 4.7 - 1. it follows system prompt very, very closely i noticed firstmate showing many new behaviors that i've never seen before, such as asking me to name specific red CI checks that i'm ok with bypassing, and refuse a simple "yolo" instruction i traced it and it's indeed how i instructed it in firstmate's system prompt, but none of the other models followed it closely enough to make this behavior visible - grok 4.7 is the first to pick that up there were a few other similar examples as well. so to me this is a clear behavioral difference 2. it's very "stable" if you've used astra then you know what a "spiky" model is. it can have some genius moments but you occasionally also wonder "how could it be so dumb and doesn't get me". grok 4.7 is the opposite of that throughout the whole day so far, i'll be honest i haven't get a "wow this is absolutely genius" moment yet, but grok 4.7 has been very steady with no big surprises. its behavior feels predictable, which does help it gain trust from me quickly 3. it's a conservative model it doesn't like to take actions without asking, and would explicitly say so this is a bit of a double edged sword, because it means i sometimes have to state the obvious "yes i do want that", but in hindsight a lot of those cases are indeed a bit ambiguous and i may not have preferred the model to just move forward without my confirmation 4. it's a bit slower and costs more than 4.5, visibly turns are taking a bit longer and my quota is draining at a visibly faster pace. i haven't quantified exactly where this is coming from yet so overall, i think it's showing some clearly different traits, and i mostly like the changes. i'm going to keep it as my primary firstmate and observe more if you've been using it, what qualitative insights have you gathered from real usage so far?
1
1
1,854
Calling a third-party API inside a database transaction is pure madness. The transaction holds its locks for however long the vendor takes to respond, and if the vendor times out after doing the work, so you roll back while they've already processed it.
4
3
77
5,405
The IRS still processes American tax returns on a system called the Individual Master File, first built in the 1960s in assembly language. Every year it handles hundreds of millions of returns. The most important financial system in the US predates the moon landing.
If you ever want to feel humble, remember there is probably a piece of Fortran written in 1974 that is currently processing your bank transactions flawlessly, while my cloud-native Docker container just crashed over a YAML indentation error.
2
1
21
2,254
Health checks that query your database are a classic trap. When the DB slows down for a few seconds, every instance fails its health check at the same time and the load balancer pulls your entire fleet out of rotation. A small DB hiccup becomes a full outage you caused yourself.
3
3
51
3,426
90 LPA–1.4 Cr/year job requirements for an L5 Senior Role or Equivalent at FAANG+ companies: ↣ hardcore DSA ↣ strong system design ↣ full-stack development ↣ AWS or GCP proficiency ↣ experience writing design docs ↣ ability to design scalable services ↣ AI/ML with solid math (Role-based) ↣ comfort with microservices and APIs ↣ deep distributed systems knowledge ↣ strong debugging and production skills ↣ clean coding in Java, Python, Go or C++ ↣ strong grasp of Authz, also if the role requires ↣ familiarity with storage engines SQL, NoSQL ↣ strong grasp of security authn, authz, encryption ↣ ability to reason about latency, throughput & availability ↣ solid understanding of caches, queues, and load balancers ↣ driving through ambiguity and cross-team collaboration ↣ minimum 4-6 years high-impact engineering experience
how the fuck do I make 80L base bhai before I turn 27 I NEED HELP
9
39
493
28,267
Replying to @system_monarch
Btw, if you’re preparing for Senior to Principal-level system design interviews, I’ve put together 90+ fundamentals like this into a guide. You can check it out here: puneetpatwari.in
1,593
Something I wish I knew when I started learning system design is that most systems follow the same exact patterns. The requirements change. The trade-offs change.  But you keep coming back to a familiar set of building blocks. Here are 25 that keep repeating themselves.
3
7
80
7,216
Btw, if you’re preparing for Senior to Principal-level system design interviews, I’ve put together 90+ fundamentals like this into a guide. You can check it out here: puneetpatwari.in
4
2,701
IF YOU NEED TO SCALE READS → Cache-aside: reuse frequently requested data. → Read replicas: spread reads across database copies. → Database indexing: find rows without scanning everything. → Materialized views: precompute expensive query results. IF YOU NEED TO SCALE WRITES → Batching: spread overhead across multiple writes. → Sharding: distribute data across database nodes. → Asynchronous writes: move persistence off the request path. → Backpressure: slow producers when consumers cannot keep up. IF YOU NEED REAL-TIME UPDATES → WebSockets: exchange messages in both directions. → Server-sent events: stream updates from server to client. → Long polling: hold a request until an update arrives or it times out. → Publish/subscribe: distribute events to interested subscribers. IF YOU HAVE LONG-RUNNING WORK → Message queues: buffer jobs for later processing. → Worker pools: process jobs with controlled concurrency. → Workflow engines: coordinate multi-step work and recovery. → Transactional outbox: commit a data change and its outgoing event together, then publish. IF YOU NEED TO HANDLE FAILURES → Retries with backoff: retry transient failures with delays and jitter. → Idempotency: make repeated operations safe. → Circuit breakers: stop calling a dependency that keeps failing. → Bulkheads: isolate resources so one failure does not exhaust everything. → Self-healing: detect and replace unhealthy instances. IF YOU NEED TO COORDINATE DATA → CQRS: separate read and write models. → Event sourcing: preserve state changes as a sequence of events. → Sagas: coordinate local transactions with compensating actions. → Change data capture: propagate database inserts, updates, and deletes.
2
21
727