San Francisco, CA
- Tweets 11,744
- Following 165
- Followers 47,178
- Likes 35,673
Ball-knowers knew all along that Twitter wasn’t just going to collapse after Elon fired 75% of the staff, because Twitter 1.0 had some of the best infrastructure engineers working on it, and when you get that calibre of people building your foundations, it lasts.
18
7
390
44,769
With agentic coding, even in the best case scenarios (where you read every line of generated code), there’s something we’ve lost, perhaps irrevocably.
That has a real cost, & not to mention, second and third order effects, that “software factories” etc don’t necessarily solve.
24
13
180
7,867
Writing a post about “how to effectively debug code when you no longer fully understand it”.
This is real bug I had to fix in ~9K LOC (100% AI generated).
None of the tactics I started with here helped “root cause” the issue, lol. Had to take a different approach to isolate it.
33
23
1,038
52,195
One thing I don’t see many talk about is how coding agents can often *amplify* weaknesses, both individual and organizational.
I think of this as the “empty calories” problem, which is exacerbated when “hyperproductivity” originates from your team’s “lowest common denominator”.
13
24
231
12,886
The people who blithely claim “you don’t need to understand the code anymore” aren’t telling you what debugging looks like in such cases
Here’s one example where I had to debug a gnarly problem at work with Claude *without* access to the code
Possible? Yes
“Easy”? Nope 🙂↔️
12
3
49
15,369
August farmer’s markets in the Bay Area are worth a visit just for the abundance of fresh, juicy figs, and in-season tomatoes. 🍅
Refrigerated, slightly overripe figs make for the best dessert, IMO.
3
1
27
4,739
adopted ASD-STE100 in my project upon a coworker’s recommendation.
Asking Opus 5 to write comments/commit messages etc in plain English has more benefits than just helping with readability.
It discovers bugs. Critical bugs.😱
I was absolutely gobsmacked after seeing this. Wtf!
18
14
474
48,847
In recent weeks, when it came to tricky bugs (a memory leak, a routing issue, a sharding bug), Claude’s initial analysis was almost always 100% wrong.
I had to give it multiple hypotheses, clues, ideas, suggestions for it to solve the bug.
When asked “so, what did you learn?”
7
8
178
20,176
Went out to touch grass, but alas if you live in or around downtown San Francisco, there’s no escaping any discussion of AI, GPUs, agents etc, in public places. 😒
Maybe will try Marin or Sausalito the next time.
At least the nosh (Gochujang Pimento Cheese Twist) was superb.
6
29
3,444
It’s a “people problem”, the oldest problem in the industry, not just a Google problem.
The cost of generating code going to zero *incentivizes* teams to unblock themselves ASAP, rather than dealing with the friction of coming up with cohesive, holistic, “cross-team” solutions.
10
6
77
6,794
OSDI and SOSP are the two most prestigious systems engineering conferences.
Claude Code categorized a decades worth of papers into four buckets.
Directionally speaking, this looks about right. There’s less research happening in databases and compute than a decade ago.
2
24
4,022
In short, this ain’t happening, when you take the full software lifecycle into account
I mean, is your org as a whole solving 10x more high priority problems?
Are you shipping 10x more features?
Has your reliability gone up by 10x?
Can every engineer debug 10x faster!
4
4
62
4,657
Elon should be the last fucking person talking about deaths caused by inexperienced appointees gives his DOGE cuts to USAID (overseen by 20 y/o kids with zero experience in government or public health) caused about 600K totally preventable deaths of the most vulnerable people.
4
5
50
7,751
The problem is that the economy is already in a recession outside the AI/data center buildout space.
If the economy continues to be this poor, with employers disingenuously attributing layoffs to AI, we’re guaranteed to see an anti-tech/AI backlash soon, and it won’t be pretty.
2
4
46
5,995
This entire article on how AI adoption by employees at Dropbox went from low double digits to 90% seems AI generated.
Counted 15 em-dashes in total. 😅
Truly incisive and thoughtful writing is going to become an endangered art form in this AI era.
7
2
39
7,450
Asked ChatGPT to analyze the very long write up and come up with ASCII diagrams explaining the most recent AWS outage that took out us-east-1 in 4 diagrams …
For those who don’t want to read walls of text, this might help
The detailed AWS incident report is out, and it’s worth a read
- DNS records managed by 2 systems; a race condition led to regional record getting unset
- EC2 lease establishment was borked as it depends on DynamoDB
- fluctuating NLB health checks leading to EC2 DNS entry purges
2
12
105
14,700
The detailed AWS incident report is out, and it’s worth a read
- DNS records managed by 2 systems; a race condition led to regional record getting unset
- EC2 lease establishment was borked as it depends on DynamoDB
- fluctuating NLB health checks leading to EC2 DNS entry purges
54
409
2,450
474,087
This is the definition of politicians having strong opinions on things they don’t understand. 🙄
Breaking up AWS won’t result in more reliable cloud providers, or customers suddenly starting to implement multi-AZ and failovers.
22
18
409
14,457
Reading the AWS status updates, seems like what first diagnosed as a DNS resolution issue was either a red herring or an incomplete diagnosis.
As hours later, a system doing health checks on the network load balancers was deemed the culprit.
Full report will be fun to read.
9
36
400
33,071
This is right, but the primary reason tech discourse has downgraded so badly is because most of the serious accounts left/no longer post (think @mipsytipsy, @Lethain etc)
The ones who post are mostly farming for engagement, or try to be edgy or are virtue-signaling to VCs.
1
24
3,464
Remember when I said anon tpot isn’t technical or even very intellectual?
Case in point: they think one of the most famous and highest ranked engineers at Google who was so successful he retired in his 40s isn’t “grinding hard enough”
Chuds doesn’t even begin to cover it. 🙄
17
11
594
45,043
Twitter tech influencers of the 2015-2021 era were, by and large, deeply technical people who had deep insights. Think Kelsey Hightower.
“tpot” is mostly edgelord chuds engagement farming and being contrarian without any substance or even any pretense of intellectual integrity.
43
44
1,039
283,199
The @nytimes doesn’t report “news” anymore.
Just “juicy collection of narratives” with click-baity headlines, which then unleashes a tsunami of hysteria against all kinds of targets.
Mainstream media, with notable exceptions, is 1-2 years “behind” most trends and happenings.
1
1
31
3,651
This might be one anecdote, but what a soul-crushing environment to work in, with no leadership, no engineering mission, no camaraderie between coworkers, no clear ownership … just isolation and disorientation.
And to think of all the VCs who said go work for Twitter 2.0 …
2
4
135
9,006
Massive, massive, absolutely massive #nokings protest on Market Street in SF happening right now. 🇺🇸❤️
3
22
3,928
Nice set of action items as well
- move toward “fail open” architecture
- not everything needs strong, instantaneous, global consistency.
- don’t run your monitoring on the infra it’s monitoring
Funnily, I blogged about some of these very lessons back in 2019 (2nd image).
7
8
254
14,885
Full report for yesterday’s Google outage
- not feature flagging a new code path
- null pointer crashing a binary when reading empty fields from Spanner
- remediation causing a thundering herd as there was no exponential backoff
- which required manual throttling to recover from
17
146
1,571
199,271
Preliminary GCP outage report is out, and as expected, it’s a bad config change pushed out globally by some automation.
A similar incident from 2020 comes to mind (though I bet there are countless other such incidents due to bad global config changes)
A whole slew of Google services (Gmail, Firebase, GKE, GCE, almost every GCP service) had an outage on Apr 13 due to a wrong global configuration change (ha) and bugs in the canary system.
Global "centralized planet-space" control plane = global outage.
bit.ly/2yOyEb6
6
15
148
21,341
Dinner at the newly opened Jules, and the pizza was definitely worth the trouble it took to get the reservation.
They were awesome with accommodations, and the menu can be made vegetarian friendly.
1
16
3,971
Looks like this tweet got “shadow banned” because I can’t see it on my profile (see screenshot)
I need to go to the “Replies” tab to see it
Anyone encountered anything like this, where a single tweet vanishes from your main timeline on your profile?
X engineer realizing what was obvious to many of us who paid attention *years* ago.
that Twitter 1.0’s infrastructure was so battle-hardened given the caliber of talent that worked on core infrastructure, that it would continue to pay dividends for years to come.
4
1
16
20,449
“The failover and redundancy should have worked, but did not”
Us infra eng greybeards remember that Twitter 1.0 had solved such “reliability” centric problems ~10y ago.
These problems span many teams, from hardware to traffic eng to storage, but if you’re fired 85% of them …
2
8
97
7,729
X engineer realizing what was obvious to many of us who paid attention *years* ago.
that Twitter 1.0’s infrastructure was so battle-hardened given the caliber of talent that worked on core infrastructure, that it would continue to pay dividends for years to come.
14
71
924
90,389
The new right wants a white ethnostate, and see African, (non East) Asian and Muslim presence in Europe/Canada a civilizational threat.
It counts among its ranks many minorities, who also want much less diversity, like this Indian guy who clearly wants to live in 1950s America.
3
1
729
The true endgame of having built a whole brand on being “anti-woke” is that ultimately you end up telling your own people to ignore true racism against them, because speaking up against it becomes “woke”, which of course, we can never have.🙃
2
1
51
4,997
Even the @nytimes is now negatively polarized against Elon’s support for H-1B and is printing misinformation.
The reason many stay on H-1B visas much longer than 6 years is due to per country green card caps!
Companies hire despite layoffs because they hire for different roles!
12
18
125
24,878
JFC so much wrong with this demagoguery.
If you abolish the H-1B program, employers aren’t going to start hiring Americans in droves.
The jobs themselves will shift to Europe/Canada/India or some other place with a good tech ecosystem and plenty of easily hireable talent.
322
41
513
69,525
This is true about tech twitter.
A friend mentioned that he’d been approached for his last 2 jobs via Twitter DMs, but for his current job search is mostly using LinkedIn.
I’ve stopped posting about infrastructure engineering as most people I used to engage with have left.
5
4
55
6,123
A better summary:
“My Indian grandfather who voted for Trump thrice didn’t care at all when the racism was hurled at Mexicans, Muslims, Haitians, Chinese etc; because he looks down upon those people/cultures as well. But draws the line when the racism impacts *his* people”
3
5
64
4,607
Every time I order something at a restaurant ever since that plastics in food report came out, all I wonder is just how much plastic I’m consuming lol.
1
13
2,881
If you’re downwardly mobile or are stuck in cycles of inter-generational poverty, seeing thriving immigrant communities or the rich get vastly richer can be jarring, as none of the purported benefits of immigration or neoliberalism“trickle down” to you and make your life better.
4
2
35
4,230
Ok, Elon going thermonuclear on nativists and racists, defending the H-1B visa to *this*,degree wasn’t on my bingo card.
A very pleasant surprise, but given the amount of vitriol and hatred thats been aimed at immigrant tech workers, this is extremely welcome.
226
135
2,159
116,136
Love to see MAGA Indians being the first ones to “find out” after fucking around with the party of “eating cats and dogs”
You believed the people who dehumanized and smeared another defenseless immigrant community (Haitians) won’t come for you?
That you’re somehow “special”?
14
8
186
12,013
Some are still pulling their punches, some are just capitulating.
Just how much influence Elon will have over the next 4 years remains to be seen.
1
4
229
Many “anti-DEI” legal immigrants who celebrated Trump are now realizing that “America first” is fundamentally incompatible with their dreams of “global meritocracy”.
And that Trump’s core base is comprised of people who find their presence in America to be rather undesirable.
134
591
4,332
269,222
Hahaha man, I don’t think I’ll ever know what it’s like to plan an international trip without *first* getting the visa sorted, often months in advance.
Americans and others with what’s called “passport privilege” won’t ever comprehend what it’s like for the rest of the world.
5
6
105
10,776
That said, this is what someone close to Vance thinks about immigration.
So this idea of Elon as some champion of tech workers is a bit unfounded. Elon might want more people on O visas/EB1 green cards, but otherwise want to see legal immigration restricted.
1
8
1,456
Will be interesting to see a lot of people twist facts and conjure new narratives in the pursuit of respectability over the next 4 years.
or they’ll stay in their narrow lane, rather than upset MAGA over things like immigration, and risk not being able to influence AI/crypto etc
2
5
942
From personal experience, I know how hard and frustrating it can be to be an Indian immigrant stuck in a never ending green card queue, but I doubt acting entitled, or worse, *disparaging* other nationalities as less deserving serves the cause very well.
Let’s not be jerks.
62
87
2,936
413,315
- Boulettes Larder in the Ferry Building
- Home Coffee Roasters (Chinatown, Sunset, Richmond) (fun seasonal flavors and fun art! See all 4 pictures)
2
344
Yep, many times.
I find their “mission” hot chocolate the best, but all others they serve at their Ferry Building store is great. Their chocolate mix is also good!
Also, check out Crow & Moss. Based out of Michigan, but they ship across the US.
crowandmoss.com/collections/…
1
1
295
Been drinking more hot chocolate lately, and this is the best in SF, IMO.
Christopher Elbow in Hayes Valley.
4
42
5,509
That’s true, but it’s also not the entire picture. Miller very much tried curbing programs like H1B, revoked the ability of spouses of H1B visas to work in the U.S. (many of my friends decided not to immigrate to the U.S. due to this reason alone)
forbes.com/sites/stuartander…
1
5
971