How Google managed to crash many iOS apps around the world
On September 28 we noticed in our React Native mobile app that crashes are occurring from the Firebase Analytics. We couldn't do anything about it.
When I took a look at the issue opened in the Firebase project, I saw that many others have the same problem.
Now, Google provided a postmortem. What they did was that they removed an old feature flag from the backend system for Firebase Analytics. The existing tests passed, but the SDK inside the apps still read that flag. When they rolled out that change globally, each device that fetched it crashed at launch time.
The bad config was served for ๐ฎ ๐ต๐ผ๐๐ฟ๐ ๐ฎ๐ป๐ฑ ๐ญ๐ญ ๐บ๐ถ๐ป๐๐๐ฒ๐. A developer opened a GitHub issue 18 minutes after the rollout began, and by 18:05 PDT, it already pointed at the config response and asked for a rollback. Google traced the cause at 19:16.
Facebook had a similar issue in May 2020, when a server-side config change made its iOS SDK crash Spotify, TikTok, Pinterest and Venmo. The lesson has been public for more than six years.
Google listed four fixes, and each one you can check in your systems:
๐น ๐ฉ๐ฎ๐น๐ถ๐ฑ๐ฎ๐๐ฒ ๐ผ๐ป ๐๐ต๐ฒ ๐ฐ๐น๐ถ๐ฒ๐ป๐. The SDK never checked that a flag had a name. Bad server data should not crash the app, so skip the broken flag and keep the cached default.
๐น ๐ง๐ฒ๐๐ ๐๐ต๐ฎ๐ ๐ฎ ๐ฑ๐ฒ๐น๐ฒ๐๐ฒ ๐น๐ฒ๐ฎ๐๐ฒ๐ ๐ฏ๐ฒ๐ต๐ถ๐ป๐ฑ. The existing tests passed on this change. Block the release when code still reads a flag you removed.
๐น ๐ฅ๐ผ๐น๐น ๐ผ๐๐ ๐ฐ๐ผ๐ป๐ณ๐ถ๐ด ๐ถ๐ป ๐๐๐ฎ๐ด๐ฒ๐. These flags also act as kill switches that turn off a buggy feature remotely. That mechanism is released globally, so the bad config went out to apps everywhere. Treat a config change like a deploy and start with a small group of users (phased rollout).
๐น ๐ฃ๐๐ ๐ฐ๐น๐ถ๐ฒ๐ป๐ ๐ต๐ฒ๐ฎ๐น๐๐ต ๐ผ๐ป ๐๐ต๐ฒ ๐๐๐ฎ๐๐๐ ๐ฝ๐ฎ๐ด๐ฒ. Google's status pages track mostly server health, and it stayed green while apps crashed.
What does it take to get to an agent that catches its own mistake and writes the eval and the fix?
Catch @hwchase17's Interrupt London session on building agents that improve over time: youtu.be/xy_7SzwsIjM