11 ms·
Bluesky April 2026 Outage Post-Mortem
- threecheese 6mo ago> What I had missed is that we deployed a new internal service last week that sent less than three GetPostRecord requests per second, but it did sometimes send batches of 15-20 thousand URIs at a time. Typically, we'd probably be doing between 1-50 post lookups per request. That’ll do it.
- bombcar 6mo agoZero, one, many, many thousands.
- 98codes 6mo agoAhh, the three relevant numbers in development: 0, 1, and infinity.
- htx80nerd 6mo agoless than ideal if I had to be frank.
- jandrese 6mo agoThe incredible part about this is because their backend is all TCP/IP they were literally exhausting the ports by leaving all 65k of them in TIME_WAIT, and the workaround was to start randomizing the localhost address to give them another trillion ports or so.
- Night_Thastus 6mo agoI mean, it's one GetPostRecord, Michael. What could it cost? 1 trillion ports?
- kyledrake 6mo agoThis is a pretty interesting solution. I could see how this could useful for certain kinds of problems (as part of a ddos attack mitigation for example).
- verdverm 6mo agoAn oldie and a goodie https://news.ycombinator.com/item?id=21865715 https://news.ycombinator.com/item?id=21865715
- LoganDark 6mo agoAnd then they fix the issue by using multiple localhost IPs rather than, perhaps, not sending 15-20 thousand URIs at a time
- odo1242 6mo agoThey mentioned it was a temporary fix that they removed after finding and fixing the true root cause, though.
- templar_snow 6mo ago[flagged]
- lavela 6mo agoWhy?
- goekjclo 6mo ago> The timing of these log spikes lined up with drops in user-facing traffic, which makes sense. Our data plane heavily uses memcached to keep load off our main Scylla database, and if we're exhausting ports, that's a huge problem. I expect this is common.
- jmclnx 6mo agoLite Blue on a dark Blue background. That is a new one, I have seen grey text on lite grey, but blue on blue ? The article does work in lynx, at least I can read it.
- rvz 6mo agoThank you for the post mortem on this outage.
- jonstaab 6mo agonostr never goes down
- pfraze 6mo agoAll support to other decentralizers but nothing never goes down.
- jonstaab 6mo ago1000x redundancy makes it vanishingly unlikely. Although I know we're due for a pole shift so all bets are off I suppose.
- numpad0 6mo agoWasn't aware there are ~2k relays now. Have inter-relay sharing situation improved? When I tried it long time ago, the idea was just a transposed Mastodon model that the client would just multi-post to dozen different servers(relays) automatically to be hopeful that the post would be available in at least one shared relays between the user and their followers. That didn't seem to scale well.
- jonstaab 6mo agoGetting clients to do the right thing is like herding cats, but there has been some progress. Early 2023 Mike Dilger came up with the "gossip model" (renamed "outbox model" for obvious reasons). Here's my write-up: https://habla.news/hodlbod/8YjqXm4SKY-TauwjOfLXS https://habla.news/hodlbod/8YjqXm4SKY-TauwjOfLXS The basic idea is that for microblogging use cases users advertise which relays their content is stored on, which clients follow (this implies that there are less-decentralized indexes that hold these pointers, but it does help distribute content to aligned relays instead of blast content everywhere). Also, relays aside, one key difference vs ActivityPub is that no third party owns your identity, which means you can move from one relay to another freely, which is not true on Mastodon.
- 6mo ago
- electrondood 6mo agoGreat write up... curious about the RCA. Thanks!
- tapoxi 6mo agoI don't really understand this architecture, but I thought Bluesky was distributed like Mastodon? How can it have an outage?
- Retr0id 6mo agoMastodon infra can have outages, too.
- tapoxi 6mo agoIt's just confined to one instance if it goes down, not all of Mastodon.
- pfraze 6mo agoThis writeup is useful for backend engineers: https://atproto.com/articles/atproto-for-distsys-engineers https://atproto.com/articles/atproto-for-distsys-engineers The simple answer is that atproto works like the web & search engines, where the apps aggregate from the distributed accounts. So the proper analogy here would be like yahoo going down in 1999.
- isodev 6mo agoGoogle and MSN Search were already available at this time. Also websites used to publish webrings and there was IRC and forums to ask people about things.
- tapoxi 6mo agoThis is a fantastic write-up, thanks for sharing!
- fiatjaf 6mo agoSorry, but this analogy is very misleading, no one browses websites through Google's servers. For example, right now in my URL bar I read "news.ycombinator.com", not "google.com/profile/news.ycombinator.com". If Google goes down now I can keep browsing this website and all the other websites I have in all my other tabs as if nothing had happened.
- gsibble 6mo agoDid all 3 users notice?
- ffsm8 6mo agoNaw, only one did. Turns out the other two were his socket accounts he used to upvote and comment on his own content. Okay, nuff trolling for today
- dogemaster2027 6mo ago[dead]
- mwkaufma 6mo agoTell us more about this buggy "new internal service" that's scraping batch data :P
- dogemaster2027 6mo ago[dead]
- drewg123 6mo agoGolang's use of a potentially unbounded number of threads is just insane. I used to be fairly bullish on golang, but this, combined with the fact that its garbage collected, makes me feel its just unsuitable for production use.
- deleted 6mo ago[deleted]
- tombert 6mo agoWhy does garbage collection make it unsuitable for production use? A lot of production software is written in garbage collected languages like Java. Pretty much the entire backend for iTunes/Apple Music is written in Java, and it's not doing any kind of fancy bump allocator tricks to avoid garbage. In my mind, kind of hard to argue that Apple Music is not "production use". There are certainly plenty of projects where garbage collection is too slow, but I don't know that they're the majority, and more people would likely prefer memory safety by default.
- slopinthebag 6mo agoEverything is understood by comparison. Unsuitable for production use, compared to what is the more apt question.
- madeofpalk 6mo agoBased on my experience of Apple Music being pretty bad at streaming music, i would say that it's not ready for 'production use'.
- tombert 6mo agoOk, judging by this job posting [1] it looks like Spotify uses Java as well. [1] https://www.lifeatspotify.com/jobs/senior-backend-engineer-audiobooks https://www.lifeatspotify.com/jobs/senior-backend-engineer-a...
- floating-io 6mo ago
- pembrook 6mo agoDistributed social media goes down? hrmmm. Email and the internet don't have "downtime." Certain key infra providers do of course. ISPs can go down. DNS providers can go down. But the internet and email itself can't go down absent a global electricity outage. You haven't built a decentralized network until you reach that standard imo. Otherwise its just "distributed protocol" cosplay. Nice costume. Kind of like how everybody has been amnesia'd into thinking Obsidian is open source when it really isn't.
- iAMkenough 6mo agoBluesky is a provider. Blacksky didn’t go down.
- pembrook 6mo agoIs there anything running on Blacksky other than Bluesky with more than say, 100 active users? AOL never even got to that level of dominance in the internet 1.0 era. The point is it's not a distributed network if one node is 99.9% of all traffic.
- opem 6mo agoAt least they aren't hiding and transparent about it unlike the big tech corps with so called SLAs
- tmpz22 6mo agoThere are no outages in Azure sing se.
- _heimdall 6mo agoGitHub's Ops team would approve this message, I assume.
- thedrexster 6mo agoi see you, brother! <3
- streetfighter64 6mo ago> They represent real user-facing downtime Off-topic, but "real" feels like the new "delve". Is there such a thing as "fake" or "virtual" downtime, or why do people feel the need to specify that all manner of things are "real" nowadays?
- mwagstaff 6mo agoWith my SRE hat on, dare I ask... could/should this have been picked up in testing? And then normally there's a nice discussion about how production is very different to the test environment.
- heliumtera 6mo agoGood to know the discussion about decentralization and federation had finally ended