8 ms·
This is hilarious. It appears that Twitter is DDOSing itself. The Twitter home feed's been down for most of this morning. Even though nothing loads, the Twitte
by user_named 3y ago
This is hilarious. It appears that Twitter is DDOSing itself.
The Twitter home feed's been down for most of this morning. Even though nothing loads, the Twitter website never stops trying and trying.
In the first video, notice the error message that I'm being rate limited. Then notice the jiggling scrollbar on the right.
The second video shows why it's jiggling. Twitter is firing off about 10 requests a second to itself to try and fetch content that never arrives because Elon's latest genius innovation is to block people from being able to read Twitter without logging in.
This likely created some hellish conditions that the engineers never envisioned and so we get this comedy of errors resulting in the most epic of self-owns, the self-DDOS.
Unbelievable. It's amateur hour.
#TwitterDown #MastodonMigration #DDOS #TwitterFail #SelfDDOS
A mostly still movie of a Twitter feed showing a rate limited error message and a jiggling scrollbar indicating repeated attempts to load a resource.
Firefox network console showing 10 requests to twitter.com zooming by per second. https://sfba.social/@sysop408/110639435788921057 https://sfba.social/@sysop408/110639435788921057
- deleted 3y ago[deleted]
- moose_man 3y agoYou mean to tell me that you can't actually run a tech company with three engineers and a Mac Mini?
- deleted 3y ago[deleted]
- WinLychee 3y agoSounds like a really interesting challenge, would be fun to see how far you can get.
- WJW 3y agoWithout specifying scale, this seems pretty easy. Pieter Levels has been able to manage a tech company with just a single engineer and with some optimization he could probably fit it all onto a mac mini without too much trouble.
- deleted 3y ago[deleted]
- cogman10 3y agoI mean, hat tip to the pie-acquisition engineers, Twitter has done better than I thought it would without maintenance.
- malfist 3y agoI'm not terribly surprised. General rule of thumb is you devote 20% of your resources to KTLO. If you fire 50% of your staff, you can still keep the lights on, just the portion of new development fell from 80% of your capacity to 60%. Now obviously firing entire teams changes that metric some. Probably why there's been some high profile outages like MFA. But most companies can sacrifice large portions of their workforce if they only want to maintain the status quo.
- cubefox 3y agoHave you checked how many engineers Mastodon has? https://joinmastodon.org/de/about https://joinmastodon.org/de/about Four.
- wilg 3y agoJust for my own interest, based on the latest data I could find and making ChatGPT's code interpreter doing the remedial math (https://chat.openai.com/share/3745dbcd-5be5-4af9-ae34-36845b2f3bc1 https://chat.openai.com/share/3745dbcd-5be5-4af9-ae34-36845b...) The Monthly Active Users (MAUs) per engineer for Twitter is approximately 600,000, while for Mastodon it's around 280,000.
- lost_tourist 3y agoBut if you count salaries in the amount of efficiency and uptime per employee per user per dollar spent is astronomically higher in favor of Mastodon
- albertopv 3y agoManaging servers/infrastructure also? AFAIK Twitter has a couple of data centers.
- rafark 3y agoDoesn’t take too many people to build a website like twitter. It’s not really hard and there are tons of mvc frameworks in pretty much any major language. Only one good programmer could probably create a clone. Of course the scale matters, but it’s not like you’d be writing an OS.
- ipaddr 3y agoThey are not involved in running your instance
- throwaway14356 3y agoone would be ideal... until they leave...
- GuB-42 3y agoI think Twitter was definitely overstaffed. You don't need that much manpower to run a tech company, at least on the technical side. The problem is that Twitter organized itself as an overstaffed company, with complex structures, complex code, lots of R&D, etc... Than Elon Musk came in and did a Thanos snap and immediately fired half of the workforce, and I guess quite a few others left, especially among the best who would have an easy way finding a good job elsewhere. But while the workforce was drastically reduced, the complexity of the system wasn't reduced and probably never will. So, you get what happens now. It is possible to run a Twitter-like website with much less resources than Twitter had before Elon Musk bought it, but you can't run Twitter as it is now.
- berkes 3y agoOnly Twitter was an advertising company. And for that to run smooth, you apparently need a lot of staff. From people that ensure your Kelloggs ads don't appear below literal neonazi movies, to people that close deals on large advertising campaigns, to people that ensure the place where those ads appear (Twitter, the product) actually keeps running.
- jklinger410 3y ago> Twitter is firing off about 10 requests a second to itself to try and fetch content that never arrives because Elon's latest genius innovation is to block people from being able to read Twitter without logging in. Hey, just as an FYI, this comment doesn't make any sense. Why would blocking people from viewing the TL on the home page cause Twitter more stress?
- lolinder 3y agoI don't know what all is involved on the backend, but when I go to twitter.com it runs 123 requests and transfers 11MB of data in order to serve me a login page.
- Maxion 3y ago> Why would blocking people from viewing the TL on the home page cause Twitter more stress? If you block stuff somewhere in the microservie hell in the backend, but forget to update the frontend code properly, and everything was built with the assumption that authentication and authorization is not a thing, and suddenly it is, there's bound to be craploads of stuff that can start spamming network requests.
- dan-robertson 3y agoI don’t think ‘built with the assumption that authentication and authorisation is not a thing’ is the best description. Requests from logged in users would already need to be authenticated to see private accounts they follow. It’s more just the client having aggressive retry logic in the case of whatever rate limit responses they’re getting.
- minimaxir 3y agoBroken retry logic due to an unexpected situation.
- adamwk 3y agoIt could be retry logic that gets stuck in certain conditions that have been met by how TL blocking is handled.
- paulmd 3y ago[flagged]
- tiedieconderoga 3y agoYou'd think they would at least throw in an exponential backoff. Here, I'll help. to = 2; while (!request(timeout=to)) { to *= 2; } I'm available for a 6-week contract. $1M plus a ride in a Dragon capsule, cash up front.
- ProjectArcturis 3y agoWe'll give you $2M and a free OceanGate tour of the Titanic; $2M paid after the tour.
- willis936 3y agoThat's better than my current life insurance policy.
- ProjectArcturis 3y agoAs far as you'll ever know, it is.
- raverbashing 3y agoMy guess is that the frontend person responsible got fired
- weird-eye-issue 3y agoIf the request immediately returns an error then this would just call it in a loop with no delay