45 ms·
Twitter Is DDOSing Itself
- ZacnyLos 3y agoI find it hilarious Mastodon is more stable than one-instanced Twitter run by some madness tech-maniac.
- dredmorbius 3y agoMastodon has had its issues, both globally and for specific instances. Sidekiq falling over is a big one. See: <https://docs.joinmastodon.org/admin/scaling/ https://docs.joinmastodon.org/admin/scaling/> and <https://nora.codes/post/scaling-mastodon-in-the-face-of-an-exodus/ https://nora.codes/post/scaling-mastodon-in-the-face-of-an-e...> (I have to email my own admin every few months to ckeck if things are OK.) And during the October Revolution as hoardes arrived from birdland, things got ssssslllloooowwww globally. Worked out eventually, but it took a bit. Individual instances also tend to run into scaling issues, with Jerry Bell's Infosec.Exchange coming to mind. (Mostly because Jerry's discussed this a bit.) And of course individual instances can be shut down or fail in various ways. I've migrated several times myself. I will say that most of the time things seem fine, and it's exceptionally rare for there to be truly Fediverse-wide issues. (I've been on Mastodon / the Fediverse since 2017, for the most part quite actively.)
- ZacnyLos 3y agoArchived: https://archive.ph/u5kNK https://archive.ph/u5kNK
- kvetching 3y agoCould this be the reason for the rate limiting? Elon thinks the server usage is scrapers when in reality it's just bad code causing a self DDOS?
- deleted 3y ago[deleted]
- bradknowles 3y agoAsk him whether he wrote the code himself that is DDOS-ing Twitter. I wouldn't be surprised if he had that much hubris.
- anigbrowl 3y agoThis could explain why my computer fan goes into overdrive when trying to read Twitter, has been going on for a few months but I don't care enough to figure it what's going on in the bg
- ttctciyf 3y ago> Twitter is firing off about 10 requests a second to itself to try and fetch content that never arrives because Elon's latest genius innovation is to block people from being able to read Twitter without logging in. It seems an outlandish claim, but then again Muskified Twitter has previous form for this kind of thing with that time when they self-derailed by locking themselves out of their own api, right?[1] 1: https://opuszine.us/posts/when-twitter-fails-itself https://opuszine.us/posts/when-twitter-fails-itself
- arter4 3y agoThis is interesting. Judging from the screenshot, a huge amount of GET /TweetDetail is generated which triggers some rate limiting, as shown by the 429. If this is indeed due to the recent decision to enforce authentication for all API calls, it means the curlprit may actually be the API gateway or something similar downstream. Also, this behavior seem to never stop, which isn't what one would expect from an exponential backoff retry. I don't claim to be a better engineer than the folks working at Twitter, but it is interesting to see something like this in the wild, all Musk-related considerations aside.
- deleted 3y ago[deleted]
- bheadmaster 3y ago> If this is indeed due to the recent decision to enforce authentication for all API calls, it means the curlprit may actually be the API gateway or something similar downstream. The way I understand it, DDoS is not caused by enforced authentication - enforced authentication is just a temporary measure against DDoS.
- Quarrelsome 3y agoI would guess the front end was written under the assumption that the back end would still work without auth. Perhaps the backend changes (mandatory auth + rate limiting) were pushed without sufficient testing of the front + back?
- cactusplant7374 3y agoDid Elon pay the AWS bill? That seems like a likely culprit. Twitter instances are being forcibly shutdown.
- amluto 3y agoTwitter operates its own datacenters.
- cactusplant7374 3y ago
- jitl 3y agoBluesky feels like it’s buckling under the refugee crisis load
- Pxtl 3y agoMastodon.social is chugging along just fine
- jtrip 3y agoMeaning.. people prefer bluesky to mastodon?
- danso 3y agoA lot of people who’ve wanted to jump ship to Mastodon have had plenty of opportunity to do so in the past year. Bsky is still invite only but had significantly increased its rate of handing out invite codes in the past 2 weeks. Today might be the day that invitees have decided to finally check things out
- bengale 3y agoIs it still invite only? I don’t think I know anyone that’s got on there yet.
- noarchy 3y agoI've been on their waiting list for what seems like months at this point. Huge missed opportunity for them, being unable to leverage Twitter's failings.
- jitl 3y agoBased on app perf today, they’re probably metering signups for good reasons.
- agnosticmantis 3y agoNow I’m really tempted to create a Twitter account and start reading. Maybe that’s 3D chess by design by the CTO to get people to sign up. /s
- rideontime 3y agoKeep in mind that Twitter engineers are under extreme pressure, knowing that their H1B visas are at stake, to implement Elon’s whims as quickly as possible on systems they’re unfamiliar with. I doubt many of us would perform better in their situation.
- xyzzyz 3y agoDo you know how many current Twitter engineers are on H1B? Is this based on some data, or just speculation?
- morelisp 3y agoAround 300. https://www.vice.com/en/article/n7z5px/twitter-employees-on-visas-cant-just-quit https://www.vice.com/en/article/n7z5px/twitter-employees-on-...
- lolinder 3y agoThat's all employees, not just engineers.
- georgeecollins 3y agoDo you think twitter has a lot of marketers, PR, product managers under H1B? I don't have data, but in the tech companies I have worked at it engineers were the H1B visa holders, almost exclusively.
- alephnerd 3y agoThey absolutely have a number of PMs on work visas. I'm friends with a lot of them.
- kibwen 3y agoH1-B is for "highly skilled" workers, what skilled workers would Twitter have other than developers?
- craig1f 3y ago[flagged]
- 1270018080 3y agoNever attribute to malice that which is adequately explained by stupidity.
- pferde 3y agoOr greed.
- woleium 3y agoHanlon, is that you? Reminds me of the joke: Pavlov was sitting at the bar when the last order bell rang. "Oh shit", he thought, "I forgot to feed the dogs".
- agubelu 3y agoConditioning works both ways!
- smilespray 3y agoMy cat nods in agreement.
- squarefoot 3y ago> Never attribute to malice that which is adequately explained by stupidity. Still so many examples of the opposite.
- superchroma 3y agoI guess you'll just have to talk to your fellow man with the hole in your face instead of your fingers. I wince thinking of it even now.
- brucethemoose2 3y ago> Where do you all get your info lately? How do you stay informed? I really don't know how to get a wide swath of information anymore with Twitter and Reddit gone News sites reddit and twitter source stuff from... Or pick your favorite aggregator. Sometimes Discord/Matrix and blogs are good for really niche topics.
- Mizoguchi 3y agoThis kind of stuff is unavoidable with what's going on at Twitter. Infrastructure changes, platform changes and mass layoffs all at once. I'm actually impressed they haven't experienced more and longer outages.
- darkwraithcov 3y agoIts a testament to how well designed and implemented the code base is, if the wheels havent started falling off yet (shockingly). I know we’re all supposed to shit on Twitter, but they had world class engineers working there before the mass layoffs and brain drain.
- Solvency 3y agoI can easily visualize a wooden wagon rolling along a rocky slow descent forever and all of the reasons its wheels would eventually break loose. In the case of a platform like Twitter, aside from random acts of god and weather, aside from money, what are the sort of stuff that slowly breaks its wheels off over time such that, without staff, it inevitably stops working completely?
- kevinventullo 3y agoThe ostensible cause for the rate limiting was “excessive scraping”. Battling stuff like this or e.g. spam bots is a constant arms race. So, without staff, the bad actors will eventually win.
- Sparyjerry 3y agoTrue, the rate of improvements has been insane. I use it pretty often and haven't even noticed a second of unavailability even today and right now, not to say I use it 24/7 though. It's pretty obvious some people have a political bend to their hatred of Twitter so any second something isn't perfect they jump on it. It's really a sad state of affairs really.
- arghandugh 3y ago
- quote 3y agoOne weekend Russia attacks Russia, next weekend Twitter attacks Twitter.
- throwaway202351 3y agoDon't forget when Reddit attacked Reddit two weeks ago.
- papito 3y agoI don't have a problem with any of the three.
- stubybubs 3y agoThe entire world now living the "hurt itself in its confusion" meme.
- spaceman_2020 3y agoEveryone else is going to do the same. They're only holding out because they still believe the Fed will cut rates and they can borrow some more cheap money to keep the gravytrain going.
- sva_ 3y agoIt is kinda funny if you consider these companies might consider their user data to be useful, especially with recent advances in LLM models. I've been thinking if you just exclude Reddit posts from training youll probably achieve much lower bullshit scores, as that seems to be what most posts on there seem to represent. I think data curation (by sources) could achieve quite a bit.
- bitwize 3y agoIt hurt itself in its confusion!
- brucethemoose2 3y agoMeanwhile, I can see this very Mastadon post with no login just fine. And the interface is not trying to assault me. It loads quick. ...Seems like a better product than Twitter for a public feed.
- bboygravity 3y agoMost of the rest of the world doesn't agree.
- p0pcult 3y ago[dead]
- teawrecks 3y agoAlso most of the rest of the world doesn't use Twitter (I'm seeing about 240M total users?). The ones that do use Twitter do so because of a significant amount of money going into marketing Twitter's platform. Conversely, relatively nothing goes into pushing people to use mastodon. It can only take off if it really does prove, not just useful, but more useful than a centralized version that's got money behind it.
- Brybry 3y agoWhat microblogging service does most of the rest of the world use? Weibo? Telegram channels?
- antiframe 3y agoI imagine 'none'. With Twitter, the most recognized micro logging site in the world only has 240M of the worlds population, I conclude micro logging isn't popular worldwide.
- Shared404 3y ago> What microblogging service does most of the rest of the world use? Most of the world doesn't.
- iguana_lawyer 3y agoEarlier I noticed the infinite reloading happening on the logout page as well
- brucethemoose2 3y agoAlso, taking Elon's word at face value for a second... is Twitter really worth scraping for AI training or whatever? Its a hive of misinformation, disinformation and toxicity. Its succinct I guess, but nothing is eloquent or descriptive because of the character limit. And its full of repetitive "filler" information. Who wants that in a foundational LLM dataset? Maybe its OK for finding labeled images... But that still seems kidna iffy.
- kitsunesoba 3y agoThe effectiveness of this sort of lockdown is questionable anyway, because the cat's already out of the bag and there's no getting it back in. Same for Reddit. The bulk of the data's already out there and nothing these companies can do will change that.
- afterburner 3y agoMaybe someone is trying to make a disinformation bot. (half-serious) I mean as far as uses for LLMs go that seems to me a pretty realistic one. Mass quick propaganda with little effort. Go for immediate impact, doesn't matter if people look deeper, you're just looking to get a swell of emotional reactions.
- brucethemoose2 3y agoYeah, I guess its a way to make an "engagement optimization" bot using follows/likes from posts as criteria. ... That is horrifying.
- Hoasi 3y ago> Also, taking Elon's word at face value for a second... is Twitter really worth scraping for AI training or whatever? Maybe... if you build a LLM scrapping for the lulz?
- TillE 3y agoIt's useful if you want your LLM to be able to generate tweet-like microblogging text. That does have some value. Or maybe you want to get an aggregate idea of what people are currently talking about in the world, stuff that doesn't rise to the level of capital-n News. There aren't a lot of alternatives for that.
- avereveard 3y agoI've a scalability problem I'll add a dedicated microservice I've now two scalability problems
- WhereIsTheTruth 3y agoThat's not a scalability problem, that's a poor engineering problem, let's not start to find excuses Talents left twitter already, I wouldn't be surprised if the ones that took over are the ones who come from the intelligence industry, as opposed to the tech industry
- turtleyacht 3y agoAs a haiku: I can't scale this more I'll add a microservice I can't scale these more
- oh_sigh 3y agoCertainly not good, but not all GET requests are equal. If these are responded to cheaply, at the point of connection termination, then it might be the case that no one has bothered to clean it up yet.
- Matthias247 3y agoNot exactly sure if that's what you recommend. But connection termination is not necessarily a good thing for DDOS mitigation. The reason is that the client might just retry immediatly - and it will do that using a new TLS connection. And the handshake for that connection has a huge cost. If you plan on disconnecting clients *after* a TLS connection had been established, you will also need to implement TLS handshake rate and connection limiting. That's possible, but I've only seen a tiny amount of services every implementing it.
- throwawat7832 3y ago[flagged]
- ninth_ant 3y agoHe’s the richest man in the world, and a bully narcissist who punches down as a hobby. He’s not the unpopular ugly kid, he’s the one who attacks those kinds of kids. And he certainly doesn’t need you to defend him.
- throwawat7832 3y ago[flagged]
- 3rdrodeo 3y ago[flagged]
- MarcoZavala 3y ago[dead]
- Shekelphile 3y agoHe spent $44 billion to build a platform to attack his own daughter. Beyond just being a bully at that point, IMO.
- mbrz 3y agolike this? https://imgur.com/Ru6ZlCK https://imgur.com/Ru6ZlCK
- throwawat7832 3y agoover what period? I left my dev tools open since my original post (14 minutes) and and it looks like this, so 197 requests / 14 minutes = 14 requests per minutes, which is not a DDOS: https://imgur.com/8sxBAPt https://imgur.com/8sxBAPt
- comboy 3y agoAnybody knows if these requests were happening before the login only change? Because it would be hilarious if huge scrapping operation was a bug in their javascript.
- globalise83 3y agoI can say for sure that a certain flow ("back" from Profile view or similar) would trigger an infinite redirecting loop on Firefox on my Android device, with probably dozens of requests over a couple of seconds until rate limiting kicked in. Maybe there were many of these little bugs which together looked like some kind of DDOS or scraping.
- andrelaszlo 3y agoPerhaps the engineers that understood how to analyze the logs were laid off?
- samb1729 3y agoI've noticed the frontend hammering the backend quite often in the past few weeks. It would not surprise me at all to learn that the "influx of scraping" was mostly Twitter's fault.
- ezoe 3y agoAre you suggesting Elon found out the true culprit the hard way?
- photonerd 3y agoSome of the scraping was because Twitter fucked up the API so bots moved to scraping. Dumb but obvious consequence
- sgammon 3y agowow. how would one even fix this without deliberate downtime? you'd have to deploy and hope that the frontend will make it through CDNs to reduce pressure, right?
- minimaxir 3y agoAt minimum, you revert the commit/deploy to prod that caused the issue. But then that would likely mean reverting the recent policies and would make Elon look weak, so he'd never support it.
- whateveracct 3y agoYeah frontend retry DDoS is not a great situation to get in. I've tripped it in a test env before with a websocket app (erroneous retries caused certain clients to open the WS over and over and break the server).
- bornfreddy 3y agoYou first remove rate limits, then implement and release exponential backoff on frontend, then apply rate limits again (on a small segment of users first, then more). No biggie, you just need to be very careful. And boss needs to chill for that time, which is unlikely to happen.
- avl999 3y agoFirst thing I would try is seeing if the front end has a different retry strategy for a different status code (say 503). If so I'd change the status returned for throttling to be that (503). Barring that, turning off server side throttling or atleast making it less aggressive to slow the retry storm seems the most reasonable.
- lapcat 3y agoDupe: https://news.ycombinator.com/item?id=36553762 https://news.ycombinator.com/item?id=36553762
- brigadier132 3y agoThis is why you always use exponential backoff.
- fathyb 3y agoAnd when you're at Twitter scale, sprinkle some jitter too.
- oblio 3y agoWhat do you mean?
- jyxent 3y agoAdding some randomization to the exponential backoff times to avoid the thundering herd problem: https://en.wikipedia.org/wiki/Thundering_herd_problem https://en.wikipedia.org/wiki/Thundering_herd_problem
- wolfgang42 3y agoSay you have a bug that caused 100,000 HTTP requests to hang, and you kick the node and make them all fail at once. One second later, 100,000 clients suddenly retry simultaneously, causing a huge spike in load which makes most of their requests fail. They use exponential backoff, so two seconds after that, 99,000 clients retry, causing a huge spike in load that makes most of their requests fail. Four seconds after that, 98,000 clients retry... If you introduce a bit of randomness into the retry timing (say, multiply by 1.8~2.2 instead of a straight doubling), that thundering herd will spread itself out and be much easier to recover from.
- 8organicbits 3y agoJitter is a little randomness in how long clients wait between retries. It ensures that you don't have a "thundering herd" all retrying at the same time. Imagine if your API used exponential backoff of [1s, 2s, 4s, 8s, ...] and a large group of requests gets a retryable error at t=0. They will all retry at exactly t=1, t=2, etc. If the group is large enough that repeated surge of requests can knock you offline. There's nasty form of this where the site is offline for a bit and then all the clients rush their requests in when it comes back online. The client requests are all coordinated on the site recovery time and end up overloading the site with their coordinated retries.
- Topfi 3y agoSpeaking from very painful, personal experience, few things are more agitating than being forced to execute on something you fully know is a horrible idea, especially when you tried and failed to communicate this fact to the individual pushing you to go against your best judgement. Even more so when that person later loudly proclaims that they never made such a request, even when provided with written proof. I can of course not say whether the people currently working at Twitter did warn that the recent measures could have such major side effects, but I would not be surprised in the slightest, considering their leadership's mode of operation. Even as someone who very much detests what Twitter has become over the last few months and in fact did not like Twitter before the acquisition, partly due to short format making nuance impossible, but mostly for the effect Tweets easy embeddability had on reporting (3 Tweets from random people should not serve as the main basis for an article in my opinion), I must say, I feel very sorry for the people forced to work at that company under that management.
- Waterluvian 3y agoAt least once have I done a CC blast as a form of insurance when being asked to do something absolutely boneheaded.
- 7thaccount 3y agoYou gotta get executives in writing sometimes or they'll throw you under the bus later.
- Waterluvian 3y agoIf executives are throwing you under a bus, make an exit. There’s no coming back from a clown show like that.
- jaggederest 3y agoI have a general rule that I make proactive, diligent efforts to inform and correct, and I do it 3 times to 3 different groups in leadership, and if that doesn't work, I'm out. I'm about 2 for 8 but you gotta try sometimes.
- lamontcg 3y agoI think we're about at the point where the people who predicted chaos at twitter after Elon basically fired most of the experienced engineers have been proven correct. The duct tape is all coming apart at the seams now. It isn't quite as decisive as a submarine imploding, and ceasing to exist, but it has turned into a brightly burning tirefire.
- bravoetch 3y ago25 years of products being honed for shareholder value, instead of customer or user value. We may be at peak consumer tolerance for anti-pattern, in-app purchase, subscription-model, ad-packed, data-siphoning, dopamine driven, gated experiences.
- OnlineGladiator 3y ago> We may be at peak consumer tolerance for anti-pattern, in-app purchase, subscription-model, ad-packed, data-siphoning, dopamine driven, gated experiences. As much as I want this to be true, I think this sentiment is really only popular on tech-savvy forums like HN. Most people don't use ad blockers, and I've had people get mad at me when I suggest that they do (directly in response to something where they are complaining about ads).
- tough 3y ago> and I've had people get mad at me when I suggest that they do. lol that's a new one, why?
- NikolaNovak 3y agoNot the op but I have that experience frequently. These are perceived as geeky needy techie things that are not for normal people (sprinkle quotations as needed). It's the same as people getting upset at suggestion to add lock of some sort to their phone (face, fingerprint, whatever) or backup their phone. After some pondering I think it's peoples' insecurity misfiring. They use these complicated layered and potentially risky and dangerous pieces of technology, aware they don't fully understand them, that they work as magic that could stop any moment. Trying to understand and secure them is a massive rabbit hole. So I think there's kind of a rejection to go down that hole or acknowledge the problem or, most of all, face the vulnerability and exposure. My 100 Croatian lipa fwiw :-)
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- shawnc 3y agoMy 14yr old daughter sent me this screenshot the other day with the comment "I think that one sentence sums up twitter pretty well". https://capture.dropbox.com/GqgTAxRimqAXzrdo https://capture.dropbox.com/GqgTAxRimqAXzrdo
- viraptor 3y agoFyi, on mobile the image doesn't load and the download link is broken.
- mkl 3y agoLoads fine for me on Firefox Android (but I don't get which sentence is supposed to sum up Twitter).
- furyofantares 3y agoI see a giant circle around like 10 sentences, none of which feel like the sum up twitter in any way? After some consideration I guess maybe you're referring to "What's happening?"
- shawnc 3y agoFair enough - the sentence in question was indeed 'What's Happening?'
- LexiMax 3y ago> "What's happening?" That's a very good question.
- tedunangst 3y agoEditor's Choice?
- dcunit3d 3y agoYour 14 y/o daughter is fired. I tried to point this out below. It's not even a DDoS, it's fake news. https://news.ycombinator.com/item?id=36561808 https://news.ycombinator.com/item?id=36561808 it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure. the HTTPS TLS terminates at a Google VM, which gets relayed depending on the VIP used to hit it, but the traffic never gets past that Google VM. This is literally /HOW/ companies deal with DDOS. https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/expand-the-edge https://blog.twitter.com/engineering/en_us/topics/infrastruc... it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life"
- deleted 3y ago[deleted]
- charcircuit 3y agoIt's not a self ddos if twitter isn't going down. You can see in the video twitter is properly handling the load and is returning HTTP 429 when the client is sending too many requests. Hitting the rate limiter or requesting a post is light weight. It's not like it's spamming login requests which require a lot of resources due to key stretching.
- thakoppno 3y ago2 Nines is sufficient at scale?
- charcircuit 3y agoFor a consumer facing surface. Yes.
- notquitehuman 3y agoIt’s Twitter. A nine is more than enough for their most critical workloads.
- 8organicbits 3y agoOne nine allows 10% downtime. That's 36 days per year. I doubt that.
- klawed 3y agoWouldn’t 1 nine be 9%?
- 8organicbits 3y agoOne nine is atleast 90.0%. Two nines is atleast 99.0%. Three nines is atleast 99.9%. and so on.
- simonw 3y agoThe theory here is that this JavaScript bug caused the huge increase in requests which is why Twitter introduced strict limits on how many tweets users could access. So the result wasn't an outage, it was a radical reduction in functionality. I think that still qualifies as a self-ddos.
- grumple 3y ago"Yeah, let's roll this out on a Friday night before the biggest national holiday weekend of the summer."
- noncoml 3y agoPerfect example of why you don’t just fire all your senior engineers. You can’t debug and root cause if you do. So you end up adding hacky point fixes
- wand3r 3y agoI definitely agree. I do think the major issue at Twitter is more managerial than engineering though. They likely could have run Twitter at the same output with the current team. The problem is a CEO who is learning on the job by altering basically everything at a superfast pace. The root cause is business & feature experimentation at scale with a tight runway & no executive oversight.
- pschuegr 3y agoTo say he's learning on the job is speculation on your part. He's on the job, I'll give you that.
- wand3r 3y agoI mean, I'll give you that, I did chuckle. Frankly, this whole thing is a huge debacle and I can only imagine the context switch penalty being paid going from cars, to ISP, to rockets to Twitter. That said, I personally am astounded by Elon's tenacity. I would definitely not bet against it succeeding, but im not sure I'd bet on its success. If I had to choose though, I would bet on success at like 49/51 odds
- pschuegr 3y agoHe may be tenacious, or at least has been in the past. And TBH I hope that Twitter succeeds, but it's becoming pretty clear that my definition of success is different than his.
- pfisch 3y agoThreads is about to launch and I have a feeling once that happens twitter will quickly become the next Truth Social.
- rsynnott 3y agoOnly without the main attraction. Always seemed like a weird strategy, that.
- ceautery 3y agoCentral Services finally caught Archibald Tuttle, now everything is falling apart.
- southwesterly 3y agoBrilliant reference. Absolutely zapped it. Legend.
- EdwardDiego 3y agoBloody typical, they've gone back to metric without telling us.
- zagrebian 3y agoThe Elon Twitter movie will be the next Borat, I can’t wait.
- hinkley 3y agoDon't interrupt your opponent when they are making a mistake.
- justinhj 3y agoWhy do people oppose Twitter? It's a social network, you can choose not to use it.
- fundad 3y ago[flagged]
- TheBlight 3y agoHow is he racist? Can you cite an example?
- phist_mcgee 3y agoHere you go: Rate limit exceeded Please wait a few moments then try again.
- vuln 3y ago[flagged]
- deleted 3y ago[deleted]
- andsoitis 3y ago> Not Twitter, the racist bullshit artist who bought it. Can you explain why a racist would give internet access (Starlink) to 50 schools in Rwanda? https://twitter.com/Starlink/status/1671930585475055618 https://twitter.com/Starlink/status/1671930585475055618
- abenga 3y agoIs this the philanthropic "I can't be racist! I have Black friends!" reply?
- DecXicon-28 3y agoUnlike what people imagine, the selfDDOS bug that occurs does not burden requests to the Twitter system, but becomes a loop for users requesting access.
- dcunit3d 3y agoyeh, i tried to mention this, but it got flagged quick. it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life" https://news.ycombinator.com/item?id=36561808 https://news.ycombinator.com/item?id=36561808 it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/expand-the-edge https://blog.twitter.com/engineering/en_us/topics/infrastruc...
- DecXicon-28 3y agoNot as people imagine , the selfDDOS bug that occurs does not burden requests to the Twitter system, but becomes a loop for users requesting access.
- wzy 3y agoSo this "unforeseen" loop now eats in the users 600 tweets per whatever, so now when you land and Twitter's homepage you are pass this measly limit with 5 rotations of your scroll wheel.
- danShumway 3y agoI do think this probably isn't the reason for rate-limiting tweets so harshly; I don't know for sure, but I suspect this isn't a technical issue and it is legitimately Elon snapping and just implementing a wildly bad business decision. But that being said, a loop of users requesting access from Twitter's system is a thing that would burden Twitter's system. Requesting access and denying access isn't free. You can definitely DDOS your own website by having all of your users repeatedly request access to it in a loop.
- nwoli 3y agoSeems like it might not even be a bug. Elon says they limited it to 600 viewed tweets per day which is an insane limit. Most people would go through that in 5 minutes of scrolling
- samb1729 3y agoI noticed the frontend hammering the backend for the past few weeks, so I suspect that these new rate limits are a response to that, even if Musk wouldn't publicly admit it. I don't doubt that Twitter saw a massive increase in traffic recently, but I feel at least somewhat confident that it's mostly self-inflicted on Twitter's part.
- impissedoff1 3y agoMight be time to reflect on how much useless information we consume
- johannes1234321 3y agoSo many things are useless, but still entertaining in one way or another. And then Twitter is used for "relevant" information as well, from emergency or transport authorities using it for quick information instead of having their own system to news of any kind or discussions and information exchange in different peer groups. Considering that they count all tweets being loaded towards the quota (for instance replies to a tweet, if you open the detail page) and they like to shove tweets from Musk and others in the timelines the quota can be reached quite quickly even on considerate use.
- jjulius 3y ago>And then Twitter is used for "relevant" information as well, from emergency or transport authorities using it for quick information instead of having their own system to news of any kind or discussions and information exchange in different peer groups. I really don't think it's wise to be putting so many eggs in one basket, convenient as that might be. This right here is a strong example of that.
- aeyes 3y agoThis bug is very unlikely to be the reason. The rate limiter on the server side is cheap and the frontend bug only gets triggered with the rate limit active. I have seen similar bugs in the systems I oversee because network libraries love to retry requests without sane limitations by default. But I never saw them make our rate limiters sweat. It's slightly more annoying when they hit an API which actually does some expensive work before returning an error but that's why we have rate limits on all public endpoints. I also guess that the webapp is the least of Twitters traffic and the native apps probably don't have this problem.
- epistasis 3y agoOne thing about having leadership that is known to lie about anything or everything, for any sort of imagined personal gain, is that the very concept of truth is destroyed. I agree that this is probably not the bug at the root of it all. But I also don't believe the story that Musk is selling for why he's in effect shutting down the site. But both could be true and I'm still thinking about other potential reasons, a complete waste of my time, but it's a weird mental honeypot. The book "Nothing is true and everything is possible" describes Putin's use of misinformation to maintain control of the populace and eliminate democratic types of politics, but it really feels like it applies here too. There will always be Musk fanbois who will parrot whatever he wants them to say, but most know it's just self-serving BS. And anybody trying to get to the root of everything gets easily sidetracked into narratives that feel right but have zero data backing them, like this bug. Anyway, highly recommend this book if you want to see a likely path for the future of the US: https://en.wikipedia.org/wiki/Nothing_Is_True_and_Everything_Is_Possible https://en.wikipedia.org/wiki/Nothing_Is_True_and_Everything...
- kristianc 3y agoThat’s exactly it. Musk a week ago was telling us that there were a record number of user seconds on the site. Now he’s telling us they’re all content-scrapers. The very concept of truth is eroded.
- partiallypro 3y ago
- fwlr 3y agoOh come on. This is console.log spam. This isn’t a “self-DDoS”.
- mananaysiempre 3y agoNo, look closely: the animation does not show the console, it shows the network pane. Each line is a (presumably unsuccessful) request to the Twitter backend. If a lot of people have the Twitter web frontend open and running in this state, it could in fact overload the backend.
- fwlr 3y agoSure, I mis-spoke in saying it was console.log spam, that’s not specifically what’s happening. But let’s be real, if you’re getting a 429 response that is at most a cached hash table lookup per request, for a site has probably top 50 in the world in terms of “scaling to handle heavy request loads” for a decade. There’s just no chance this specific thing is actually what’s causing issues for Twitter, it’s obviously a consequence of the heavy scraping and steps to stop the heavy scraping. It frustrates me to no end that smart technical software people, who can have intelligent discussions on not just code but a wide range of topics from nuclear powers to superconductors, suddenly lose their ability to have these discussions when a certain person is involved.
- bubblematrix 3y agoAgreed, it's fud - and there are professional botnets on HackerNews to amplify what they want. What I've noticed is there's little technical discussion, and more so emotional-provoking replies.
- dcunit3d 3y agoTHANK YOU! i tried to mention this, but it got flagged quick. it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life" https://news.ycombinator.com/item?id=36561808 https://news.ycombinator.com/item?id=36561808 it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/expand-the-edge https://blog.twitter.com/engineering/en_us/topics/infrastruc...
- reneberlin 3y agoThe kind of input you have when feeding tweets into an LLM is toxicity. It looks like something you can blackmail LLM for. Look, you have all this toxic trash in your model. Like: we have the sources and you could detox you model if you pay for it.
- jaimex2 3y agoSo you say, but its been working fine all day so... Yeah, gonna keep using it.
- jaimex2 3y agoits still online...
- lmeyerov 3y agoApparently their 5-year cloud contract failed to renew today So, there may be a much simpler explanation for why their new rate limits on regular users to can-barely-scroll levels, and how that has all sorts of unintended consequences they weren't ready for
- ikrenji 3y agogoogle wouldn't just shut off services for a big customer like that...
- lolinder 3y agoIf someone completely stops paying the bills, it doesn't really matter how high of a roller they are, you can't keep serving them indefinitely.
- dehrmann 3y agoA medium-sized customer making a point of not paying the bills to negotiate hard? If they come back, I'd put them on a pre-paid plan.
- wnevets 3y agoDon't you have to pay your bill to be considered a customer?
- playingalong 3y agoNo.
- manquer 3y agoLegally no, you are a customer if you have a contract . same way we are a tenant( and have rights) even if you don’t pay rent. Disputes are resolved via arbitration and courts , no payment does not give GCP ( or any vendor) unilateral right to terminate (the contract however possibly might ) . Usually vendor has to give notice of termination and there would be a time window before termination can occur . Unpaid dues are like unpaid mortgage there are many steps before the vendor can get a court to force the party to make payment or seize assets in lieu
- sakopov 3y agoWhat I find more ironic than this bug is is that all of these Mastodon evangelists still have Twitter accounts after their Elon meltdown. So much so that they even use it after Twitter stopped public access.
- ImaCake 3y agoHave you considered that a lot of people use both platforms and don’t really care about the politics of it? Those who do care are very loud (on both platforms) but they might be a minority in practice.
- raincole 3y agoYou're replying to a thread on HN. The thread is a link to a Mastodon post. The post has the following tags: #TwitterDown #MastodonMigration #DDOS #TwitterFail #SelfDDOS This is why context matters.
- rsynnott 3y agoI’ve been off Twitter since naughty ol’ mr car started his war on the journalists (I did return briefly to make a silly joke when he briefly banned mention of the dread term ‘mastodon’). I’ve kept the account, because, well, I’ve had it for over 15 years (back when they had a visible auto-incrementing numeric ID it was six figures), and maybe at some point he’ll lose interest and Twitter will return to semi-competent (it was never great) management. I suspect a lot of people are in the same boat. But also, yes, I use it to observe the latest meltdown; why shouldn’t I?
- jjulius 3y agoBecause maybe it'd be better if we stopped giving him attention. He thrives on it in a negative way.
- summerlight 3y ago> If you missed the initial news of the impending showdown, Platformer reported on June 10th that Twitter had been refusing to pay Google for its cloud services ahead of their contract’s June 30th renewal date. > Twitter’s Google Cloud contract dates back to 2018. https://www.engadget.com/twitter-has-supposedly-started-paying-its-google-cloud-bill-again-213824844.html https://www.engadget.com/twitter-has-supposedly-started-payi... Oh... this explains all this fiasco :facepalm:
- partiallypro 3y agoAs the article says, and as Bloomberg, etc have also reported. Twitter finally agreed to play nice with Google and the issue was resolved. They are probably trying to get off of GCP, but this isn't because they suddenly lost access to GCP from payment refusal.
- deleted 3y ago[deleted]
- BlackjackCF 3y agoWell, I'm sure that migration off of GCP will go smoothly with no problems. Won't cause any more stability issues with Twitter. /s
- 18pfsmt 3y agohttps://www.reuters.com/technology/twitter-resumes-paying-google-cloud-bloomberg-news-2023-06-21/ https://www.reuters.com/technology/twitter-resumes-paying-go... That's from a week ago.
- mmcwilliams 3y agoThe sourcing on that story is a single unnamed person whereas both companies refused to comment. Seems like there could be more to the story than "Twitter started paying again and everything's fine."
- nightshadetrie 3y ago
- fredgrott 3y agoUse A marketing idiot as engineer chief, win stupid prizes
- Coneylake 3y agoHave they tried batching API calls? /s
- xains 3y ago[flagged]
- mcint 3y agoMatters little, but the rate limits are opening back up. 6k/600/300 -> 8k/800/400 (~noon) -> 10k/1k/500 (~3pm) https://twitter.com/elonmusk/status/1675214274627530754 https://twitter.com/elonmusk/status/1675214274627530754 and self-reply
- kennywinker 3y agoI can't read that. "Something went wrong". And I believe under the new logged-in-only rules, I wouldn't even be able to read it if the site was up. Twitter links are now poisoned, and basically useless.
- justcool393 3y agohere are the tweets >To address extreme levels of data scraping & system manipulation, we’ve applied the following temporary limits: > >- Verified accounts are limited to reading 6000 posts/day >- Unverified accounts to 600 posts/day >- New unverified accounts to 300/day https://twitter.com/elonmusk/status/1675187969420828672 https://twitter.com/elonmusk/status/1675187969420828672 > Rate limits increasing soon to 8000 for verified, 800 for unverified & 400 for new unverified https://twitter.com/elonmusk/status/1675214274627530754 https://twitter.com/elonmusk/status/1675214274627530754 > Now to 10k, 1k & 0.5k https://twitter.com/elonmusk/status/1675260424109928449 https://twitter.com/elonmusk/status/1675260424109928449
- kevinventullo 3y agoCan you imagine Mark Zuckerberg writing these type of granular site outage status updates?
- NorwegianDude 3y agoI have an idea. We should create a scaping system that scrapes all tweets and makes them availabe so that people are able to see what other people think they are linking to when they link to a broken Twitter page. /s Either that or people should stop linking to broken pages.
- 3y ago
- TheBlight 3y agoI don't see any unusual behavior when looking at the debugger in Chrome. People have a political issue with Elon. Which is fine. People are entitled to it.
- eyelidlessness 3y agoThe post suggests a race condition as the underlying cause of the behavior shown in screen captures. It’s perfectly possible the race condition does exist, and being a race condition you just didn’t happen to trigger it. It’s also perfectly possible to discover such a defect with or without any personal bias towards Musk or Twitter. Reporting it publicly this way can also be a favor to both, as it gives them a fairly malleable narrative for reversing course, or directing responsibility, or both. Even if it’s simultaneously embarrassing to have it out there. And it’s not like either Musk or Twitter is a stranger to embarrassment, or particularly shy about courting it.
- TheBlight 3y agoClearly there is something wrong with the site. So I don't doubt there is a bug. But the OP gives no helpful information that could be used to reproduce the issue and fails to describe it in any helpful detail really. They seem content to just effectively say "Twitter is dumb." Which, again, is fine but not all that technical and interesting.
- eyelidlessness 3y agoThere’s information in the post that very probably points at the underlying cause. The scrollbar jumping is a dead giveaway: Twitter loads more content based on scroll position. If your viewport is sized such that displaying the error message triggers another load event by its corresponding change in scroll position, it will try again indefinitely absent some additional logic to bail out. Edit: this of course depends on some other implementation details, like the rendering flow and browser behavior. If showing the error is memoized, it shouldn’t trigger the loop unless they’re also rendering some intermediate loading state asynchronously.
- berniedurfee 3y agoTwitter told me I spend too much time on Twitter. I agree. Thanks!
- tmpz22 3y agoCan we please recognize that this is a holiday weekend and Elon rammed a big release forcing his engineers to come in and iterate multiple patches over the last 12 hours.
- truemotive 3y agoextremely recognized.
- EGreg 3y agorecognized even more
- gitfan86 3y ago[flagged]
- intunderflow 3y agoSome of the remaining engineers may be stuck due to H1B
- ertian 3y agoCanada just changed their immigration program such that they'll accept all H1B holders (working or not) into Canada with a similar visa, but without the requirement to find a job immediately. Something to look into for any Twitter (or other) engineers who are bound to a company due to lack of options. https://www.forbes.com/sites/andyjsemotiuk/2023/06/28/us-h1-b-visa-holders-targeted-by-canadas-new-immigration-program/ https://www.forbes.com/sites/andyjsemotiuk/2023/06/28/us-h1-...
- intunderflow 3y agoSure, but if you've built your life in the United States that still means moving country
- 3y ago
- highmastdon 3y agoWhat a toxic thread... Both here and the post on sfba.social
- tasubotadas 3y agoLots of FAANG developers here hate Musk because he has shown that 80% of them is a waste of salary.
- BryantD 3y agoTwitter has been severely hampered for 12+ hours, and that’s only the latest issue. I don’t think Musk has demonstrated anything conclusively quite yet.
- x86x87 3y agoLol. Right on buddy. You know what all corporations should do to maximize short term profit? Fire everyone. It's amazing how much money we spend on people /s
- rs999gti 3y agoSounds more like bad QA that forgot to test an use case
- minimaxir 3y agoIn that case, it's not QA's fault, it's the organization's fault since it hit production already.
- throwawayswiss 3y agoAt my last start-up, there were 3-4 times the number of engineers and yet they failed to do basic testing and engineers ended up do that job too. Funnily some of them were paid more than engineers
- teslashill 3y ago[dead]
- seadan83 3y agoFocusing on the technical issue at hand - can anyone tell the exact issue at play? Is this an issue of client retry logic that on a 429 without backoff? Hence, the self-ddos?
- phillipcarter 3y agoAll Elon had to do was leave it alone and, oh, right, the 44 billion. Hahahahaha! Man, Parag Agrawal and his team knew exactly what they were doing. A fool and his money are soon parted, indeed.
- bottlepalm 3y agoPlaying 5 dimensional chess with that poison pill huh?
- Sai_ 3y agoI have an alternative theory for this. Once Elon was forced by the DE court to buy Twitter, he went to autocrats and dictators with money and told them he'd burn Twitter to the ground on their behalf for 44B. No Twitter, no Arab Spring. No live updates of disasters, no need to shutdown the internet to prevent dissenting voices from spreading.
- adamcblodgett 3y agoAre we doing theories? Mine is that Peter Thiel sold Musk on buying Twitter by convincing him of his ability to make it profitable with his genius, knowing full well he'd sink the thing in 18 months, and another pesky tool would be off the table for awhile.
- extasia 3y agoThat theory is pure speculation on the basis of motive with no actual grounding in reality, no offense. There's not a shred of evidence that GlobalAutocracy™ banded together to cough up $50B to kill of Twitter...
- regularjack 3y agoYou thought Musk was playing 4D chess? Think again. He was in fact playing 29D chess, with dictators, and autocrats.
- inpdx 3y agoPeople commenting that people who are still at Twitter because of their own volition, just stop. Certainly some are still there because they want to be, but there are absolutely people trapped there economically one way or another, working for a tyrant.
- TheHappyOddish 3y agoWhat nonsense. Silicon Valley employees are some of the best paid in the world and have chosen to enter into a contract in a capitalist society. Even if any are "economically trapped" (they're not) and it's not of their own volition (it is), it's been a year since Musk bought the company, plenty of time to find another job. I don't think you understand what a "tyrant" is. Companies aren't democracies.
- aetimmes 3y agoAnecdotally, you are wildly incorrect.
- CodeWriter23 3y agoSo move fast and break things apparently isn’t HN’s chosen methodology any longer.
- minimaxir 3y agoHN was mocking Facebook's slogan even a decade ago after their numerous outages back then. They had to change it to "Move Fast and Build Stable Infra" for a reason.
- chaxor 3y agohttps://youtu.be/FeAMiBKi_EM https://youtu.be/FeAMiBKi_EM The webdev or JavaScript mentality of if it breaks, don't worry, we can fix it live strikes again.
- mrcwinn 3y agoI assume the new CEO will hold Twitter’s tech leader responsible.
- 0xDEF 3y agoI made an account around the time ChatGPT API (GPT-3.5) access was released to follow all the LLM news. The account is now getting aggressively rate limited that I cannot even see new tweets for 2-3 minutes.
- pineapple_guy 3y agoOn a side note, I must say that Mastadon having a side bar on mobile is a very poor design choice.
- pschuegr 3y agoI want to agree with you since I generally hate this kind of thing but honestly the Mastodon web ui on mobile doesn't bother me for some reason.
- throwawayswiss 3y agoIt's a waste of space but in overall it looks amazing
- m3kw9 3y agoWorked fine with me. It’s a bit annoying seeing a mastodon user shtting on twitter, there was no self own, the post sounded good because it resonates with Elon haters.
- justinator 3y agoDid the Fail Whale appear, or was it too busy capsizing boats?
- penjelly 3y agothe irony of all the premature celebrations over the past 8 months. If you stick your head in the sand and pretend there are no problems, i guess there really arent any, until maybe there is?
- runsWphotons 3y agoIs there anywhere in this thread where there is actual info and not just people complaining about and taking shots at Elon Musk because they dont like him or his politics?
- dcunit3d 3y agoyeh, i tried, but it got flagged quick. https://news.ycombinator.com/item?id=36561808 https://news.ycombinator.com/item?id=36561808 it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/expand-the-edge https://blog.twitter.com/engineering/en_us/topics/infrastruc...
- olalonde 3y agoIt's an interesting theory but the DDoS predates the decision to disable anonymous access. In fact, that decision was taken in order to mitigate an ongoing DDoS[0][1]. So while it's possible that the questionnable web front end retry logic is not helping things, it's not the root cause. [0] https://twitter.com/elonmusk/status/1674865731136020505 https://twitter.com/elonmusk/status/1674865731136020505 > Temporary emergency measure. We were getting data pillaged so much that it was degrading service for normal users! [1] https://twitter.com/elonmusk/status/1674942336583757825 https://twitter.com/elonmusk/status/1674942336583757825 > This will be unlocked shortly. Per my earlier post, drastic & immediate action was necessary due to EXTREME levels of data scraping. > Almost every company doing AI, from startups to some of the biggest corporations on Earth, was scraping vast amounts of data. > It is rather galling to have to bring large numbers of servers online on an emergency basis just to facilitate some AI startup’s outrageous valuation.
- koboll 3y agoHonestly it probably has more to do with the decision to basically shut down API access without exorbitant fees than it has to do with AI startups or whatever. Surely there are some colossal archives of all tweets before whatever date floating around that those startups are using instead.
- tsimionescu 3y agoThat is a post-backlash claim by Elon Musk, so it should be taken with a huge grain of salt.
- olalonde 3y agoIt seems more plausible than the alternative. Let's say that Twitter availability issues were really caused by disabling anonymous access and that Elon wants to back out of his decision due to public backlash, wouldn't he just reverse the decision? That would fix both of his issues at once. He could even claim to have solved the scraping issues through a clever trick that doesn't require disabling anonymous access. The fact he hasn't done so indicates that there is a genuine issue that he can't just fix by restoring anonymous access.
- brigandish 3y ago> There must have been an insane nest of logic that the devs had to tiptoe through to make the service workable as a private only service. It seems that this statement can lead to different conclusions based on one's biases for or against Musk / freedom of speech: 1) The code base is a mess because the smaller, current team don't know what they're doing and/or management (i.e. Musk) is pushing them too hard and making poor decisions 2) The code base is a mess because the far greater number of engineers there before didn't know what they were doing and/or management weren't pushing them hard enough and were making poor decisions. Both could be true but I'm going with something the more complex view that it's a little bit of everything, and that Twitter does seem to be moving in a positive direction overall.
- makeitdouble 3y agoThis makes me realize "the code base is a mess" has been so overused it could be a standard greeting in our profession. Everyone has their horror story and extreme examples, but for any mildly profitable company that went through a decade or more of operation, I think it's kinda of a meaningless statement.
- brigandish 3y ago> This makes me realize "the code base is a mess" has been so overused it could be a standard greeting in our profession. I'm printing up t-shirts already, pre-order now and get 10% off the insane nest of logic tee while stocks still last!
- smcl 3y agoWhy do you say Twitter is moving in a positive direction? My experience is that I dislike their app and they killed the one I liked (Tweetbot), it's frequently down (I still get the "Something went wrong" error), and any kind of conversation is dead because blue-check replies (usually quite low-quality) are prioritised. There are other reasons to dislike today's Twitter that I needn't go into here, but these are enough to turn me away any time I decide to come back and check it out again.
- andreygrehov 3y ago> Twitter is firing off about 10 requests a second to itself to try and fetch content that never arrives because Elon's latest genius innovation is to block people from being able to read Twitter without logging in Well, for one, this sounds just like yet another Elon-hater's comment. In _most_ cases, Elon's businesses are fairly successful, so a sarcastic "Elon's latest genius innovation" doesn't belong here. For second, how do you know the idea came directly from Elon? It could easily be the result of a brainstorming session led by a product team. Elon is not writing Twitter's code. He is not a Software Engineer at Twitter. Even if the idea came from him, why is he being attacked for a bad deployment? And lastly, why do people think that the change is bad? There are plenty of web sites that do not allow you to view their content without creating an account first. Is Twitter not allowed to experiment? Why not? I would appreciate hearing well-thought-out arguments that explain the potential negative impact on the business.
- vore 3y agoHe did make the call to do that, did he not? He publicly stated it on Twitter.
- andreygrehov 3y agoWhy does it matter that he made the call to do that? I get plenty of "do this, do that" requests from my manager. And if I break things, is it my manager's fault? I'm trying to understand your logic. Please, explain. Edit: here is my take on it + the most likely scenario of what's going to happen next. This was a bad deployment and such things are not unusual in software industry. In this specific case, there are zero reasons to blame the CTO. From what I'm observing right now, Twitter has already fixed the issue - the website loads just fine. Next, they will do a retro, learn from their mistakes and try to not repeat the same mistakes again in the future. There will be more bad deployments and that's normal. However, with time, they'll make things better, the SLA will go up and the overall stability of their services will stabilize, – one of the biggest social networks on the planet will have the least number of engineers running that same social network.
- 3y ago
- alfor 3y agoThis is expected with a faster rate of pushing features. Funny things is Twitter views are hiher than ever and it will be fixed quickly. (No reason to make a big deal out of it) But I think tests and refactor of the codebase will be necessary to increase their speed of innovation.
- indus 3y agoWasn’t Elon our North Star first principle thinker?
- jrockway 3y agoWho knew that firing all of your software engineers could make for bad software.
- deleted 3y ago[deleted]
- sh34r 3y agoI'm astounded it took this long to finally die. All those Twitter engineers who were harassed by Elmo's very online fan club, and were treated with such cruelty and disdain on their way out, clearly did an incredible job. Hats off to them. The curse of IT Operations: if you do everything right, management grifters never think you did anything at all. Why do I need you? Everything works fine! Most Internet platforms are put together by popsicle sticks and bubble gum. If Reddit didn't have 2000 full time employees babysitting that steaming POS, it would be offline by the end of the week. But Twitter was like a chicken running around with its head chopped off, and it didn't stop running for 9 months. It's a testament to how incredible those engineers were. This collapse at Twitter should have happened by January. It's a real shame it didn't. How many thousands of tech workers were laid off because Musk's fellow parasitic oligarchs saw Twitter running "fine" after the lobotomy, and followed him off the cliff like a bunch of lemmings? How many billions of dollars has the economy lost because of this one despicable man? Elon Musk is a fraud. He is not an engineer. He's a lazy bum who mooched his whole life off his groomer daddy's apartheid emerald money. He would be nothing without that disgusting, incestuous old man. Despite his enormous financial privilege, Elmo was too lazy to enter the US legally. He was in the US for many years as an illegal immigrant. He only bothered to get off his lazy ass and finish his Economics degree because the risk of his deportation was becoming a real problem for X.com. That crooked philistine doesn't know the first thing about hard work, engineering, or finance. He's a damn good conman, he's good at shitposting, and that's about it. He has no other skills. He is not an engineer. He has no STEM degree. His only real accomplishment in life is proving that you can, in fact, spend billions of dollars in one lifetime, if you simply buy a bird app, run it into the ground, and salt the earth behind it.
- quenix 3y ago> He has no STEM degree. Musk has a bachelor's degree in physics from the University of Pennsylvania: https://www.snopes.com/fact-check/musk-physics-degree/ https://www.snopes.com/fact-check/musk-physics-degree/. Further, while I share some criticism of the man, many very technical people from companies such as SpaceX and Tesla have come forward with public comments, praising him for "truly grasping the engineering" and "being involved in every technical design decision". Make of these what you will: Kevin Watson, Falcon 9 avionics: Elon is brilliant. He’s involved in just about everything. He understands everything. If he asks you a question, you learn very quickly not to go give him a gut reaction. He wants answers that get down to the fundamental laws of physics. One thing he understands really well is the physics of the rockets. He understands that like nobody else. The stuff I have seen him do in his head is crazy. He can get in discussions about flying a satellite and whether we can make the right orbit and deliver Dragon at the same time and solve all these equations in real time. It’s amazing to watch the amount of knowledge he has accumulated over the years. Tom Mueller, SpaceX founding employee: We’ll have, you know, a group of people sitting in a room, making a key decision. And everybody in that room will say, you know, basically, “We need to turn left,” and Elon will say “No, we’re gonna turn right.” You know, to put it in a metaphor. And that’s how he thinks. He’s like, “You guys are taking the easy way out; we need to take the hard way.”* And, uh, I’ve seen that hurt us before, I’ve seen that fail, but I’ve also seen— where nobody thought it would work— it was the right decision. It was the harder way to do it, but in the end, it was the right thing.* Garrett Reisman, engineer and former NASA astronaut: What's really remarkable to me is the breadth of his knowledge. I mean I've met a lot of super super smart people but they're usually super super smart on one thing and he's able to have conversations with our top engineers about the software, and the most arcane aspects of that and then he'll turn to our manufacturing engineers and have discussions about some really esoteric welding process for some crazy alloy and he'll just go back and forth and his ability to do that across the different technologies that go into rockets cars and everything else he does. Josh Boehm, former Head of Software QA at SpaceX: Elon is both the Chief Executive Officer and Chief Technology Officer of SpaceX, so of course he does more than just ‘some very technical work’. He is integrally involved in the actual design and engineering of the rocket, and at least touches every other aspect of the business (but I would say the former takes up much more of his mental real estate). Elon is an engineer at heart, and that’s where and how he works best. Along with many others.
- deleted 3y ago[deleted]
- pc_edwin 3y agoIts funny how the entirety of silicon valley just forgot that Twitter was infamous for their unholy mess of a codebase which at some points looked like it had a secret mission to increase world entropy and terraform the planet. I still remember watching podcasts, reading blogposts, hearing prominent VC's like Elad talking about etc.. Now everbody silent because Elon is the baba yaga...
- pc_edwin 3y agoTo go further, its crazy how much cognitive dissonance can penetrate even the smartest among us.. If you had some understanding of how modern systems at twitters scale works, you'd know such a bug is highly unlikely. You have multiple clients (android, ios, web) and a whole cluster of microservices calling each other. The microservices calling each other will easily use up more requests. IMO the likely culprit is rate limits being triggered by a huge increase in scraping caused by the new api pricing.
- guax 3y agoThe difference is that Twitter was not arrogant about it. Saying they where hardcore engineers with insane working hours and other bollocks. Musk dislike comes from him being extremely dickish about other peoples work while being an absolute clusterfuck of a manager. Twitter was not bad because of its tech. Musk made it worse.
- didip 3y agoAny reading materials on how messy the codebase was?
- tzekid 3y agoNot quite reading material, but look up George Hotz's videos about it. I think there's one Twitter Space he hosted / recorded where Musk was also in there. Hotz is very vocal and blunt about the messy codebase situation at Twitter.
- eql5 3y agoHere's a guy who understands what's REALLY going on: https://twitter.com/i/status/1675254646942896128 https://twitter.com/i/status/1675254646942896128
- yafbum 3y ago"Something went wrong. Try reloading" Man, we do need a public square.
- hiepph 3y agoAlso 500 for me. This is almost a joke. At least if it's blocking me from viewing the tweet's comments, returns me some decent messages. But no, just a plain "Something went wrong." like a hackathon page.
- kaonashi 3y agoAmerican censorship has long relied on the manipulation model rather than the overt removal model, because the former is far more effective.
- Eric_WVGG 3y agoI was skeptical, but this was a really good watch. Pity that it’s a video. For the to;dw crowd: he makes the case that Twitter is by far the most useful social network to government agencies, because the graph is mostly public (unlike Facebook) and all users are both consumers and creators (unlike YouTube). Thus, it is being used by “researchers” and state departments to build what he calls “the Death Star of AI censorship engines,” and these moves to rate limit will cripple those efforts.
- jquery 3y agoWhen people say "Elon is a cult" this video is exactly what they mean. Elon does something incredibly dumb and you immediately have people just making up, whole cloth, fantastical reasons why it's a 5D chess master plan. It's comical at this point.
- ajuc 3y agoHow is Elon allowed to transport people to space?
- activiation 3y agoLuckily he is not the one doing the work
- PaywallBuster 3y agodrama queen 429 requests handled at the load balancer are extremely light weight
- deleted 3y ago[deleted]
- rochak 3y agoElon is truly one of the CEOs of all time
- activiation 3y agoThere's a word missing... Starts with W
- x86x87 3y agoWonderful? /s
- continuational 3y agoThe comments in there gloating over somebody else's mistakes is exactly why I don't use Twitter - except it's apparently the same on Mastodon.
- insaider 3y agoMy favourite part of this whole fiasco is that 'Space Karen' is now trending on twitter
- timeon 3y agoLess funny is that those people are still on twitter.
- choas 3y agothey forgot to fix the Settings page on anonymous tweets which sending constantly data and receives a HTTP 429 Too Many Requests
- therouwboat 3y agoI managed to get in without login few minutes ago and its still sending api request every 130ms and getting 429 Too Many Requests error, how is this possible? :)
- dcunit3d 3y ago[flagged]
- dcunit3d 3y ago[flagged]
- joeythedolphin 3y agoIt seems the majority of Twitter articles are about the poor quality of engineering solutions or Elon's inability to "pay his Google bill". It strikes me as propaganda when what we are truly witnessing is the rush to wall-garden the internet as AI LLM takes over.
- Patrol8394 3y agoWe are staring to see the effects of the layoffs…it can only get worse from here…
- CatDadCode 3y agoI finally did it. I finally got rid of my Twitter account and migrated to Mastodon. I wish I had done it sooner because Mastodon is great. Particularly the Hachyderm.io server for the tech community <3