7 ms·
This is interesting. Judging from the screenshot, a huge amount of GET /TweetDetail is generated which triggers some rate limiting, as shown by the 429. If th
by arter4 3y ago
This is interesting.
Judging from the screenshot, a huge amount of GET /TweetDetail is generated which triggers some rate limiting, as shown by the 429.
If this is indeed due to the recent decision to enforce authentication for all API calls, it means the curlprit may actually be the API gateway or something similar downstream.
Also, this behavior seem to never stop, which isn't what one would expect from an exponential backoff retry.
I don't claim to be a better engineer than the folks working at Twitter, but it is interesting to see something like this in the wild, all Musk-related considerations aside.
- deleted 3y ago[deleted]
- bheadmaster 3y ago> If this is indeed due to the recent decision to enforce authentication for all API calls, it means the curlprit may actually be the API gateway or something similar downstream. The way I understand it, DDoS is not caused by enforced authentication - enforced authentication is just a temporary measure against DDoS.
- Quarrelsome 3y agoI would guess the front end was written under the assumption that the back end would still work without auth. Perhaps the backend changes (mandatory auth + rate limiting) were pushed without sufficient testing of the front + back?
- cactusplant7374 3y agoDid Elon pay the AWS bill? That seems like a likely culprit. Twitter instances are being forcibly shutdown.
- amluto 3y agoTwitter operates its own datacenters.
- cactusplant7374 3y ago"Twitter and AWS signed a five-and-a-half-year contract in 2020, which AWS is not willing to renegotiate." https://gritdaily.com/twitter-owes-aws-millions/ https://gritdaily.com/twitter-owes-aws-millions/
- williamsmj 3y agoTwitter.com and the associated user-facing services do not run on AWS.
- vGPU 3y agoWhile it looks like they never started the move over to AWS, the press release makes it sound like they do use some AWS services. > In addition, Twitter will continue to use AWS services such as Amazon CloudFront (AWS’s fast content delivery network service that securely delivers data, videos, applications, and APIs with low latency and high transfer speeds to customers globally) and Amazon DynamoDB (AWS’s key-value database that delivers single-digit millisecond performance at any scale).
- williamsmj 3y agoI worked there. Services running on GCP are a significant part of the internal service infra (ml platform, etc.) and it's not impossible that the abrupt loss of GCP would cause user-facing problems. The GCP spend was many, many times the AWS spend. Unless things changed since last November, AWS is not a meaningful part of the internal or user-facing infra. With respect to DynamoDB specifically, Twitter has its own custom distributed key-value store: https://blog.twitter.com/engineering/en_us/a/2014/manhattan-our-real-time-multi-tenant-distributed-database-for-twitter-scale https://blog.twitter.com/engineering/en_us/a/2014/manhattan-... that twitter.com itself runs on.
- 18pfsmt 3y agoThanks for weighing in with some actual first-hand knowledge. It is appreciated. The latest on cloud hosting is from a week ago, and I'm guessing you don't have any more recent info than this: https://www.reuters.com/technology/twitter-resumes-paying-google-cloud-bloomberg-news-2023-06-21/ https://www.reuters.com/technology/twitter-resumes-paying-go...
- badwolf 3y agoWell, they haven't paid their GCP bill... https://theconversation.com/twitter-is-refusing-to-pay-google-for-cloud-services-heres-why-it-matters-and-what-the-fallout-could-be-for-users-207718 https://theconversation.com/twitter-is-refusing-to-pay-googl...
- colechristensen 3y agoThis is not remotely a likely culprit.
- romseb 3y ago"curlprit" for too many GET's causing a 429 is just the perfect typo.
- readyplayernull 3y agoCould someone report the error at press@twitter.com and see what they think about it?
- vuln 3y agoQuite the knee slapper. Thanks. I hope you didn’t spend many brain cycles on it.
- Scalene2 3y agoMaking jokes about Twitter is too easy.
- meepmorp 3y agoWhy do you take people criticizing Elon Musk so personally?
- oneeyedpigeon 3y agoDo you think it's more or less funny than auto-replying to all PR enquiries with poo, when you are an incredibly large company?
- ineedasername 3y agoWhoever is in charge of that account went all Oregon Trail on things and caught dysentery
- kccqzy 3y agoWhile exponential backoff is theoretically optimal, I doubt it's actually used that often in practice. I've seen too many cases where someone decides serving user requests with low latency is so important that they'd rather have a constant randomized backoff than exponential backoff. I've been in many design meetings and seen enough documents where the decision not to use exponential backoff is explicitly made, understanding the tradeoff with overloading and system recovery.
- klabb3 3y agoI wouldn’t be surprised if this has no back off or limit at all. 10 RPS is fast enough that it may simply be sequential.
- colechristensen 3y agoI’ve had to… uhh… eagerly advocate for exponential backoff for weeks of constant uptime issues before someone listened and actually implemented it and solved the problems. Like several times in different roles. People do it, exponential backoff is everywhere in your stack, but it doesn’t end up in your application layer until you have enough traffic that you actually have to manage throughout.