4 ms·
“[Twitter] is doing 1000 poorly batched RPCs just to render a home timeline”
- minimaxir 4y ago"Our app has performance issues even though we fired all the engineers who could fix it" is definitely a leadership take. EDIT: > As the former tech lead for timelines infrastructure at Twitter, I can confidently say this man has no idea wtf he's talking about https://twitter.com/bgleib/status/1591875679377031169 https://twitter.com/bgleib/status/1591875679377031169
- willcipriano 4y agoIf they were the engineers that could fix it, why haven't they?
- csande17 4y agoI suspect most of Twitter's performance issues are of the too-many-cooks variety, where services are poorly coupled or have a lot of dependencies because a million different teams are working on each one. (Heavy layoffs are a good step towards fixing that, along with a technical redesign.)
- nitwit005 4y agoFeels like he latched onto some number in a power point presentation on performance, without totally understanding it.
- ilaksh 4y agoI suspect this is not quite what that sounds like (possibly slightly misunderstood), really about the complexity of the task and the amount of data points, and human nature than it is some gross engineering mistake. The engineers brought in are in the position where they are expected to trash the existing teams code. And then in the context of a slow page load, they might look at it and see up to 1000 of something in RPC and tell Musk and it comes out in a slightly mixed up way when he tries to repeat it. The reason the number of calls gets high is (I guess) not from 1000 individual different types of requests, but from pulling many pages or records each of several things. Anyway it is easy to trash other people's code especially if that's what your boss wants you to do to justify bringing you aboard. I have been the victim of it and also on one or two occasions done the same thing myself to some degree. Another aspect is that no engineer really wants to maintain someone else's code if they don't have to. They are going to have their own viewpoint on architecture etc. and coming onboard is a prime opportunity to start a rewrite. So slamming the existing architecture is just an easy way to try to get that authorized. I think it's a bit of an integrity test coming in and talking about someone else's code base. But people are unfortunately creatures of circumstance, and these things can bring out the worst in some as far as fairness goes. But again, even for a fair person it might be easy in that position to scoff at a codebase. Most people are not particularly high integrity though so will happily take a dump when it comes down to it.
- pyinstallwoes 4y agoPulling many records leads to n*n optimization problems. Which leads to run-away rpcs.
- karp773 4y agoNot to defend Musk but, as a matter of fact, he has written his fair share of code so he probably knows what he's talking about in this case. I just refreshed twitter's home page and for me it made 111 http requests (per Chrome's dev tools). That includes static resources like images and style sheets. So only ~10% are made from the browser and the other 90% must be due to microservice architecture on the backend, I suppose?
- paulryanrogers 4y agoWhere can I read more about Musk's coding prowess? Because the little I've gleaned is that he insisted on porting PayPal to Microsoft stack, got pushed out, and picked up enough about rocketry to impress Carmack in casual conversation.
- karp773 4y agohttps://en.wikipedia.org/wiki/Zip2 https://en.wikipedia.org/wiki/Zip2 "Elon Musk was appointed the Chief Technology Officer..." He created a web based company as early as 1995. In a three person startup one cannot become a CTO without writing code. So my bet is he understands something about http vs local calls.
- Spivak 4y agoDoes this account for pipelining and multiplexing because that’s like the whole reason those exist so that every client doesn’t have to think about batching and the browser can do it for you and present a fiction that they aren’t. Because Twitter definitely supports HTTP 1.1 and HTTP/2. We’re currently in the process of ripping out a lot of client side opportunistic model hydration in favor of just making multiple requests because it’s faster.
- deleted 4y ago[deleted]
- lost_tourist 4y agoI'm honestly curious, what is opportunistic model hydration? Short explanation would be great, I don't need all the gory details.
- bagels 4y agoProbably: fetching related items or fields that the client didn't ask for... yet
- prirun 4y agoI'm guessing Twitter is a highly distributed system, so of course it's doing a bunch of poorly batched RPCs. That's more or less the definition of a highly distributed system, where all the data can't be stored in 1 place.
- dzhiurgis 4y agoIf it’s all same server then it can be batched up. Also doesn’t the new fangled http/2 protocols and the like solve this, at least partially?
- nitwit005 4y agoIt can only be batched if the source and destination are the same. Different parts of the page likely hit different servers, and query different backends.
- deleted 4y ago[deleted]
- kuberlog 4y agoNetflix has 1000 Uber has 4000 Why can't twitter have 1000?