5 ms·
Rearchitecting Core Services at X
- AI_beffr 2y agocool
- chx 2y ago[flagged]
- jiggawatts 2y agoWhy do I get the impression that all of these FAANG-like companies spend billions of dollars of engineering reinventing partitioned databases, but with HTTP as the query protocol? They talk about “granular field-level access” — that’s just a SELECT statement. They talk about separating the read only workloads — that’s read scale out replicas. They talk about Arrow and similar technologies that sound an awful lot like SQL queries.
- joshuamorton 2y agoBecause you have a hammer and so everything sounds like a nail. When you discuss a problem at a high enough level of abstraction, then yes everything looks like a database, and at a high enough level of traffic all databases need to be partitioned/replicated. The much more interesting question would be "why doesn't the thing I'm familiar with solve this problem"?
- CSMastermind 2y agoMy hot take is that 90% of software engineering is just transforming data from one shape to another and there's only so many way to do that. It doesn't matter if you're syncing between a database and server, between a presentation layer and an object model, or across two devices over a network. Fundamentally you're dealing with an impedance mismatch that only has so many types of solutions.
- eightysixfour 2y agoI don’t think that’s a hot take at all, I’m pretty sure that was lesson one in the data structures course at my school.
- amw-zero 2y agoThat’s not a hot take. Literally thousands of people have written and said this exact same thing, for years.
- vinkelhake 2y agoIt could be that they're plowing billions of dollars into engineering and missing some obvious solution. I think it's more likely that once you get down the details (which are going to much more intricate than a post like this can convey), then it may turn out to actually be a hard problem that requires a bunch of engineering. For a relevant example of the "how hard could it be"-mindset: See George Hotz's short stint at Twitter.
- mvdtnz 2y ago> See George Hotz's short stint at Twitter Do you have any more information about this, or could provide a link?
- mplewis 2y agoInfamous iPhone hacker geohotz joined Twitter for a three-month internship where he planned to fix search. He quit after one month having accomplished nearly nothing.
- jazzyjackson 2y agoYeah it may be a case of tucking tail, but that says nothing about whether it was a difficult engineering problem, could just as well be an impossible political situation to navigate (whose feelings are you going to hurt by deleting some microservice)
- wmf 2y agoDoing 60 million queries per second on a partitioned database isn't going to be easy either, and when you have problems you'll have to dig into the code of a database you didn't develop.
- Demiurge 2y agoYeah, is there a PostgreSQL deployment that does 60m qps?
- jiggawatts 2y agoDoes Twitter actually need 60 qps, or is this just a consequence of not having JOIN operators?
- darth_avocado 2y agoThere's a lot of finer details that influence how you design system. Yeah reading data is just SELECT, but there's too many things to consider. - Volume of reads: how many people are reading tweets? - Volume of tweets: how many unique tweets are being served to people? - Data skew: Elon's tweets see hundreds of millions of reads, vs some tweets only see like 10. - Size of data: A tweet looks like only a few characters of data, but every tweet has probably hundreds of columns of metadata. These are just a few considerations. Plus, scaling out replicas isn't always the feasible after a point. Otherwise, why not just spin up 1 server per tweet and keep things simple.
- preommr 2y agoBecause they make billions, and they have to spend it on software development. That these software devs then go out and build slightly different offshoots because of business requirements and technical requirements is, in many ways, incidental. Devs at smaller companies end up writing poor-man's equivalent of various techs all the time, or write interesting things in their spare time. It just doesn't get the same attention because they're not as big.
- numpad0 2y agoIt's how they get paid. Replacing bloats with extremely simplistic equivalents like stock MySQL instance isn't going to convert well to paychecks. It also means your output will be erratic(because it'll be opportunistic), architecture design will be less resilient, and you will be a brilliant jerk that are hard to anticipate. Elon Musk was supposed to be an uber-genious charisma who've overcome that problem. I suppose he no longer is.
- jiggawatts 2y agoMusk fired thousands of Twitter employees and the system is still running. It’s even getting significant core feature upgrades! Clearly, most of those engineers were superfluous. I’ve thought Twitter was over-engineered years before that event and feel a tiny bit vindicated in holding that opinion against the HN gestalt.
- kgeist 2y agoReminds of this: https://news.ycombinator.com/item?id=17135841 https://news.ycombinator.com/item?id=17135841
- marcinzm 2y agoThat proves the opposite point. At small scale FAANG solutions are non-sensical. At 100 petabytes good luck running your code on a laptop.
- kgeist 2y agoOn the other hand, we often create over-engineered solutions because taking the simpler route would require us to rewrite existing cruft, which is often just more expensive.
- deleted 2y ago[deleted]
- michaelt 2y agoAh, but what about the tracking for ad targeting? That tweet viewed by a million people isn't just a million tweet database reads, it's also a million tracking database writes.
- Wololooo 2y agoActually, that's not a problem anymore...
- cryptonector 2y agoI interviewed at a ride-share company (one that was not then working on self-driving cars mind you) once some 5 years ago or so and was shocked to find out that they had 1,000 software engineers. What did they need so many engineers for?! But then all their queries were hand-written to some key/value store w/o a query engine -- that has to explode your engineering needs a bit, though maybe still not to 1,000.
- whalesalad 2y agoDistributed systems is the answer. Data isn’t in one sql database.
- deleted 2y ago[deleted]
- blipvert 2y agoIs this why I see the “Something went wrong - Retry” dialogue about fifty times a day now?
- kyrra 2y agoMy take on this is that the client front end UI will load basic pages, but when it tries to load the feed there's somewhat Network error that causes the feed not to load.
- aeturnum 2y agoThere is a curious lack of data showing the impacts of these changes (though perhaps that's forthcoming as they roll it out at scale).
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- chx 2y agoIt should be obvious no one good stayed there. Either they were fired or left disgusted. You need to be really desperate to stick with Elmo.
- jazzyjackson 2y ago[flagged]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- eska 2y agoPretty light on technical specifics. What I find odd about all of this is that while they (finally) split reads from writes, there’s no distinction between hot and cold data. I assume 80% of all tweets are only read 20% of the time, if that. This would heavily influence my architecture regarding caching for example. RPC is also mentioned, leading to “pull” architecture, when I would assume most of the work to be a “push” pipeline, and work to be embarrassingly parallel. So there’s no real software complexity from interconnected data, just a large amount of it but it’s easily sharded and solved by good hardware? So in summary it doesn’t actually sound interesting from an algorithm perspective. What services are they talking about for RPC, analytics? Do those really need to be so dynamic?