6 ms·
A Detailed Five Step Twitter Scaling Plan
- axod 18y agoThis has been solved many times, many years ago. It's not a hard problem. A one step scaling plan would be "write it properly this time".
- chaostheory 18y agomany things are easier said than done.
- axod 18y agoOf course. But there are solid examples from web1.0 and before that did more than twitter, and scaled to more users. Twitter is pretty simple in terms of functionality.
- chaostheory 18y agoclimbing Mt Everest/K2 has been done many times already, but it's still hard; and not something taken lightly I just don't like gloating on someone else's mistakes and failures (essentially kicking someone who's already on the ground), especially when I'll probably have my own set of spectacular f-ups to deal with related to my own startup
- davidu 18y agoSo wrong it's hard to know where to start...
- Retric 18y agoHow about: 1) It's not just a DB problem.
- antidaily 18y agoexactly.
- hank777 18y agoUh... yes it is. All scaling problems are database problems. ALL of them. They can be solved by different strategies such as caching, distributed design, etc. But at the end of the day the problem is always that you cant get to the data fast enough. That is a database problem.
- nostrademons 18y agoIncluding systems that don't use databases, or are you defining a database loosely as "anything that stores data"?
- sanswork 18y agoUnless it's a problem with a bad algorithm slowing down the processing of the data. So not all scaling problems are database problems.
- edw519 18y ago"All scaling problems are database problems." This is an argument in semantics. You suggest storing all users physical addresses in memory across multiple machines. You also advocate "shard splitting" a la B-tree indexes which split when the node's population reaches 2n. OK, so when a split occurs, how do you maintain that massive table in memory. Do you lock it? Do you examine your existing queue of requests (which may soon point to the wrong node)? How do you handle new queue requests during "shard split maintenance"? Calling these "database problems" is waving your hands. Why not just call them "computer problems"?
- hank777 18y agoIts fascinating funny how you are so smart you can't even deign to share your wisdom on this. If I am wrong, tell me where. Given that I have spent the last several years studying/developing distributed databases if I just don't know what I am doing I'd certainly like someone as smart as you to tell me.
- cstejerean 18y ago"Given that I have spent the last several years studying/developing distributed databases..." That doesn't make your opinion of what's wrong with Twitter any more valid than the crap TC keeps printing unless you happen to also have lots of internal details of the Twitter architecture.
- hank777 18y agotrue, but it makes me curious what brilliance someone bring to the table when they have nothing specific to say other than you are wrong. Also given that much of what I am talking about is running code, I at least know that it solves a major class of problems. If it doesnt apply to twitter it should be pretty easy to explain why.
- nostrademons 18y agoI don't use Twitter, but as I understand it, it's a massively many-to-many messaging service, where it's not unusual to follow hundreds of people's tweets and have thousands of people follow yours. Can you imagine the performance if every message required hundreds of disk writes as the "received" copies get spread across a hundred shards?
- hank777 18y agoActually, this is the beauty of the design: All of the writes happen in parallel. So there if you write to 2000 recipients, but all of the writes happen in parallel you have no bottleneck. And any time a given machine's load gets too heavy, you do a shard split. You really dont care how much writing you do as long as on every machine you write to is not being hit too hard. So if a given server has lets say 10,000 users on it. And each one of those users receives 10 updates an hour. Then it would mean handling, lets say 100,000 writes in an hour, or 27 writes a second. If this was the threshold for this machine (which would be low) you then split the shard so that it then is only handling 13 or 14 writes per second.
- andrewparker 18y agoThis is post is little more than link bait (and I guess I'm falling for it by commenting).
- hank777 18y agoWith all due respect andrew why dont you share your technical analysis of what exactly I am saying that is wrong here. Btw, have you ever actually worked on any database scaling problems? Just checking.
- davidu 18y agoI have. I have managed systems that handle millions of clicks a day in databases and billions of dynamic (PHP) pageviews on a daily basis. I am currently managing a system that serves over 100,000 DNS queries per second, each of which is dynamically evaluated as to how we respond to it. And each of which is logged, amounting to over 7,000,000,000 log lines a day, all which get processed, parsed and in many cases handed over to a database store. Yep, we're dealing with BILLIONS of rows of MySQL per day.
- hank777 18y agoNevertheless you believe that there is no way to handle a distributed in-memory lookup system. It just goes to show you cant know everything.
- davidu 18y agoWhat do you mean? I've used distributed hash tables (in memory or otherwise) for years...
- Xichekolas 18y agoDo you have anything online about your architecture? I think it'd be interesting to see how those 100k queries per second get served. Hey thanks for OpenDNS by the way, I'm a long time user, and I love it.
- bluelu 18y agoHis approach isn't so good, as it creates a copy for the same message sent to multiple users. Better safe the messages in a central store by id and only write the ids to the different pages. http://news.ycombinator.com/item?id=188080 http://news.ycombinator.com/item?id=188080
- hank777 18y agoI considered this, but given that the messages are small and the primary issue is scaling not storage this, it does not seem to me is a good solution. The entirety of everything ever written on twitter is probably not big enough to worry about storage size. The issue is performance. By the way, this architecture is essentially how email works. All your inbound messages are stored in one place separate from outbound messages from the sender. This allows for infinite horizontal scaling.
- mtw 18y agoimho, it's rails. rails (and any other ruby framework) is not suited for a real-time communication platform like twitter. also you forgot about messaging queues.
- KirinDave 18y agoWhy would you say that? It's not like the Ruby code is handling that many messages. Even Rails sites don't use ruby for their concurrency and messaging. They push down to a database or to another concurrency level like ActiveMQ. Above them lives another piece of code or hardware to do load balancing. This isn't even about Ruby, it's about the Rails architecture.
- xirium 18y agoThe suggested plan seems reasonable. Yes, if you denormalise users inbound and outbound messages then you store the data twice. Yes, shards give almost limitless scalability. However, could it be that during peak load, the number of messages to be written plus indexes to be maintained exceeds the total available disk seek operations?
- nostrademons 18y agoYou don't store it twice - you store it hundreds of times, once for each recipient. Twitter isn't person-to-person messaging, it's broadcast.
- wmf 18y agoSo shard more, or get more disks, or get flash or whatever. (Yes, I don't know what I'm talking about.)
- xirium 18y ago> So shard more, or get more disks, or get flash Your intuition is excellent. These are all valid techniques to alleviate disk seek limitations. They are likely to be successful. Unfortunately, they're expensive options. You can partially overcome disk seek limitations by performing database batch inserts. So, rather than inserting messages indiviually and updating indexes for each message, you insert 100 messages and perform less disk seeks updating the indexes for each batch. Unfortunately, it can be hard to reliably merge data into suitable batches.
- wmf 18y agoHank's proposed solution is expensive; we should just admit that up front. But is it more expensive than being down and fighting fires all the time?
- xirium 18y agoThe economics of downtime is a very important consideration. I read that three minutes of downtime reduces the value of a website by 1%. However, that may be a figure bandied by vendors of high-availability equipment. It is also generally acknowledged that high-availability is achieved through strict processes and sheer redundancy of hardware. Therefore, it gets exponentially more expensive to minimise downtime. So, you've got two non-linear curves, cost and benefit. If you've got an exceptional team, a great architecture and good finances then getting 99.999% uptime should be easy. Of course, this assumes that you don't have unforeseen circumstances, glaring omissions in redundancy - or exponential growth and utilisation without a revenue stream. Even with generous financing, exponential growth can be deadly. A small but sustained increment in load can be enough to cause a backlog in requests that never recovers. If you're growing fast then it can occur anywhere in your system at any time. I've been a DBA for a renderfarm with thousands of cores and this situation is stressing. We've also had this situation while working on search. Thankfully, it occurred on a smaller scale and most thankfully it occurred before launch. We had two database servers, two app servers, no redundancy, on unreliable hardware. We didn't make much headway with efficiency improvements because long tests were likely to encounter hardware failure. It was demoralising. This slowed the development and testing of a database failover feature in the database wrapper. Now this has been implemented, progress is much better. A test that previously took three hours now takes five minutes and we've had much more time to migrate to reliable hardware and introduce redundancy. We've also got a setup which can be used to test resilience to real modes of failure.
- livecrunch 18y agoI think the post is well structured and informative for Twitter Team.
- menloparkbum 18y agoPaying attention to half-baked advice like this is half the reason Twitter is fucked.
- edw519 18y agoThese design techniques could very well make a nice impact on performance. Then again, they might not. Why? They are proposals to solve a problem that has not yet been defined. Fact is, you don't know what the problem is. So proposing a database solution to what could just as easily be a memory management or throughput problem is premature.
- hank777 18y agoThis is true. Though I can say that if one were building twitter and you didnt either have a design like the one I suggested, or a design with some kind of distributed caching then you are going to have exactly the kinds of scaling problems they have had. In other words if you use tradition techniques for something like twitter you are screwed. However you are absolutely right that (as I said in my piece) they may already, for example have thought of all of this and either implemented or dismissed it because they have a better design, and I have no way of knowing.
- menloparkbum 18y agoThis isn't true. You don't need a distributed sharded setup in order to scale. (You probably do if you are limiting yourself to MySQL) I built financial applications way back in the day that had a lot more reads and writes than twitter, and we just used ONE gigantic, super expensive database server.
- sutro 18y agoHank, good article, and good defense of your article here on HN. Regarding the knee-jerk negative responses you've received here, I can only say that you are a more patient man than I. Best of luck with your current project.