16 ms·
The Architecture Twitter Uses to Deal with 150M Active Users
- brown9-2 13y agoHere is a similar talk that Jeremy Cloud gave at QCon NY a few weeks ago: http://www.infoq.com/presentations/twitter-soa http://www.infoq.com/presentations/twitter-soa Jeremy Cloud discusses SOA at Twitter, approaches taken for maintaining high levels of concurrency, and briefly touches on some functional design patterns used to manage code complexity.
- mistertrotsky 13y agoI would absolutely trust someone named Jeremy Cloud on this subject.
- bpicolo 13y agoThey have a lot of RAM. Dang.
- buro9 13y agoNot as much as I would have thought. 2TB can be one server: http://www.supermicro.co.uk/products/system/5U/5086/SYS-5086B-TRF.cfm http://www.supermicro.co.uk/products/system/5U/5086/SYS-5086... An expensive server for sure... but it's just one server.
- vidarh 13y agoIn case people have a hard time finding prices for this, my vendor gives ~57k GBP ($85k) as the lower end price for configurations with that SuperMicro cabinet, 64 cores (8x 8-core CPU's) and 2TB of RAM. Dropping to 1TB lets you pick from a lot of cheaper cabinets and so the price drops by quite a bit more than half for the basic configurations.
- papsosouid 13y agoEven the low end of single servers goes up to a couple TB of RAM. "Enterprise" hardware has been there for ages.
- binarycrusader 13y agoThat's really not a lot, especially when you consider that it's distributed among a cluster. A single SPARC M5-32 can have up to 32TB of memory: http://www.oracle.com/us/products/servers-storage/servers/sparc/oracle-sparc/m5-32/overview/index.html http://www.oracle.com/us/products/servers-storage/servers/sp...
- druiid 13y agoAs others have said they don't actually have a lot of RAM dedicated to Redis. You can put 1.5TB of memory in many 'inexpensive' Dell servers (single machines, not clusters). So basically a cabinet of machines you could have over 30TB of memory available to you. Basically some of the design choices Twitter has made seem tailored to their (previous? Not sure if they have their own hardware now) choices to run on 'cloud' services in the past. With good hardware and a bit of a budget you can easily scale to crazy numbers of processor cores and memory. That isn't to say the software side of the solution is going to be any easier to solve.
- NelsonMinar 13y agoThis article cites its source as a talk by Twitter VP Raffi Krikorian. 38 minutes of video, audio, and slides are at http://www.infoq.com/presentations/Twitter-Timeline-Scalability http://www.infoq.com/presentations/Twitter-Timeline-Scalabil...
- webwanderings 13y ago> Twitter knows a lot about you from who you follow and what links you click on. No kidding. But we don't care as we live in the glass house of a celebrity culture. PS: downvote the quoted text if you must. My point is not obvious.
- mjolk 13y agoIf you expect others to misunderstand you, perhaps try to better represent your thoughts?
- StavrosK 13y agoYou realize it's public data, right? When I tweet something, I don't expect it to be private. HN knows a lot about me too.
- webwanderings 13y agoYup, I mentioned glass houses.
- Myrmornis 13y agoThe meaning of your metaphor certainly seems obvious; explicit even. Are you referring to a layer of more profound non-obviousness?
- papsosouid 13y agoI really question the current trend of creating big, complex, fragile architectures to "be able to scale". These numbers are a great example of why, the entire thing could run on a single server, in a very straight forward setup. When you are creating a cluster for scalability, and it has less CPU, RAM and IO than a single server, what are you gaining? They are only doing 6k writes a second for crying out loud.
- thezilch 13y agoThey may average 6K w/s, but history has shown higher peaks, and those peaks often include tweets from those with large 30e6+ followers -- large events naturally have celebrities chiming in. Having more CPU [cores], RAM, and IO is not going to solve for the need to shard your Redis, which is bound by a single core. For the fanout, that is anywhere from 90 to 90e3 cores at 2e6 Redis IOPs -- 1-6e3 Lady Gagas per second. Before pipelining, which you'd probably consider complex or fragile. All you've done is moved some networking parts around. Your code and topology hasn't really changed.
- papsosouid 13y ago>Having more CPU [cores], RAM, and IO is not going to solve for the need to shard your Redis, which is bound by a single core But they chose redis in the first place as part of their messy, crummy pile of barely working junkware solution. You wouldn't use redis in the first place if you were building a solution that scales up.
- rozap 13y agoYou're funny. I like you.
- thezilch 13y agoSo, now we also have to assume per-core software and support licenses to accompany our purchase of several $10M racks? What platform are we talking about now? Presumably I have to also purchase the IDE and toolchain licenses, per developer, too? Twitter may have and will still make missteps, but you're being egregiously vague and unhelpful. Your comments are as vaporous as you make Twitter's architecture to be.
- cwt137 13y agoThis near hour long video is a deep look into Twitter's backend. Specially into the Firehose feature, Flock, etc. They go into detail on how they use Redis and even show one of the actual data structures they store in Redis. A must see video for anyone into high scalability. http://www.infoq.com/presentations/Real-Time-Delivery-Twitter http://www.infoq.com/presentations/Real-Time-Delivery-Twitte...
- aroman 13y ago> Twitter no longer wants to be a web app. Twitter wants to be a set of APIs that power mobile clients worldwide, acting as one of the largest real-time event busses on the planet. Wait, then why are they actively destroying their third-party app ecosystem...?
- grbalaffa 13y agoYeah, somebody hasn't gotten the memo it seems.
- blazingfrog2 13y agoThe piece did not say "third party". I'm assuming these clients are the official Twitter mobile apps.
- mey 13y agoThe two are not the same. It sounds like they are becoming API driven to support multiple interfaces and to put clear boundaries between systems.
- zanny 13y agoSounds like html. Or qt. Not 140 characters.
- Yhippa 13y agoProbably their mobile clients. After I read the article perhaps Twitter felt that having control of the clients reduces variability in stress to their system. Like bad actors accessing the firehose suboptimally.
- tschellenbach 13y agoTheir setup is very similar to what we use at Fashiolista. (though of course we only have millions and not hundreds of millions of users). We've open sourced our approach and you can see an early example here: https://github.com/tschellenbach/Feedly/ https://github.com/tschellenbach/Feedly/
- mikemoka 13y agoI am playing the armchair architect and my question will be probably wrong in infinite ways,but I might learn something, what is the reason why the service has to write a tweet on two million timelines, wouldn't it be cheaper if they let the client build the page on its own via restful apis?
- rythie 13y agoRequesting the timelines of each of the people you follow is slow process for the client, meaning that the client has to make 100s or even 1000s or requests. Also, much of the data is thrown away because it's replies to people you don't follow or too old. It's hard for clients to reconstruct the timeline that way. Also, it would vastly increase the number of HTTP requests and data Twitter has to ship out. Also, it's basically impossible to make a system like that real time because you have to check 1000 feeds to see if anything is new. I used to write a multi-network client, basically the combined home timeline request is the only feasible method for a client to use.
- deleted 13y ago[deleted]
- brown9-2 13y agoThey are making a tradeoff between cost of reading your timeline and cost of a write at tweet-time to optimize the former.
- logic 13y agoIn addition to what others have mentioned, consider passive endpoints, such as SMS and push notifications.
- beagle3 13y agoBut they don't all have to fit within the same framework - indeed, don't fit into the same framework. e.g., if you can't send the SMS or push right now, you have to do a retry 5 minutes later. If you can't update a followers queue (if that's what you do), you have to display a fail whale....
- 13y ago
- davidw 13y ago> Your home timeline sits in a Redis cluster and has a maximum of 800 entries. Wow, that's pretty cool. Congrats to antirez - it must be a nice feeling knowing that your software powers such a big system!
- ampersandy 13y agoNo one else seems to have noticed that these details are out of date. Twitter has publicly stated that they currently have "well over 200M active users". The stats are also misleading in that -- I'm pretty sure -- 300k reads and 6k writes per second are only referencing tweets. Flock, Twitter's graph database, handles more than 20k writes and 100k reads per second on its own (peak numbers available from the two year old Readme on Github). https://blog.twitter.com/2013/celebrating-twitter7 https://blog.twitter.com/2013/celebrating-twitter7
- smandou 13y agoWell... It seems that hisghscalability doesn't scale... Error from squarespace on that page.
- kushti 13y agoScala is awesome language to implement scalable things, like Twitter services
- swah 13y agoWhy?
- pesenti 13y agoDoes anybody know how that compares to Facebook? I believe they do not use write fanout but instead rely on the search federation model (the model claimed not to have worked for Twitter).
- sheldoan 13y agoFacebook uses a fan-out-on-read approach http://www.quora.com/Facebook-Engineering/Did-Facebook-develop-a-custom-in-memory-database-to-manage-its-News-Feeds http://www.quora.com/Facebook-Engineering/Did-Facebook-devel...
- apaprocki 13y agoSince tweets are very much like financial ticks (fixed size, tiny), we can put the numbers in perspective. Let's compare Twitter vs say, OPRA (North American options), a single "exchange" feed. Twitter: 300K QPS, firehose 22 MB/sec, 400 million tweets/day OPRA: 12.73M MPS, firehose 3.36 GB/sec, 26.4 billion messages/day http://opradata.com/specs/2012_2013_Traffic_Projections.pdf http://opradata.com/specs/2012_2013_Traffic_Projections.pdf edit: Also worth noting, the diff between OPRA 1/1/13 & 7/1/13 is ~256 MB/sec, 2.0 billion messages/day. So in just 6 months the existing firehose increased roughly 10x Twitter's entire output and roughly 5x'd the number of messages/day.
- squarecog 13y agoIt's not the data coming in that makes things hard. It's the fanout of messages into their subscribers' timelines.
- ak39 13y agoI've read before that Twitter uses NoSQL (or non-relational databases) and that each tweet requires a copy to be pushed to each follower every time someone tweets (same for updates or deletes). I am of a very much traditional RDBMS mindset though I am willing to bend a bit :) ... but this idea of fanning out tweets (copying tweets) to each and every follower just sounds grossly inefficient. The amount of writes must be exponential. The way I see it, here are the WRITES they'll ever need: 1:User -> n:Users (Followers) (Temporal links with effective dates for follow history) 1:User -> n:Tweets (Temporal with tweet datetime) - this includes Retweets. The rest are just READS (pulls instead of fanning out the same data). What am I missing here folks? :-) Educate me please.
- jacques_chester 13y agoFiguratively, Twitter have switched from doing a design rooted in Databases 101 (OLTP) to a design more rooted in Databases 102 (OLAP). That is, they moved processing from query time to write time. And that's a perfectly legitimate strategy; it's the basis of data warehousing. OLTP is about write-time speed. It's great for stuff like credit card transactions, where you only really care about the tally at the end of the banking day. The dominant mode of operation is writing. OLAP is about read-time speed. You do a bunch of upfront processing to turn your data into something that can be queried quickly and flexibly. One thing that's problematic about the teaching of database technology is that the read/write balance isn't explicitly taught. Your head is filled with chapter and verse of relational algebra, which is essential for good OLTP design. But the problems of querying large datasets is usually left for a different course, if it's treated at all.
- joshuaellinger 13y agoMaybe at a high level, but OLAP involves precomputing aggregations and bit indexes. It's a pretty different beast. OLAP is very rarely under real-time constraints and, when it is, it tends to push the heavy lifting out to OLTP.
- jacques_chester 13y agoOh, I'm handwaving, not talking about the underlying details of star schemata, clever column representations and whatnot. But I think the analogy is still correct. Every such system has two functional requirements: 1. Store data. 2. Query data. And every system has the same non-functional requirement: 1. Storage (write) should be as fast as possible. 2. Queries (read) should be as fast as possible. However, per an observation I made a while back, complexity in the problem domain is a conserved value. Insofar as your data requires processing to be useful, that complexity cannot be made to go away. You can only decide where the complexity cost is paid. You can pay it at write time and amortise that across reads. You can pay it at read time and excuse writes. Or you can pay it in the middle with some sort of ETL pipeline or processing queue. But you must always pay it. The experiences of data warehousing made that bitterly clear. So really, the job of a software architect is to take the business requirements as a non-functional requirement (an -ility) and then pick the architecture that fits that NFR. That includes dropping other nice-to-have non-functionals. Twitter's non-functional requirement is that they want end-to-end latency to be 5 seconds or less, under conditions of very low write:read ratio. This suggests paying the complexity cost up front and amortising it over reads. And that's what they've done.
- jebblue 13y ago>> it can take up to 5 minutes for a tweet to flow from Lady Gaga’s fingers to her 31 million followers Why not break the load up among a farm of servers? 5 minutes to deliver a single message? It's too bad multicast can't be made to work for this use case. At least analyze to see if there's a pattern of geographic concentration of her followers and optimize for where their datacenters are. Use peer to peer, let the clients help distribute the messages.
- joshuaellinger 13y agoThe surprise for me is that the core component is Redis. My first guess would have been custom C code. Yeah, you have to do everything yourself. Yeah, it would be hard to write. But you'd control every little bit of it. Obviously, I must not fully understand the problem and what Redis buys. Sam Puralla (if you are reading) -- do you know why didn't Twitter go with a full custom system at its heart? Josh
- nikcub 13y agoThink of Redis as a nicer interface to data structures that otherwise would have been written in custom C code. If you read the redis manifesto, it describes itself in point 1 as a DSL for abstract data types: http://oldblog.antirez.com/post/redis-manifesto.html http://oldblog.antirez.com/post/redis-manifesto.html
- bitbckt 13y agoI'm probably a better person to answer this than Sam - I'm a former lead on this project - so I'll take a swing: We chose Redis because it gave us the specific, incremental improvements over our existing memcached-based system that we required, without requiring us to write (yet another) component. There was enough to do, and this choice has turned out to be good enough, I think. As the project progressed though, we treated Redis much like we did own it. We altered the wire protocol, changed the eviction strategy, and reduced protocol parsing overhead, for example. Much of that work has long since made it upstream. [edit] grammar
- joshuaellinger 13y agoSounds like good engineering choices. I think of Twitter as having unlimited resources but, of course, that can't be true. Two follow-on questions: 1. Did your changes make it back into open source or were they only relevant to Twitter? When you say upstream to you mean on Redis or earlier in the Twitter pipeline? 2. How much is Redis on the critical path? Is it 90% of the processing work in the large fanout cases?
- 13y ago
- akuchlous 13y agoi wrote a synchronous twitter push model. single machine, 4GB RAM x86_64 GIST: push 48 M messages in less than 5 seconds. some data from experiment: Total Super Users : 100 Each Super User Follower Count: 1234567 Total Connections Made 123456700 Time To Make NetWork : 32 secs Making New 40 messages Time To Push 49,382,720 messages : 5 secs Making New 40 messages Time To Push 49382720 messages : 5 secs Making New 40 messages Time To Push 49382720 messages : 5 secs Making New 40 messages Time To Push 49382720 messages : 6 secs -------------------------- code at : https://github.com/akuchlous/Twitter-TimeLine-Push-Experiment https://github.com/akuchlous/Twitter-TimeLine-Push-Experimen... -------------------------- with distributed, + twitter constraints (filter messages, links .. etc), 5 seconds seems to be decent.