5 ms·

You're an order of magnitude past their peak load - New Years Eve they set a new record of almost 7000/second. http://thenextweb.com/twitter/2011/01/06/new-yea

You're an order of magnitude past their peak load - New Years Eve they set a new record of almost 7000/second. http://thenextweb.com/twitter/2011/01/06/new-years-eve-set-a-new-twitter-record-6939-tweets-per-second-sent/ http://thenextweb.com/twitter/2011/01/06/new-years-eve-set-a... I don't believe Twitter is trivial, but I think they're perceived as more complex than they are - WordPress + Reddit + Heroku only have like 10% that number combined. Apples to oranges, but those 3 companies would be doing more everythings per second than Twitter when you combine them.

zohebv 16y ago
Handling updates to 7000 user accounts per second isn't what makes it hard for twitter. It is the propagation of these updates to a gazillion followers that taxes the system. Their primary overhead is the messaging update to followers. Ashton Kutcher has 6 million followers. Sending instantaneous notifications to all those million twitter clients is what makes it a killer. TLDR; It is not the vertices in the twitter graph but the number of edges in it that is unprecedented.
- benologist 16y ago
  Unprecedented, except is it really unprecedented or are we just looking at it from the sexy social startup whatever must say graph and also disrupt angle, where everything is new and super hard and nobody has problems that have ever been solved before. RSS strikes me as an almost identical process other than the time subscribers wait to check for new content, and there are feeds with millions of subscribers. So now the problem is reduced to doing it in a timely fashion. This is why I wonder if they're really that immensely complicated - isn't Google doing much the same thing, and possibly even at a bigger scale at one point, with their reader and FeedBurner?
  - jimbokun 16y ago
    "isn't Google doing much the same thing, and possibly even at a bigger scale at one point, with their reader and FeedBurner?" Another company with a ton of employees. Probably not that many on the FeedBurner team itself, but a lot of people at Google working on the general problem of keeping things unreasonably responsive at massive scale.
- Splines 16y ago
  This is exactly the reason why every IM network has a limit to the number of contacts that you can have. I recall reading several years ago that this was an incredibly hard problem for IM providers to solve, and AFAIK none of them have allow you to have an infinite number of contacts.
nostrademons 16y ago
As has been pointed out elsewhere on the thread, complexity doesn't scale linearly. It's far easier to write ten sites that each gets 30 qps than it is to write one site that gets 300 qps, and it's easier to write ten sites that get 300 qps than one site that gets 3000 qps. Twitter's fundamental problem also is a harder one to scale than something like Heroku or Wordpress. For those hosted sites, you can shard easily by host, so that each of the 100,000 Heroku-hosted sites can get its own EC2 instance(s) and behave pretty much independently. You can't do that when the point of your site is that any action might instantly be broadcast to thousands of followers. High-fanout writes are not an easy problem to solve.
- beagle3 16y ago
  But if you design it well enough, it does. Here's a copy-pasta from a reddit comment of mine from a few months ago: http://www.reddit.com/r/programming/comments/b2u6t/twitter_opensource_a_list_of_all_open_source/c0kq4le http://www.reddit.com/r/programming/comments/b2u6t/twitter_o... ------------------ Color me unimpressed. At some point, I was collecting 40GB/day of financial data (and that's after bzip2ing them .. probably 200GB/day before); This was done on hardware costing $30K (which was two equivalent machines with 4GB, each having 20*1TB in a raid configuration -- this was a hot-backup configuration) and the operation run (coded, supervised, administered) by 2 people. I'm extrapolating from your numbers: Let's say you have 70GB over 14 days = 5GB/day; Let's assume Twitter has 100GB/day of text twits (which, incidentally, means ~1 billion tweets per day which I highly doubt, as they took few years to get to the 1 billion mark, and last I heard they were at less than 100 million twits/day) Then, at this day and age (numbers selected for 2 years ago, when they had their last infrastructure revision), what you do is buy 20 servers with 8GB of memory each (for, say $5K each), plus a little redundancy, and store all the latest twits in memory, and the most popular user's older twits as well; everything else on disk. Throw in cheap web front-ends that don't even need a local disk, load balancing, and a gigabit ethernet backplane. You're still under $200K in equipment. Yes, the code is not going to be trivial, but for $100K and 3 months you can get a stunt programmer (I know a few who can do it and won't charge as much). A run-of-the-mill RDBMS is the wrong tool for this job; Basically, run of the mill anything; but that does not make it incredibly hard. I think $300K for hardware and software, you can get a Twitter clone that performs as well. Twitter is successful, but that's not thanks to good engineering.
tzs 16y ago
Yeah, the number I had was for total API calls (6 billion a day), not just updates. http://techcrunch.com/2010/09/17/twitter-seeing-6-billion-api-calls-per-day-70k-per-second/ http://techcrunch.com/2010/09/17/twitter-seeing-6-billion-ap...

zohebv 16y ago

Handling updates to 7000 user accounts per second isn't what makes it hard for twitter. It is the propagation of these updates to a gazillion followers that taxes the system. Their primary overhead is the messaging update to followers. Ashton Kutcher has 6 million followers. Sending instantaneous notifications to all those million twitter clients is what makes it a killer. TLDR; It is not the vertices in the twitter graph but the number of edges in it that is unprecedented.

benologist 16y ago
Unprecedented, except is it really unprecedented or are we just looking at it from the sexy social startup whatever must say graph and also disrupt angle, where everything is new and super hard and nobody has problems that have ever been solved before. RSS strikes me as an almost identical process other than the time subscribers wait to check for new content, and there are feeds with millions of subscribers. So now the problem is reduced to doing it in a timely fashion. This is why I wonder if they're really that immensely complicated - isn't Google doing much the same thing, and possibly even at a bigger scale at one point, with their reader and FeedBurner?
- jimbokun 16y ago
  "isn't Google doing much the same thing, and possibly even at a bigger scale at one point, with their reader and FeedBurner?" Another company with a ton of employees. Probably not that many on the FeedBurner team itself, but a lot of people at Google working on the general problem of keeping things unreasonably responsive at massive scale.
Splines 16y ago
This is exactly the reason why every IM network has a limit to the number of contacts that you can have. I recall reading several years ago that this was an incredibly hard problem for IM providers to solve, and AFAIK none of them have allow you to have an infinite number of contacts.

nostrademons 16y ago

As has been pointed out elsewhere on the thread, complexity doesn't scale linearly. It's far easier to write ten sites that each gets 30 qps than it is to write one site that gets 300 qps, and it's easier to write ten sites that get 300 qps than one site that gets 3000 qps. Twitter's fundamental problem also is a harder one to scale than something like Heroku or Wordpress. For those hosted sites, you can shard easily by host, so that each of the 100,000 Heroku-hosted sites can get its own EC2 instance(s) and behave pretty much independently. You can't do that when the point of your site is that any action might instantly be broadcast to thousands of followers. High-fanout writes are not an easy problem to solve.

beagle3 16y ago
But if you design it well enough, it does. Here's a copy-pasta from a reddit comment of mine from a few months ago: http://www.reddit.com/r/programming/comments/b2u6t/twitter_opensource_a_list_of_all_open_source/c0kq4le http://www.reddit.com/r/programming/comments/b2u6t/twitter_o... ------------------ Color me unimpressed. At some point, I was collecting 40GB/day of financial data (and that's after bzip2ing them .. probably 200GB/day before); This was done on hardware costing $30K (which was two equivalent machines with 4GB, each having 20*1TB in a raid configuration -- this was a hot-backup configuration) and the operation run (coded, supervised, administered) by 2 people. I'm extrapolating from your numbers: Let's say you have 70GB over 14 days = 5GB/day; Let's assume Twitter has 100GB/day of text twits (which, incidentally, means ~1 billion tweets per day which I highly doubt, as they took few years to get to the 1 billion mark, and last I heard they were at less than 100 million twits/day) Then, at this day and age (numbers selected for 2 years ago, when they had their last infrastructure revision), what you do is buy 20 servers with 8GB of memory each (for, say $5K each), plus a little redundancy, and store all the latest twits in memory, and the most popular user's older twits as well; everything else on disk. Throw in cheap web front-ends that don't even need a local disk, load balancing, and a gigabit ethernet backplane. You're still under $200K in equipment. Yes, the code is not going to be trivial, but for $100K and 3 months you can get a stunt programmer (I know a few who can do it and won't charge as much). A run-of-the-mill RDBMS is the wrong tool for this job; Basically, run of the mill anything; but that does not make it incredibly hard. I think $300K for hardware and software, you can get a Twitter clone that performs as well. Twitter is successful, but that's not thanks to good engineering.

tzs 16y ago

Yeah, the number I had was for total API calls (6 billion a day), not just updates. http://techcrunch.com/2010/09/17/twitter-seeing-6-billion-api-calls-per-day-70k-per-second/ http://techcrunch.com/2010/09/17/twitter-seeing-6-billion-ap...