7 ms·

Twitter is doing something vastly more technically challenging that WhatsApp. WhatsApp just delivers messages. It doesn't store them, and it certainly doesn't

Twitter is doing something vastly more technically challenging that WhatsApp. WhatsApp just delivers messages. It doesn't store them, and it certainly doesn't, say, try to make them available in a searchable away to everyone in the world. This isn't a knock on WhatsApp -- I use WhatsApp and don't use Twitter -- but Twitter's scale challenges are vastly more complex than WhatsApp's, which doesn't have many features that can't just very simply horizontally scale.

hugh4 11y ago
I'm still not sure how that requires a number of employees comparable to, say, NASA JPL.
beagle3 11y ago
> It doesn't store them, and it certainly doesn't, say, try to make them available in a searchable away to everyone in the world. Back of the envelope calculation: If your messages are 256 char messages, on a 128GB ram server you can fit 500 billion of them in memory; incidentally, that's one day's worth of tweets these days. I recently got a quote for a beefy 256GB ram server from Dell for $6,000. To make things easier, let's assume $10,000 with redundant 1TB SSD and everything else you might need. The data is extremely easy to shard. With proper caching, you can probably keep 99.9999% of all tweets shown in just one such servers, and serve them within a few ms. I don't have recent experience with web frontends, but a properly configured nginx could probably server on meager hardware can do a few tens-of-thousands requests per second. a Seastar server on beefy hardware can probably handle a significant percentage of Twitter's load. Search is a little harder, but not by much. Is this engineering trivial? Far from it. But I've worked with stunt programmers who can build this in a few months with a 3 person team. At least one of them commands a salary of about $500K last I heard. Is that all Twitter is doing? No, I'm sure they have 10 backend "things" for every user visible "thing". They needed something like Bootstrap for internal use, after all. I don't really believe you can run Twitter with just $1M/year in capex, just $1M/year in opex and just $1M/year in engineering salary - you need a whole lot more than that; In fact, to pull in revenue of $2B/year (which Twitter seems to be on track for), you probably need 50 people just in accounting. What I am saying, though, is that the technical challenges faced by Twitter (at least those that are visible to users) are highly overrated. If you know what you are doing, Twitter can be run on $50,000 worth of hardware; This has been true since 2007 - Computers were weaker and had less memory then, but Twitter had much less traffic too. I've looked at the numbers every single year since 2007, and at any point in time, the "bare bones essential hardware for interactive response" was in the $50,000 range. Which probably means if they spent more than $1M/year on hardware (or depreciation-equivalent-hosting) at any point in time, they should have hired better engineers.
- jacques_chester 11y ago
  > you can fit 500 billion of them in memory You will also need space for fulltext indexes, indexes per hashtag (a set which grows constantly and never shrinks, but has a power distribution, meaning that some hot tags will overwhelm any single machine and it constantly changes), the ability to call up any tweet from any point in history (growing by one beefy Dell server per day), the ability to fan out to 1 or 10 or 10 million followers and so on. The fanout problem turns sharding into a lumpy problem because again, there is a power law distribution that can overwhelm any single machine. Then there's the fun part that Twitter is so reliable these days that when something else on the internet breaks, we check Twitter for updates. Robustness at that scale is more than buying a pair of SSDs and setting up RAID. One thing I've learnt from being a consulting engineer is to stop assuming other engineers are morons. Sometimes we do things out of ignorance, or out of fear, or out of necessity. But often, when you inquire, there is a solid non-obvious reason for non-trivial diversions from the simplest thing that could possibly work. I always find it useful to give the benefit of the doubt.
  - beagle3 11y ago
    I do not claim it is trivial. But .. fulltext index done right, including per hashtag, doubles the storage requirements at most. And the ability to call up any tweet from history does not need to be as instantaneous is recent tweets (It's ok to wait 5 seconds for a tweet from 5 years ago). > The fanout problem turns sharding into a lumpy problem because again, there is a power law distribution that can overwhelm any single machine. Not really, in the case of twitter (where you only trace edges, not edges-of-edges and edges-of-edges-of-edges like Facebook and Linkedin do). In the simplified model I described, the web frontends all pull from the backend, but for the hot 10-million account followers -- of which there aren't that many --, you can just push them to the webservers so they don't even need to query) > Robustness at that scale is more than buying a pair of SSDs and setting up RAID. I most definitely agree. But it is cheaply doable when everything is so perfectly shardable as it is in Twitter. > Then there's the fun part that Twitter is so reliable these days that when something else on the internet breaks, we check Twitter for updates. That's possibly an illusion. Yes, Twitter is mostly reliable - but do you have any latency stats? e.g., would you know if, on a daily bases, 10% of tweets take 60 seconds until they appear on a viewer's refresh? The "hot" accounts would be cached and immediately updated, but the long tail might have minutes delay and almost no one would notice. > But often, when you inquire, there is a solid non-obvious reason for non-trivial diversions from the simplest thing that could possibly work. > I always find it useful to give the benefit of the doubt. I do not assume other engineers are morons. In my experience as a consultant, though, those solid non-obvious reasons for non-trivial diversions are much more often than not "historical, do not apply any more", "we didn't have time or resources to do it the right way", "the guy who did the initial design did it wrong for whatever reason, and now we are stuck with it", "there's a legal reason that's not obvious why we can't do it that way", and a few others. Twitter had the resources to do it better from very early on, and they didn't (I think it was 2012 or so before Twitter became reliable). For all I know, their system now could be the most efficient beast ever, and run on a ZX81 with 1K Ram with ultimate reliability, way better than I could ever hope to build. I was just pointing out that the user facing side of twitter, technically speaking, is not very impressive. I've been doing it for a while on various technical forums - and not once did anyone offer a reason for why it's much harder than it would seem -- most of the responses were along the lines of "but mysql/pgsql/oracle can't take the firehose load, I tried!". Which is correct, but irrelevant.

hugh4 11y ago

I'm still not sure how that requires a number of employees comparable to, say, NASA JPL.

beagle3 11y ago

> It doesn't store them, and it certainly doesn't, say, try to make them available in a searchable away to everyone in the world. Back of the envelope calculation: If your messages are 256 char messages, on a 128GB ram server you can fit 500 billion of them in memory; incidentally, that's one day's worth of tweets these days. I recently got a quote for a beefy 256GB ram server from Dell for $6,000. To make things easier, let's assume $10,000 with redundant 1TB SSD and everything else you might need. The data is extremely easy to shard. With proper caching, you can probably keep 99.9999% of all tweets shown in just one such servers, and serve them within a few ms. I don't have recent experience with web frontends, but a properly configured nginx could probably server on meager hardware can do a few tens-of-thousands requests per second. a Seastar server on beefy hardware can probably handle a significant percentage of Twitter's load. Search is a little harder, but not by much. Is this engineering trivial? Far from it. But I've worked with stunt programmers who can build this in a few months with a 3 person team. At least one of them commands a salary of about $500K last I heard. Is that all Twitter is doing? No, I'm sure they have 10 backend "things" for every user visible "thing". They needed something like Bootstrap for internal use, after all. I don't really believe you can run Twitter with just $1M/year in capex, just $1M/year in opex and just $1M/year in engineering salary - you need a whole lot more than that; In fact, to pull in revenue of $2B/year (which Twitter seems to be on track for), you probably need 50 people just in accounting. What I am saying, though, is that the technical challenges faced by Twitter (at least those that are visible to users) are highly overrated. If you know what you are doing, Twitter can be run on $50,000 worth of hardware; This has been true since 2007 - Computers were weaker and had less memory then, but Twitter had much less traffic too. I've looked at the numbers every single year since 2007, and at any point in time, the "bare bones essential hardware for interactive response" was in the $50,000 range. Which probably means if they spent more than $1M/year on hardware (or depreciation-equivalent-hosting) at any point in time, they should have hired better engineers.

jacques_chester 11y ago
> you can fit 500 billion of them in memory You will also need space for fulltext indexes, indexes per hashtag (a set which grows constantly and never shrinks, but has a power distribution, meaning that some hot tags will overwhelm any single machine and it constantly changes), the ability to call up any tweet from any point in history (growing by one beefy Dell server per day), the ability to fan out to 1 or 10 or 10 million followers and so on. The fanout problem turns sharding into a lumpy problem because again, there is a power law distribution that can overwhelm any single machine. Then there's the fun part that Twitter is so reliable these days that when something else on the internet breaks, we check Twitter for updates. Robustness at that scale is more than buying a pair of SSDs and setting up RAID. One thing I've learnt from being a consulting engineer is to stop assuming other engineers are morons. Sometimes we do things out of ignorance, or out of fear, or out of necessity. But often, when you inquire, there is a solid non-obvious reason for non-trivial diversions from the simplest thing that could possibly work. I always find it useful to give the benefit of the doubt.
- beagle3 11y ago
  I do not claim it is trivial. But .. fulltext index done right, including per hashtag, doubles the storage requirements at most. And the ability to call up any tweet from history does not need to be as instantaneous is recent tweets (It's ok to wait 5 seconds for a tweet from 5 years ago). > The fanout problem turns sharding into a lumpy problem because again, there is a power law distribution that can overwhelm any single machine. Not really, in the case of twitter (where you only trace edges, not edges-of-edges and edges-of-edges-of-edges like Facebook and Linkedin do). In the simplified model I described, the web frontends all pull from the backend, but for the hot 10-million account followers -- of which there aren't that many --, you can just push them to the webservers so they don't even need to query) > Robustness at that scale is more than buying a pair of SSDs and setting up RAID. I most definitely agree. But it is cheaply doable when everything is so perfectly shardable as it is in Twitter. > Then there's the fun part that Twitter is so reliable these days that when something else on the internet breaks, we check Twitter for updates. That's possibly an illusion. Yes, Twitter is mostly reliable - but do you have any latency stats? e.g., would you know if, on a daily bases, 10% of tweets take 60 seconds until they appear on a viewer's refresh? The "hot" accounts would be cached and immediately updated, but the long tail might have minutes delay and almost no one would notice. > But often, when you inquire, there is a solid non-obvious reason for non-trivial diversions from the simplest thing that could possibly work. > I always find it useful to give the benefit of the doubt. I do not assume other engineers are morons. In my experience as a consultant, though, those solid non-obvious reasons for non-trivial diversions are much more often than not "historical, do not apply any more", "we didn't have time or resources to do it the right way", "the guy who did the initial design did it wrong for whatever reason, and now we are stuck with it", "there's a legal reason that's not obvious why we can't do it that way", and a few others. Twitter had the resources to do it better from very early on, and they didn't (I think it was 2012 or so before Twitter became reliable). For all I know, their system now could be the most efficient beast ever, and run on a ZX81 with 1K Ram with ultimate reliability, way better than I could ever hope to build. I was just pointing out that the user facing side of twitter, technically speaking, is not very impressive. I've been doing it for a while on various technical forums - and not once did anyone offer a reason for why it's much harder than it would seem -- most of the responses were along the lines of "but mysql/pgsql/oracle can't take the firehose load, I tried!". Which is correct, but irrelevant.