4 ms·

Large scale distributed systems aren't that simple. I don't know any internal details of Twitter, but from what I've gathered they are trying to hit the kind of

Large scale distributed systems aren't that simple. I don't know any internal details of Twitter, but from what I've gathered they are trying to hit the kind of QoS you are dismissing. Imagine some celebrity with millions of followers says something controversial: it would be confusing to a lot of people if they start getting retweets or discussions of that tweet significantly before they see that tweet themselves. And that's just one scenario. I don't know about the 500 vs. 1000 employee question, but Twitter is a big engineering challenge.

morgante 11y ago
> Large scale distributed systems aren't that simple. Nobody's debating that. If it weren't a "large scale distributed system" then the debate would be about 3 employees vs. 10. The core experience without scale could be made by a single engineer. But running a service like Twitter doesn't require 4,000 employees, even at scale. Facebook has "only" twice as many employees, despite a 10x more complicated product and 12x as much revenue.
mason55 11y ago
How many employees did it take to run IG? Architecturally they are very similar. Now, I do think there's more to it, once you start adding in monetization and business features, but it doesn't take that many people to run a highly scalable distributed pub/sub.
r721 11y ago
>I don't know any internal details of Twitter There were quite a couple of relevant posts on Twitter Engineering Blog if you are interested: https://blog.twitter.com/2011/spiderduck-twitters-real-time-url-fetcher https://blog.twitter.com/2011/spiderduck-twitters-real-time-... https://blog.twitter.com/2012/blobstore-twitter-s-in-house-photo-storage-system https://blog.twitter.com/2012/blobstore-twitter-s-in-house-p... https://blog.twitter.com/2014/manhattan-our-real-time-multi-tenant-distributed-database-for-twitter-scale https://blog.twitter.com/2014/manhattan-our-real-time-multi-... https://blog.twitter.com/2014/push-our-limits-reliability-testing-at-twitter https://blog.twitter.com/2014/push-our-limits-reliability-te... https://blog.twitter.com/2015/building-a-new-trends-experience https://blog.twitter.com/2015/building-a-new-trends-experien...
rdtsc 11y ago
> Large scale distributed systems aren't that simple. It is not simple but it also doesn't have to be 1000 employee-type of complexity. As a counter-example I'd point to WhatsApp was handling 1B users with 50 employees. That includes custom apps for fringe phones used in 3rd world countries. It included media sharing, it including managing and ops. I know they are not Twitter but still 50 vs 1000 is a huge difference. They've traditionally credited Erlang and FreeBSD as one the techologies that was responsible for their success. Not sure if Facebook will strong-arm them into switching to whatever Facebook is using. One can then ask, well, why don't companies look at that and learn and do the same? And I think the answer is once they headed in one particular direction with their choices, they are more susceptable to the sunk cost fallacy. Coming in one morning and telling everyone, screw it, we picked the wrong direction 2 years ago. We'll lay off 50% of the people and re-write the whole thing in Swift using this other stack. We'll cut half the products and work on new ones. Etc. That is almost impossible to do. So they are sort of bound to heading in the direction they've been heading, even if individual people will all see it is probably the wrong direction.
- mahyarm 11y ago
  WhatsApp chose engineering cost first. It's a singular device only service, they do not retain any messages and a slew of other decisions. Look at their iOS client for more examples of engineering cost first. A new client for an obscure phone costs about 1 engineer for WhatsApp too. Twitter is not an engineering cost first company.
  - rdtsc 11y ago
    > they do not retain any messages Right, though they retain messages until their recipient comes online.

morgante 11y ago

> Large scale distributed systems aren't that simple. Nobody's debating that. If it weren't a "large scale distributed system" then the debate would be about 3 employees vs. 10. The core experience without scale could be made by a single engineer. But running a service like Twitter doesn't require 4,000 employees, even at scale. Facebook has "only" twice as many employees, despite a 10x more complicated product and 12x as much revenue.

mason55 11y ago

How many employees did it take to run IG? Architecturally they are very similar. Now, I do think there's more to it, once you start adding in monetization and business features, but it doesn't take that many people to run a highly scalable distributed pub/sub.

r721 11y ago

>I don't know any internal details of Twitter There were quite a couple of relevant posts on Twitter Engineering Blog if you are interested: https://blog.twitter.com/2011/spiderduck-twitters-real-time-url-fetcher https://blog.twitter.com/2011/spiderduck-twitters-real-time-... https://blog.twitter.com/2012/blobstore-twitter-s-in-house-photo-storage-system https://blog.twitter.com/2012/blobstore-twitter-s-in-house-p... https://blog.twitter.com/2014/manhattan-our-real-time-multi-tenant-distributed-database-for-twitter-scale https://blog.twitter.com/2014/manhattan-our-real-time-multi-... https://blog.twitter.com/2014/push-our-limits-reliability-testing-at-twitter https://blog.twitter.com/2014/push-our-limits-reliability-te... https://blog.twitter.com/2015/building-a-new-trends-experience https://blog.twitter.com/2015/building-a-new-trends-experien...

rdtsc 11y ago

> Large scale distributed systems aren't that simple. It is not simple but it also doesn't have to be 1000 employee-type of complexity. As a counter-example I'd point to WhatsApp was handling 1B users with 50 employees. That includes custom apps for fringe phones used in 3rd world countries. It included media sharing, it including managing and ops. I know they are not Twitter but still 50 vs 1000 is a huge difference. They've traditionally credited Erlang and FreeBSD as one the techologies that was responsible for their success. Not sure if Facebook will strong-arm them into switching to whatever Facebook is using. One can then ask, well, why don't companies look at that and learn and do the same? And I think the answer is once they headed in one particular direction with their choices, they are more susceptable to the sunk cost fallacy. Coming in one morning and telling everyone, screw it, we picked the wrong direction 2 years ago. We'll lay off 50% of the people and re-write the whole thing in Swift using this other stack. We'll cut half the products and work on new ones. Etc. That is almost impossible to do. So they are sort of bound to heading in the direction they've been heading, even if individual people will all see it is probably the wrong direction.

mahyarm 11y ago
WhatsApp chose engineering cost first. It's a singular device only service, they do not retain any messages and a slew of other decisions. Look at their iOS client for more examples of engineering cost first. A new client for an obscure phone costs about 1 engineer for WhatsApp too. Twitter is not an engineering cost first company.
- rdtsc 11y ago
  > they do not retain any messages Right, though they retain messages until their recipient comes online.