5 ms·

thats a naive assumption, there could be a lot of hidden scaling issues when you think deeper about how it works

by menzoic 11y ago

xienze 11y ago
I don't doubt it's challenging to keep the thing up, but 4100 employees? And meanwhile WhatsApp has 50 or so employees in total.
- werber 11y ago
  It's crazy that it makes sense a billion+ company would need that few people now
  - swingbridge 11y ago
    Well a lot of other people are saying "It's crazy that a company with that few people and that little profit can be worth that many billions." Hence Twitter's problem at the moment.
    - toomuchtodo 11y ago
      Somewhat curious if this is the beginning of the unicorn pop.
      - eclipxe 11y ago
        Yes, it is.
        toomuchtodo 11y ago
        A bit of history in the making if we're at the precipice.
  - tootie 11y ago
    I wonder how many of the 4100 are tech. Probably less than half at this point. The rest are there to work with advertisers, publishers, celebrities. All to maximize profitability and all of them failing at it.
    - eclipxe 11y ago
      50% are tech
- deleted 11y ago
  [deleted]
- aetherson 11y ago
  Twitter is doing something vastly more technically challenging that WhatsApp. WhatsApp just delivers messages. It doesn't store them, and it certainly doesn't, say, try to make them available in a searchable away to everyone in the world. This isn't a knock on WhatsApp -- I use WhatsApp and don't use Twitter -- but Twitter's scale challenges are vastly more complex than WhatsApp's, which doesn't have many features that can't just very simply horizontally scale.
  - hugh4 11y ago
    I'm still not sure how that requires a number of employees comparable to, say, NASA JPL.
  - beagle3 11y ago
    > It doesn't store them, and it certainly doesn't, say, try to make them available in a searchable away to everyone in the world. Back of the envelope calculation: If your messages are 256 char messages, on a 128GB ram server you can fit 500 billion of them in memory; incidentally, that's one day's worth of tweets these days. I recently got a quote for a beefy 256GB ram server from Dell for $6,000. To make things easier, let's assume $10,000 with redundant 1TB SSD and everything else you might need. The data is extremely easy to shard. With proper caching, you can probably keep 99.9999% of all tweets shown in just one such servers, and serve them within a few ms. I don't have recent experience with web frontends, but a properly configured nginx could probably server on meager hardware can do a few tens-of-thousands requests per second. a Seastar server on beefy hardware can probably handle a significant percentage of Twitter's load. Search is a little harder, but not by much. Is this engineering trivial? Far from it. But I've worked with stunt programmers who can build this in a few months with a 3 person team. At least one of them commands a salary of about $500K last I heard. Is that all Twitter is doing? No, I'm sure they have 10 backend "things" for every user visible "thing". They needed something like Bootstrap for internal use, after all. I don't really believe you can run Twitter with just $1M/year in capex, just $1M/year in opex and just $1M/year in engineering salary - you need a whole lot more than that; In fact, to pull in revenue of $2B/year (which Twitter seems to be on track for), you probably need 50 people just in accounting. What I am saying, though, is that the technical challenges faced by Twitter (at least those that are visible to users) are highly overrated. If you know what you are doing, Twitter can be run on $50,000 worth of hardware; This has been true since 2007 - Computers were weaker and had less memory then, but Twitter had much less traffic too. I've looked at the numbers every single year since 2007, and at any point in time, the "bare bones essential hardware for interactive response" was in the $50,000 range. Which probably means if they spent more than $1M/year on hardware (or depreciation-equivalent-hosting) at any point in time, they should have hired better engineers.
- nemothekid 11y ago
  WhatsApp can't tell me how many users were talking about Peanut Butter Sandwiches 30 seconds ago. WhatsApp can't alert me when people suddenly start talking about earthquakes, super bowls, or Elon Musk. WhatsApp doesn't need to fanout a message to millions of consumers, millions of times a day (and with multi-second latency to boot!). WhatsApp also doesn't provide a CDN, mobile analytics/monetization/debugging platform. WhatsApp also doesn't contribute to open source like Twitter does (Storm, Bootstrap, Aurora). Describing twitter as a "140-char" mailing list is very reductionist. OTOH WhatsApp problem is very well served by Erlang/Mnesia an open source distributed messaging library developed by telecommunications company who were likely doing WhatApp-scale 10 yeas before WhatsApp came around. While WhatsApp engineers are great, there is very much a "standing on the shoulders of giants" - giants who have well researched the problem of building a system that lets two or three individuals communicate. There wasn't anyone doing Twitter 10 years ago (well, maybe Google). I don't mean to imply that Twitter is an efficient well oiled machine, but the comparison to WhatsApp is daft.
  - oinksoft 11y ago
    You're underestimating the difficulty of scaling Mnesia the way WhatsApp did (one source says "In-memory Mnesia database using about 2TB of RAM sharded across 16 partitions to store about 18 billion records." [1]). The prevailing wisdom in the Erlang community was that Mnesia could not be made to do such things. The only well-known user of Mnesia at anywhere near this scale prior to WhatsApp was Klarna. To achieve this, Klarna employed at least one of the original Mnesia developers who wrote proprietary patches for their system [2]. So while you accuse parent of being reductionist, your views of WhatsApp's engineering challenges also seem reductionist. Erlang/OTP software doesn't write and scale itself ... large-scale distributed systems are decidedly non-trivial no matter what tools you've chosen. [1] http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-to-nearly-500-million-users-11000-cores-an.html http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-... [2] https://www.erlang-factory.com/upload/presentations/601/EUC2012-009.pdf https://www.erlang-factory.com/upload/presentations/601/EUC2...
    - nemothekid 11y ago

thats a naive assumption, there could be a lot of hidden scaling issues when you think deeper about how it works

by menzoic 11y ago

thats a naive assumption, there could be a lot of hidden scaling issues when you think deeper about how it works

I don't doubt it's challenging to keep the thing up, but 4100 employees? And meanwhile WhatsApp has 50 or so employees in total.

werber 11y ago
It's crazy that it makes sense a billion+ company would need that few people now
- swingbridge 11y ago
  Well a lot of other people are saying "It's crazy that a company with that few people and that little profit can be worth that many billions." Hence Twitter's problem at the moment.
  - toomuchtodo 11y ago
    Somewhat curious if this is the beginning of the unicorn pop.
    - eclipxe 11y ago
      Yes, it is.
      - toomuchtodo 11y ago
        A bit of history in the making if we're at the precipice.
- tootie 11y ago
  I wonder how many of the 4100 are tech. Probably less than half at this point. The rest are there to work with advertisers, publishers, celebrities. All to maximize profitability and all of them failing at it.
  - eclipxe 11y ago
    50% are tech
deleted 11y ago
[deleted]
aetherson 11y ago
Twitter is doing something vastly more technically challenging that WhatsApp. WhatsApp just delivers messages. It doesn't store them, and it certainly doesn't, say, try to make them available in a searchable away to everyone in the world. This isn't a knock on WhatsApp -- I use WhatsApp and don't use Twitter -- but Twitter's scale challenges are vastly more complex than WhatsApp's, which doesn't have many features that can't just very simply horizontally scale.
- hugh4 11y ago
  I'm still not sure how that requires a number of employees comparable to, say, NASA JPL.
- beagle3 11y ago
  > It doesn't store them, and it certainly doesn't, say, try to make them available in a searchable away to everyone in the world. Back of the envelope calculation: If your messages are 256 char messages, on a 128GB ram server you can fit 500 billion of them in memory; incidentally, that's one day's worth of tweets these days. I recently got a quote for a beefy 256GB ram server from Dell for $6,000. To make things easier, let's assume $10,000 with redundant 1TB SSD and everything else you might need. The data is extremely easy to shard. With proper caching, you can probably keep 99.9999% of all tweets shown in just one such servers, and serve them within a few ms. I don't have recent experience with web frontends, but a properly configured nginx could probably server on meager hardware can do a few tens-of-thousands requests per second. a Seastar server on beefy hardware can probably handle a significant percentage of Twitter's load. Search is a little harder, but not by much. Is this engineering trivial? Far from it. But I've worked with stunt programmers who can build this in a few months with a 3 person team. At least one of them commands a salary of about $500K last I heard. Is that all Twitter is doing? No, I'm sure they have 10 backend "things" for every user visible "thing". They needed something like Bootstrap for internal use, after all. I don't really believe you can run Twitter with just $1M/year in capex, just $1M/year in opex and just $1M/year in engineering salary - you need a whole lot more than that; In fact, to pull in revenue of $2B/year (which Twitter seems to be on track for), you probably need 50 people just in accounting. What I am saying, though, is that the technical challenges faced by Twitter (at least those that are visible to users) are highly overrated. If you know what you are doing, Twitter can be run on $50,000 worth of hardware; This has been true since 2007 - Computers were weaker and had less memory then, but Twitter had much less traffic too. I've looked at the numbers every single year since 2007, and at any point in time, the "bare bones essential hardware for interactive response" was in the $50,000 range. Which probably means if they spent more than $1M/year on hardware (or depreciation-equivalent-hosting) at any point in time, they should have hired better engineers.
nemothekid 11y ago
WhatsApp can't tell me how many users were talking about Peanut Butter Sandwiches 30 seconds ago. WhatsApp can't alert me when people suddenly start talking about earthquakes, super bowls, or Elon Musk. WhatsApp doesn't need to fanout a message to millions of consumers, millions of times a day (and with multi-second latency to boot!). WhatsApp also doesn't provide a CDN, mobile analytics/monetization/debugging platform. WhatsApp also doesn't contribute to open source like Twitter does (Storm, Bootstrap, Aurora). Describing twitter as a "140-char" mailing list is very reductionist. OTOH WhatsApp problem is very well served by Erlang/Mnesia an open source distributed messaging library developed by telecommunications company who were likely doing WhatApp-scale 10 yeas before WhatsApp came around. While WhatsApp engineers are great, there is very much a "standing on the shoulders of giants" - giants who have well researched the problem of building a system that lets two or three individuals communicate. There wasn't anyone doing Twitter 10 years ago (well, maybe Google). I don't mean to imply that Twitter is an efficient well oiled machine, but the comparison to WhatsApp is daft.
- oinksoft 11y ago
  You're underestimating the difficulty of scaling Mnesia the way WhatsApp did (one source says "In-memory Mnesia database using about 2TB of RAM sharded across 16 partitions to store about 18 billion records." [1]). The prevailing wisdom in the Erlang community was that Mnesia could not be made to do such things. The only well-known user of Mnesia at anywhere near this scale prior to WhatsApp was Klarna. To achieve this, Klarna employed at least one of the original Mnesia developers who wrote proprietary patches for their system [2]. So while you accuse parent of being reductionist, your views of WhatsApp's engineering challenges also seem reductionist. Erlang/OTP software doesn't write and scale itself ... large-scale distributed systems are decidedly non-trivial no matter what tools you've chosen. [1] http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-to-nearly-500-million-users-11000-cores-an.html http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-... [2] https://www.erlang-factory.com/upload/presentations/601/EUC2012-009.pdf https://www.erlang-factory.com/upload/presentations/601/EUC2...
  - nemothekid 11y ago