6 ms·

4100 employees for a service dedicated to moving around 140 character messages? Yeah, there might be a little fat to trim there.

by xienze 11y ago

menzoic 11y ago
thats a naive assumption, there could be a lot of hidden scaling issues when you think deeper about how it works
- xienze 11y ago
  I don't doubt it's challenging to keep the thing up, but 4100 employees? And meanwhile WhatsApp has 50 or so employees in total.
  - werber 11y ago
    It's crazy that it makes sense a billion+ company would need that few people now
    - swingbridge 11y ago
      Well a lot of other people are saying "It's crazy that a company with that few people and that little profit can be worth that many billions." Hence Twitter's problem at the moment.
      - toomuchtodo 11y ago
        Somewhat curious if this is the beginning of the unicorn pop.
        eclipxe 11y ago
        Yes, it is.
        toomuchtodo 11y ago
        A bit of history in the making if we're at the precipice.
    - tootie 11y ago
      I wonder how many of the 4100 are tech. Probably less than half at this point. The rest are there to work with advertisers, publishers, celebrities. All to maximize profitability and all of them failing at it.
      - eclipxe 11y ago
        50% are tech
  - deleted 11y ago
    [deleted]
  - aetherson 11y ago
    Twitter is doing something vastly more technically challenging that WhatsApp. WhatsApp just delivers messages. It doesn't store them, and it certainly doesn't, say, try to make them available in a searchable away to everyone in the world. This isn't a knock on WhatsApp -- I use WhatsApp and don't use Twitter -- but Twitter's scale challenges are vastly more complex than WhatsApp's, which doesn't have many features that can't just very simply horizontally scale.
    - hugh4 11y ago
      I'm still not sure how that requires a number of employees comparable to, say, NASA JPL.
    - beagle3 11y ago
      > It doesn't store them, and it certainly doesn't, say, try to make them available in a searchable away to everyone in the world. Back of the envelope calculation: If your messages are 256 char messages, on a 128GB ram server you can fit 500 billion of them in memory; incidentally, that's one day's worth of tweets these days. I recently got a quote for a beefy 256GB ram server from Dell for $6,000. To make things easier, let's assume $10,000 with redundant 1TB SSD and everything else you might need. The data is extremely easy to shard. With proper caching, you can probably keep 99.9999% of all tweets shown in just one such servers, and serve them within a few ms. I don't have recent experience with web frontends, but a properly configured nginx could probably server on meager hardware can do a few tens-of-thousands requests per second. a Seastar server on beefy hardware can probably handle a significant percentage of Twitter's load. Search is a little harder, but not by much. Is this engineering trivial? Far from it. But I've worked with stunt programmers who can build this in a few months with a 3 person team. At least one of them commands a salary of about $500K last I heard. Is that all Twitter is doing? No, I'm sure they have 10 backend "things" for every user visible "thing". They needed something like Bootstrap for internal use, after all. I don't really believe you can run Twitter with just $1M/year in capex, just $1M/year in opex and just $1M/year in engineering salary - you need a whole lot more than that; In fact, to pull in revenue of $2B/year (which Twitter seems to be on track for), you probably need 50 people just in accounting. What I am saying, though, is that the technical challenges faced by Twitter (at least those that are visible to users) are highly overrated. If you know what you are doing, Twitter can be run on $50,000 worth of hardware; This has been true since 2007 - Computers were weaker and had less memory then, but Twitter had much less traffic too. I've looked at the numbers every single year since 2007, and at any point in time, the "bare bones essential hardware for interactive response" was in the $50,000 range. Which probably means if they spent more than $1M/year on hardware (or depreciation-equivalent-hosting) at any point in time, they should have hired better engineers.
  - nemothekid 11y ago
    WhatsApp can't tell me how many users were talking about Peanut Butter Sandwiches 30 seconds ago. WhatsApp can't alert me when people suddenly start talking about earthquakes, super bowls, or Elon Musk. WhatsApp doesn't need to fanout a message to millions of consumers, millions of times a day (and with multi-second latency to boot!). WhatsApp also doesn't provide a CDN, mobile analytics/monetization/debugging platform. WhatsApp also doesn't contribute to open source like Twitter does (Storm, Bootstrap, Aurora). Describing twitter as a "140-char" mailing list is very reductionist. OTOH WhatsApp problem is very well served by Erlang/Mnesia an open source distributed messaging library developed by telecommunications company who were likely doing WhatApp-scale 10 yeas before WhatsApp came around. While WhatsApp engineers are great, there is very much a "standing on the shoulders of giants" - giants who have well researched the problem of building a system that lets two or three individuals communicate. There wasn't anyone doing Twitter 10 years ago (well, maybe Google). I don't mean to imply that Twitter is an efficient well oiled machine, but the comparison to WhatsApp is daft.
    - oinksoft 11y ago
      You're underestimating the difficulty of scaling Mnesia the way WhatsApp did (one source says "In-memory Mnesia database using about 2TB of RAM sharded across 16 partitions to store about 18 billion records." [1]). The prevailing wisdom in the Erlang community was that Mnesia could not be made to do such things. The only well-known user of Mnesia at anywhere near this scale prior to WhatsApp was Klarna. To achieve this, Klarna employed at least one of the original Mnesia developers who wrote proprietary patches for their system [2]. So while you accuse parent of being reductionist, your views of WhatsApp's engineering challenges also seem reductionist. Erlang/OTP software doesn't write and scale itself ... large-scale distributed systems are decidedly non-trivial no matter what tools you've chosen. [1] http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-to-nearly-500-million-users-11000-cores-an.html http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-... [2] https://www.erlang-factory.com/upload/presentations/601/EUC2012-009.pdf https://www.erlang-factory.com/upload/presentations/601/EUC2...
      - nemothekid 11y ago
deleted 11y ago
[deleted]
yid 11y ago
> 4100 employees for a service dedicated to moving around 140 character messages? Yeah, there might be a little fat to trim there. Absolutely! You could probably get a single rack to handle storage and broadcast of millions of tweets per minute, perhaps hook it up to a Teradata appliance to handle search and storage at that throughput. Throw it in a nice distributed database indexed by user. Then just another couple of indices for the hashtags and mentions, and the follows/follower graph. Throw in another few tables for handling API permissions, managing schema migrations while ensuring that data from day 1 is still as transparent as yesterday's logs, a highly replicated ad server hooked up to a realtime recommendation engine that analyzes an active user's current stream and recommends the most relevant ad (to maximize revenue of course), set up a few staffed policy teams to comply with the differences in global regulations, handle lawsuits and legal requests, and of course keep competing for more revenue with the likes of Facebook and Google. It's pretty trivial.
- xienze 11y ago
  OK, I'm hearing a lot about hardware there but not seeing where thousands of people are needed to keep it all up. Again, WhatsApp -- 50 employees.
  - philipjoubert 11y ago
    Whatsapp is an outlier in many ways and comparing them with any company will make that company seem bloated. As for the comparison, Whatsapp has no monetization model, no marketing and a simpler product.
    - Scoundreller 11y ago
      WhatsApp charges a subscription fee after the first year of $1/year. When you have several hundred million users, that adds up. An effective monetization model, if unusual in SV.
      - aetherson 11y ago
        But it actually doesn't charge that subscription fee. At least not to the overwhelming majority of its users.
robryan 11y ago
Without knowing the breakdown I assume that there are a lot of people working in marketing type roles where they are working with big business and celebrities to get them using the platform.
- deleted 11y ago
  [deleted]
- pkaye 11y ago
  Their website says that 50% percent of their employees are engineers. (https://about.twitter.com/company https://about.twitter.com/company)
  - Scoundreller 11y ago
    The engineers may still work to support advertising/marketing/sales. Deciding which ad is the most optimal to show benefits from every input (metadata?) you can make available and analyze. That must require some multiple of effort vs. making sure 140 chars get shown where they're directed to go.
meeper16 11y ago
I completely agree. Twitter was never really a technology company to me considering it was essentially just a mimic of a wrapper around an email list.
- rboyd 11y ago
  Any company that operates at web scale is a technology company. Besides that look at their opensource contributions. Twitter has/had some impressive engineering talent.
  - sirkneeland 11y ago
    But is it "has" or is it already "had"?
  - meeper16 11y ago
    Try duplicating Google algorithms compared to Twitters wrappers and other functionality that can be easily duplicated.
dang 11y ago
Please don't post dismissive comments to Hacker News.

4100 employees for a service dedicated to moving around 140 character messages? Yeah, there might be a little fat to trim there.

by xienze 11y ago

4100 employees for a service dedicated to moving around 140 character messages? Yeah, there might be a little fat to trim there.

menzoic 11y ago

thats a naive assumption, there could be a lot of hidden scaling issues when you think deeper about how it works

xienze 11y ago
I don't doubt it's challenging to keep the thing up, but 4100 employees? And meanwhile WhatsApp has 50 or so employees in total.
- werber 11y ago
  It's crazy that it makes sense a billion+ company would need that few people now
  - swingbridge 11y ago
    Well a lot of other people are saying "It's crazy that a company with that few people and that little profit can be worth that many billions." Hence Twitter's problem at the moment.
    - toomuchtodo 11y ago
      Somewhat curious if this is the beginning of the unicorn pop.
      - eclipxe 11y ago
        Yes, it is.
        toomuchtodo 11y ago
        A bit of history in the making if we're at the precipice.
  - tootie 11y ago
    I wonder how many of the 4100 are tech. Probably less than half at this point. The rest are there to work with advertisers, publishers, celebrities. All to maximize profitability and all of them failing at it.
    - eclipxe 11y ago
      50% are tech
- deleted 11y ago
  [deleted]
- aetherson 11y ago
  Twitter is doing something vastly more technically challenging that WhatsApp. WhatsApp just delivers messages. It doesn't store them, and it certainly doesn't, say, try to make them available in a searchable away to everyone in the world. This isn't a knock on WhatsApp -- I use WhatsApp and don't use Twitter -- but Twitter's scale challenges are vastly more complex than WhatsApp's, which doesn't have many features that can't just very simply horizontally scale.
  - hugh4 11y ago
    I'm still not sure how that requires a number of employees comparable to, say, NASA JPL.
  - beagle3 11y ago
    > It doesn't store them, and it certainly doesn't, say, try to make them available in a searchable away to everyone in the world. Back of the envelope calculation: If your messages are 256 char messages, on a 128GB ram server you can fit 500 billion of them in memory; incidentally, that's one day's worth of tweets these days. I recently got a quote for a beefy 256GB ram server from Dell for $6,000. To make things easier, let's assume $10,000 with redundant 1TB SSD and everything else you might need. The data is extremely easy to shard. With proper caching, you can probably keep 99.9999% of all tweets shown in just one such servers, and serve them within a few ms. I don't have recent experience with web frontends, but a properly configured nginx could probably server on meager hardware can do a few tens-of-thousands requests per second. a Seastar server on beefy hardware can probably handle a significant percentage of Twitter's load. Search is a little harder, but not by much. Is this engineering trivial? Far from it. But I've worked with stunt programmers who can build this in a few months with a 3 person team. At least one of them commands a salary of about $500K last I heard. Is that all Twitter is doing? No, I'm sure they have 10 backend "things" for every user visible "thing". They needed something like Bootstrap for internal use, after all. I don't really believe you can run Twitter with just $1M/year in capex, just $1M/year in opex and just $1M/year in engineering salary - you need a whole lot more than that; In fact, to pull in revenue of $2B/year (which Twitter seems to be on track for), you probably need 50 people just in accounting. What I am saying, though, is that the technical challenges faced by Twitter (at least those that are visible to users) are highly overrated. If you know what you are doing, Twitter can be run on $50,000 worth of hardware; This has been true since 2007 - Computers were weaker and had less memory then, but Twitter had much less traffic too. I've looked at the numbers every single year since 2007, and at any point in time, the "bare bones essential hardware for interactive response" was in the $50,000 range. Which probably means if they spent more than $1M/year on hardware (or depreciation-equivalent-hosting) at any point in time, they should have hired better engineers.
- nemothekid 11y ago
  WhatsApp can't tell me how many users were talking about Peanut Butter Sandwiches 30 seconds ago. WhatsApp can't alert me when people suddenly start talking about earthquakes, super bowls, or Elon Musk. WhatsApp doesn't need to fanout a message to millions of consumers, millions of times a day (and with multi-second latency to boot!). WhatsApp also doesn't provide a CDN, mobile analytics/monetization/debugging platform. WhatsApp also doesn't contribute to open source like Twitter does (Storm, Bootstrap, Aurora). Describing twitter as a "140-char" mailing list is very reductionist. OTOH WhatsApp problem is very well served by Erlang/Mnesia an open source distributed messaging library developed by telecommunications company who were likely doing WhatApp-scale 10 yeas before WhatsApp came around. While WhatsApp engineers are great, there is very much a "standing on the shoulders of giants" - giants who have well researched the problem of building a system that lets two or three individuals communicate. There wasn't anyone doing Twitter 10 years ago (well, maybe Google). I don't mean to imply that Twitter is an efficient well oiled machine, but the comparison to WhatsApp is daft.
  - oinksoft 11y ago
    You're underestimating the difficulty of scaling Mnesia the way WhatsApp did (one source says "In-memory Mnesia database using about 2TB of RAM sharded across 16 partitions to store about 18 billion records." [1]). The prevailing wisdom in the Erlang community was that Mnesia could not be made to do such things. The only well-known user of Mnesia at anywhere near this scale prior to WhatsApp was Klarna. To achieve this, Klarna employed at least one of the original Mnesia developers who wrote proprietary patches for their system [2]. So while you accuse parent of being reductionist, your views of WhatsApp's engineering challenges also seem reductionist. Erlang/OTP software doesn't write and scale itself ... large-scale distributed systems are decidedly non-trivial no matter what tools you've chosen. [1] http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-to-nearly-500-million-users-11000-cores-an.html http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-... [2] https://www.erlang-factory.com/upload/presentations/601/EUC2012-009.pdf https://www.erlang-factory.com/upload/presentations/601/EUC2...
    - nemothekid 11y ago

deleted 11y ago

[deleted]

yid 11y ago

> 4100 employees for a service dedicated to moving around 140 character messages? Yeah, there might be a little fat to trim there. Absolutely! You could probably get a single rack to handle storage and broadcast of millions of tweets per minute, perhaps hook it up to a Teradata appliance to handle search and storage at that throughput. Throw it in a nice distributed database indexed by user. Then just another couple of indices for the hashtags and mentions, and the follows/follower graph. Throw in another few tables for handling API permissions, managing schema migrations while ensuring that data from day 1 is still as transparent as yesterday's logs, a highly replicated ad server hooked up to a realtime recommendation engine that analyzes an active user's current stream and recommends the most relevant ad (to maximize revenue of course), set up a few staffed policy teams to comply with the differences in global regulations, handle lawsuits and legal requests, and of course keep competing for more revenue with the likes of Facebook and Google. It's pretty trivial.

xienze 11y ago
OK, I'm hearing a lot about hardware there but not seeing where thousands of people are needed to keep it all up. Again, WhatsApp -- 50 employees.
- philipjoubert 11y ago
  Whatsapp is an outlier in many ways and comparing them with any company will make that company seem bloated. As for the comparison, Whatsapp has no monetization model, no marketing and a simpler product.
  - Scoundreller 11y ago
    WhatsApp charges a subscription fee after the first year of $1/year. When you have several hundred million users, that adds up. An effective monetization model, if unusual in SV.
    - aetherson 11y ago
      But it actually doesn't charge that subscription fee. At least not to the overwhelming majority of its users.

robryan 11y ago

Without knowing the breakdown I assume that there are a lot of people working in marketing type roles where they are working with big business and celebrities to get them using the platform.

deleted 11y ago
[deleted]
pkaye 11y ago
Their website says that 50% percent of their employees are engineers. (https://about.twitter.com/company https://about.twitter.com/company)
- Scoundreller 11y ago
  The engineers may still work to support advertising/marketing/sales. Deciding which ad is the most optimal to show benefits from every input (metadata?) you can make available and analyze. That must require some multiple of effort vs. making sure 140 chars get shown where they're directed to go.

meeper16 11y ago

I completely agree. Twitter was never really a technology company to me considering it was essentially just a mimic of a wrapper around an email list.

rboyd 11y ago
Any company that operates at web scale is a technology company. Besides that look at their opensource contributions. Twitter has/had some impressive engineering talent.
- sirkneeland 11y ago
  But is it "has" or is it already "had"?
- meeper16 11y ago
  Try duplicating Google algorithms compared to Twitters wrappers and other functionality that can be easily duplicated.

dang 11y ago

Please don't post dismissive comments to Hacker News.