47 ms·

WhatsApp scaled to 1B users with only 50 engineers

636 points by unripe_syntax 5y ago

bradhe 5y ago
Some of these points are a bit suspicious, and I bet don't tell "the whole story" the way they're intended. For instance, the statement about not investing in automation unless absolutely necessary or the point about hot-swapping Erlang code. Would be good to actually hear from one of these engineers for a discussion on the paint points of working on WhatsApp back then.
- tdrdt 5y ago
  Automation can be good, but will also be another system to support. I think this XKCD applies: https://xkcd.com/1205/ https://xkcd.com/1205/
  - bryanrasmussen 5y ago
    If you shave off 1 second I have a hard time believing that 1 second will not just be swallowed up by the entropy of coffee breaks.
  - bradhe 5y ago
    I'd like to see what their criteria for "absolutely necessary" is--I bet it doesn't fall on the right side of that table.
amelius 5y ago
Chat is the number 1 toy problem that is explained in every Erlang book out there. Therefore, honestly I'm not really surprised to see 50 engineers capable of building a product around it.
TacticalCoder 5y ago
What is amazing to me is that they did scale to 400 million monthly active users with less than 50 engineers... In 2014. That is: on hardware from seven years ago and older. Which is an eternity in tech. Since then we've seen incredible new hardware come out (EPYC 2 in 2018 for example). One can wonder: up to how many MAU could a service scale today, on today's hardware, with a small team? And what about tomorrow's hardware... This is fascinating.
- hnarn 5y ago
  Seven years can be an eternity in tech, but is it an eternity counting from 2021? I mean, AWS launched in 2006, Netflix expanded to Europe in 2012, Docker had their initial release in 2013 and so on, so none of these major events are that "new" anymore. Obviously things have changed (a lot) during the past seven years, but have they changed as much as they did between 2007 and 2014? What I'm saying is that for the general case of "scaling with few engineers", it seems to me that the tools available matter a lot more than the specifics of what hardware is available at the time. Launching an international service "from scratch" with a small team and a modest budget just wouldn't have been possible 20 years ago, but not because CPUs were too slow.
adrianlmm 5y ago
For a similar project Google would use 250 enginers, 200 of those would be diversity hires and the rest for the real work.
tiffanyh 5y ago
DuckDuckGo does around 35B searches per year with 160 employees. https://duckduckgo.com/traffic https://duckduckgo.com/traffic Not the same efficiency but it goes to show how certain domains and technology stacks can scale exceedingly well. Plus in DDG case I believe they outsource a lot of their tech to 3rd parties.
jokoon 5y ago
I remember reading they worked very hard to make their app available on many, many types of phones. Which is weird because they regularly announce they drop support for certain phone OS.
- nnx 5y ago
  It made sense years ago... before the iOS/Android duopoly we have now.
- toast0 5y ago
  I was there from 2011 to 2019. We had pretty clear numerical guidelines for continuing support of a platform or OS. If the numbers are close, things like upgradability or difficulty of support make a difference: if all the devices running OS 1.1 can actually be upgraded to 1.2 and 1.2 fixes important things, then dropping support for OS 1.1 can happen with more users still there. If builds need to be signed by the OS maker and they won't sign any more builds, user count doesn't matter (sorry Nokia S40 and iPhone 3G). We did make some longer lasting builds for platforms we were ending support for, but there needs to be an end. Older clients don't support newer features and we didn't want to have a network where that was the status quo. I don't know what the current expirations are, but mandatory updates a couple times a year helps keep the feature support similar.
kosolam 5y ago
WhatsApp is a spy app of Facebook nowadays
- oblio 5y ago
  That was its fate anyway. Some things just can't really make money unless they do it in scummy ways (or we haven't found a viable alternative or a way to protect them). Public instant messengers, public forums, public file hosting, public image hosting.
  - shawabawa3 5y ago
    With 50 employees their "please pay $1/year" model might have actually made enough revenue
    - oblio 5y ago
      Nobody I know paid it even back then. That's why they used it, because it was free. It's like Microsoft Office these days. If your home license expires they keep threatening you they'll disable it. For years and years.
    - jmrm 5y ago
      I paid totally happy with the service they gave me, and I would pay monthly if that would mean having total privacy.
  - alanfranz 5y ago
    Whatsapp used to charge 1 eur or usd per year per user. Assuming a 20% vat, that’s $800M in yearly revenue. If you spend $500k per engineer per year, you’re still left with $775M to handle infrastructure and all other expenses. Whatsapp would have been sustainable without fb. That’s the regret.
    - harpratap 5y ago
      You are very naively assuming it would actually reach 1B users if it was paid.
seanhandley 5y ago
I wonder if WhatsApp could be considered the most successful user-facing application with an Erlang backend.
- polyomino 5y ago
  What other candidates are there?
  - peoplefromibiza 5y ago
    while WhatsApp is the largest I am aware of (it's one of the leading platforms Worldwide anyway), there are quite a few, sometimes in the form of Elixir: - Discord - Pinterest - Spotify - Klarna - Riot games (the messaging service) - PepsiCo (for their billion making e-commerce) - Toyota (for the connected platform)
  - bradhe 5y ago
    Facebook Messenger, if I remember correctly, was also originally authored in Erlang.
    - mewpmewp2 5y ago
      Discord also if I remember correctly?
      - d3nj4l 5y ago
        Discord was written in Elixir! https://blog.discord.com/scaling-elixir-f9b8e1e7c29b https://blog.discord.com/scaling-elixir-f9b8e1e7c29b
        mewpmewp2 5y ago
        Does it mean it's technically also based on Erlang?
        5y ago
bjarneh 5y ago
This article does very little to explain how they did that. It basically states the type of tech they used (Erlang, FreeBSD and SoftLayer), and something about not trying to over engineer stuff. The title also makes it sound like 500 engineers would make it easier to scale up WhatsApp, which does not make too much sense either.
- Cthulhu_ 5y ago
  The article is honestly kinda low on details, depth or new insights, but at least it links to some sources.
- willejs 5y ago
  Theres a high scalability post from years ago that has a bit more information. http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-to-nearly-500-million-users-11000-cores-an.html http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-...
  - bjarneh 5y ago
    Thanks, it does seem like choosing a language that was made for doing things in parallel to begin with was a good idea. To be honest, I've always felt that programming in functional languages creates less bugs, but perhaps that's just my imagination.
    - dunefox 5y ago
      It certainly creates fewer bugs of a specific nature, depending on the language.
  - vadfa 5y ago
    >19B messages in & 40B out per day whatsapp being a closed system I don't understand how this disparity is possible.
    - the_other 5y ago
      Most people are in group chats.
corobo 5y ago
Is there even a correlation between scale and engineers, other than scale = VC money = hire more cause we can afford it?
- Ekaros 5y ago
  I would expect there to be certain scaling. Specially when you move to more severs and more redundancy. Cloud solves part, but still not everything. If single person can support 10k or 100k users. You might still need more than that one to move to million or hundred million. Also some features like language support and such get more complicated when you want to widen userbase from just English speaking ones.
  - corobo 5y ago
    > If single person can support 10k or 100k users I asked if there was a correlation here, using an if statement as an answer doesn't answer it haha. You're now asking the same question I was, just in-line
tdrdt 5y ago
"The simpler product makes it much easier to maintain and scale." This is good news for the developers, the product owners and the customers. I still can't understand why today people choose a (Javascript) stack of build systems, ton of dependencies, and all kinds of exotic tech that is the latest and most hyped. As a developer you need to support this in the future. It might be nice to build today but it will be a nightmare later. I have seen days go to waste because of Docker misconfigurations, problems with dependency versions, outdated dependencies not available anymore, bugs deep down in an unknown dependencies, and so on. Personally I want to build systems. I don't want to spend my day debugging all kinds of weird problems. The (old) WhatsApp teams sounds like a great workplace.
- hasperdi 5y ago
  These issues that you described can happen with any language, ask any programmer or system engineer with long enough experience they’ll have stories with weird problems. Regardless which language you pick, there will always be these kinds of issues.
  - Cthulhu_ 5y ago
    > These issues that you described can happen with any language While you're not wrong, I do strongly feel it depends on the language and architecture chosen. I'm a Go developer these days, and there's a big mindset to e.g. avoid dependencies, and for those dependencies to avoid including even more dependencies, keeping things fairly lightweight and with a low 'attack surface'. I mean I personally wouldn't mind a stricter type system and native enum support and some other things, but for now, I enjoy how basic it is, whilst avoiding the footguns that C/C++ brings.
- johnfn 5y ago
  > Personally I want to build systems. Sure. And so do I. Specifically, I don't want to rebuild systems that have already been built for me which I can use as dependencies. And I don't want to spend all my time troubleshooting bugs, either - I'll take a well-maintained repository on GitHub with thousands of users bug-testing it for me over some janky thing some guy on another team threw together a few quarters ago before he quit.
Traster 5y ago
I think one thing to consider is that WhatsApp is probably by nature a scalable application. It is in the end, simply a client app that sends data to other client apps. So by nature, you scale in parallel very well, there are no 1-all communications. So unlike something like facebook where the data all needs to go to a central server and be polled by an unknown (but potentially arbitrary large) number of clients, in this case you have a huge number of point to point communications. It also means your server application can be largely stateless. I'm not saying that it's an easy problem to solve, but it is a much easier problem to solve than a lot of other applications.
- crate_barre 5y ago
  To add to this, not only is it a simpler problem, but they stood on the shoulders of giants that already solved it. They used ejabberd, a full-featured xmpp server written in Erlang. Imagine half your job already done for you. Granted, they heavily customized it by this point, but they did a great job picking the right off the shelf solution to begin with.
- lampe3 5y ago
  I fully agree here. Sending messages to a known (small) number of people is a solved problem for a long time. Email for example. With WhatsApp, it is even easier than email because WhatsApp is a closed system, and you can optimize. They used an uncommon setup? Okay cool. How does this help me? or any other company? I'm not sure what I should learn from this story? The only thing is: Feature creep is bad.
- Tepix 5y ago
  Why do you assume that WhatsApp uses point-to-point connections? My assumption is that most devices cannot talk to each other directly and require central servers.
  - hbn 5y ago
    I could be wrong but I don't think he was saying "point-to-point communication" in the technical aspect of the term (i.e. peer-to-peer). Rather, just making the point that they're not storing tons of data on their server that has to be polled on demand by an unknown amount of clients forever. Even if there's a server between the users, once they receive a message and pass it onto whoever necessary, their job is done and they don't need to worry about that data any more.
M0r13n 5y ago
I'm honestly not that surprised about the size. In my experience, efficiency and the number of engineers have not necessarily correlated. For example, I used to work for a company with about 500 employees (relatively large by German standards) and now work for a competing company in the same industry with less than 150 employees. The core engineering team only consists of a dozen engineers. The output is noticeably larger. The number of new features and especially the stability is significantly better. It is noticeable that due to the small team, each project has one or two "heroes". They are responsible for most of the commits and know the system inside out. These "heroes" have been working on the same or similar systems for over 30 years in some cases and have experienced a lot in their careers. As such, they can bring experience and advice that is just out of reach for others. Given the choice, I would always prefer a small team over a large team.
- yodsanklai 5y ago
  > Given the choice, I would always prefer a small team over a large team. I think that's one of the conclusion of the "mythical man-month" book.
- altacc 5y ago
  Having worked in a range of companies, there is definitely an inverse correlation between company size and speed, especially for individual teams. At 500 people you're well into enterprise level organisation, where a lot of processes are put in place to minimize the amount of damage a single person can do to the organisation. That also puts a speed limit on those efficient and knowledgeable contributors. Some companies spend a lot of time & energy trying to implement operating models that try to overcome this but actually end up putting in place more decision processes & oversight. Sometimes the real solution is to reduce control and let the efficient teams lead the way. But then what would all that management do to fill their days...
  - mywittyname 5y ago
    I think this is more correlated to maturity than size. It's just that size is highly correlated to maturity. I've been in large companies where I was the admin/owner of their AWS account. And I've worked at companies with 200 people where I have to submit tickets to get anything created in AWS. The major difference was how mature the product was.
topdancing 5y ago
You can very easily build your own WhatsApp server today by simply deploying https://www.ejabberd.im/ https://www.ejabberd.im/ on a box and using https://conversations.im/ https://conversations.im/ on an Android phone.
- dopamean 5y ago
  Comments like this always remind me of that old comment on here about how dropbox could easily be recreated with fsync or something.
  - 1f60c 5y ago
    Here it is, for the uninitiated: https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
- southerntofu 5y ago
  Why is this being downvoted? Whatsapp is notoriously "just" a highly-optimized fork of ejabberd [0]. ejabberd is written in Erlang/OTP which enables quasi-horizontal scaling and amazing failure recovery, along with incredible uptime. What's different about Whatsapp compared to a traditional XMPP experience is: - it uses a single centralized server, without federating - it uses phone numbers as identifiers instead of nicknames - it has a tightly integrated (closed-source) client So you can not only "build your own WhatsApp", but also federate with others doing the same, which is pretty cool and prevents a single actor from abusing the whole network. [0] https://www.infoq.com/presentations/whatsapp-scalability/ https://www.infoq.com/presentations/whatsapp-scalability/
  - AshamedCaptain 5y ago
    Doesn't anybody remember how literally Whatsapp's founder wrote on one of the ejabberd mailing lists with something like "I just installed the server, please help me configure authentication" or the like? Retrospectively that has always colored my perception of how relevant technical aspects are for a successful startup (i.e. not much). Marketing (and viral, user-capturing & monopolist practices) are (sadly) much more important.
LudwigNagasena 5y ago
Well, what number would one expect? Surely, the workforce doesn’t scale linearly with users.
zibzab 5y ago
Remember the fast-cheap-secure triangle of product development? Now guess which one they skipped
- tim333 5y ago
  I'm not sure the development was super fast. They also kept it fairly simple.
- southerntofu 5y ago
  The client is another story, but for the server they forked ejabberd which is a reliable, established XMPP server. So they probably had "fast-cheap-secure" on the server-side, because 6 years had been spent ironing out ejabberd bugs before the first version of Whatsapp came out. My understanding is that vulns in Whatsapp so far were client-side, but i'm interested if i missed some on the server side.
  - Zash 5y ago
    They had some homebrew encryption and authentication before, see eg https://blog.thijsalkema.de/blog/2013/10/08/piercing-through-whatsapp-s-encryption/ https://blog.thijsalkema.de/blog/2013/10/08/piercing-through... https://blog.thijsalkema.de/blog/2013/10/08/piercing-through-whatsapps-encryption-2/ https://blog.thijsalkema.de/blog/2013/10/08/piercing-through...
    - southerntofu 5y ago
      > homebrew encryption What could possibly go wrong? :) Thanks for the links!
varispeed 5y ago
I bet these engineers got only a tiny tiny fraction of the billions of revenue. It always fascinates me how in the engineers' mind it reconciles that their work brings corporations they work for billions and yet they have to rent a tiny apartment for a substantial portion of their salary and otherwise lead a pretty average life. Then going to lavish offices witnessing how the money they could make use of is wasted on vanity.
- southerntofu 5y ago
  That's the founding principle of capitalism: private property. People living someplace don't own it, and people working some field don't own it. Some "owner" owns the land and means of production and extracts value from people doing the actual work. Yes, it's a deeply broken system. A slightly better system revolves around self-organized workers coop where everyone gets equal pay and there are no shares to hold (or everyone owns the same amount). An even better system abolishes money and private property so that people can live meaningful/useful lives without worrying about imaginary numbers ruining their entire existence.
xchip 5y ago
...the other 450 engineers are just for spying and tracking people
up6w6 5y ago
This just makes more clear that we didn't try enough creating an open-source instant messaging app that respects our privacy as it should be. It's probably much more difficult to do the same thing as WhatsApp did because of the dependency we created around it but I think we just need the right initiative to do so.
- baby 5y ago
  There are several hard problems linked to what people really mean when they say privacy.
- snarf21 5y ago
  It is hard to get the general public to care about privacy when they post pictures of every meal and share every thought and emotion they have with no filter.
  - jodrellblank 5y ago
    It’s hard to get the general public to care about inferior things pushed on them by patronising people sneering at them. How is a photo of some food important to privacy enough to be the first thing you teach for? Seems more like punching down at people you see as inferior because photographing meals is “so lower class, right guys?”.
12ian34 5y ago
The many comments alluding to the technical challenge of building a chat app with 50 engineers being relatively easy are forgetting that scaling said chat app to 1 billion users is absolutely not easy and a function of more than just the engineering quality and technical challenge. There's MUCH more to a successful product than engineering...
- tantalor 5y ago
  ... such as? Don't leave us hanging!
  - dunefox 5y ago
    I'd imagine getting the users in the first place. That has nothing to do with engineering quality and scaling up, though.
jimbojet 5y ago
User count is fairly meaningless. How much was that in daily active / weekly active users? How many of those were bot / spammer accounts being “seasoned” to then sell to abusers?
shmatt 5y ago
There was a joke, or maybe anecdote, going around the Israeli high tech industry when Google purchased Waze When it came time to get an overview of the codebase being purchased. The Google Maps car navigation iOS team had over 200 engineers. The Waze iOS team had 2
- lightning19 5y ago
  off topic but I recently started working with an Israeli tech company and I'm amazed by the culture they have, what about Israel produces such high levels of entrepreneurship?
- Jensson 5y ago
  > The Google Maps car navigation iOS team had over 200 engineers. I don't really believe that. I worked at Google and the frontend teams were usually just a couple of engineers per target client, I doubt any projects had 200 entire engineers just for a single target client. Not even big projects like gmail has nearly that many. Edit: If I were to guess Google maps would have 200 engineers in total, or at least 200 figure would be the entire team of frontend engineers for every target client, and then many more working in other roles. 1 engineer per million users is a pretty good rule of thumb for Google products, and a large majority of those will work on backend systems.
JohnJamesRambo 5y ago
Remember this when people talk about Tether and how it can’t be real since it has X employees.
- danrocks 5y ago
  Tether can't be real because it's a scam, not because it has X employees.
streamofdigits 5y ago
Beware of survivorship bias. Team does X, does not achieve scale, nobody knows about it Team does X, achieves scale, X is the new sliced bread
- papito 5y ago
  The first team does not achieve scale because they are still building microservices. The word of the day is "lean".
  - streamofdigits 5y ago
    intuitively I would agree, but the point of my comment was that intuition might be a poor guide if we want to understand what kind of team and software architectures "succeed". if you rewind some years microservices was also the word of the day (and probably there were some iconic projects picked to justify...)
unsungNovelty 5y ago
They not only used FreeBSD for their servers, their CEO at the time Jan Koum donated $1 million dollars to FreeBSD foundation too. Talk about giving back! The only thing they failed at was making money. I really wished they had. They knew what Whatsapp meant to the world. Look at WA now. https://freebsdfoundation.blogspot.com/2014/11/freebsd-foundation-announces-generous.html https://freebsdfoundation.blogspot.com/2014/11/freebsd-found...
- moralestapia 5y ago
  >The only thing they failed at was making money. Huh? They're all million/billionaires now. That part of the plan seems to have worked.
  - btbuildem 5y ago
    Because they sold to FB. I think parent meant "they failed to make money on their own"
    - moralestapia 5y ago
      You probably don't know the story. Selling was part of the plan all along. Whatsapp didn't even try to make money on their own.
      - unsungNovelty 5y ago
        The founders hearts were in the right place. Brian Acton created Signal Foundation and Jan Koum like I said also have done good things. They also tried the paid version for a while if you remember. And they were also dead against ads on the app. Unless I missed something or if I am wrong somewhere (At which point I would love to be corrected), Selling was the plan all along doesn't seem like a right observation.
        moralestapia 5y ago
        >The founders hearts were in the right place. Brian Acton created Signal Foundation and Jan Koum like I said also have done good things. I'm not saying they are bad people, and selling is not a bad thing either. I'm just pointing out that selling was the plan all along. The WhatsApp story that gets told regularly (like in this instance) is a story of how such few employees and such small infrastructure is enough to build a massive product. You know what that translates to? Low opex. If they wanted to, WhatsApp would've been quite profitable if they wanted but instead they chose to sell to the highest bidder. Again, that's not a bad thing, but also, I'm sure there were PLENTY of people that were interested in buying a billion user platform with 99% user retention. Why did they sold to Facebook? Only they know, but back then Facebook already had a tarnished reputation, so they definitely didn't do it "because of their mission and values".
papito 5y ago
Because complexity kills. Something that has yet to be learned by the new generation of developers. While disciples of FAANG create a gazillion microservices that they spend most of their time debugging in production ("observability"), smaller teams just get shit done. Since "monolith" has become a dirty word, it would be instructive to remember that some of the major companies out there are still light on distributed systems, which we all knew even back in the olden days of the internet were an absolute killer of efficiency and productivity. Yes, "microservices" is just another word for it - believe it or not, there was life before Node and Docker. Whatsapp, Dropbox, Instagram, StackOverflow - all existed or still exist as one easy-to-maintain application. But go ahead, "be like Google".
lumost 5y ago
My 2cents. Messaging is a weird app to focus on mau/dau. There have been many messaging protocols which scale to hundreds of millions of users, and there is prior art of large not particularly valuable chat systems like aim, gchat etc. No one is going to use a consumer messaging system which charges money or shows ads. Which brings to mind substantial questions on the business model others are using the messaging platform for e.g. data collection.
- closeparen 5y ago
  WhatsApp at this stage had a small signup fee.
pyuser583 5y ago
Scaling a single, discreet, product shouldn’t take a ton of engineers.
dano 5y ago
I looked at it this way. The thing most engineers, hero's or not, want to work on is new projects and to build products customers enjoy. The last thing they want to work on is maintenance. In my last personally owned company, small as we were, the engineering team and I documented how things worked as part of what we did every day. We found that by having the inner workings documented so that anyone in the team could effect a repair or update a piece of code freed us to work on new products and services. Just having the system continually documented freed our minds and kept us mostly happy (everyone has a challenge from time to time) and productive. The documentation, not that it really matters, was done in a MoinMoin wiki. It was written by engineers, for engineers. We didn't stop marketing or sales or anyone else from reading it, but the target audience was the engineering and operation staff thus no one had to over-explain the vocabulary of our architecture.
- KennyFromIT 5y ago
  How did you manage to keep the documentation from going stale or efforts being duplicated?
- arthur_sav 5y ago
  > Just having the system continually documented In my experience this is the hardest part. Can't count how many times my team started documenting and then slowly, for one or the other reason, the documentation became outdated. The cycle is: Meetings for communication -> becomes too cumbersome and time consuming -> implement policy to document things -> everyone start documenting (for a month or two) -> docs are not up-to-date because people forget/leave etc.. -> go back to meetings
  - castlecrasher2 5y ago
    In my experience it's a culture thing. When I learned to love Jira/Confluence was when the product team conducted meetings directly from Jira and during task discussion almost always asked us if tasks and documentation were done. It could have been really annoying but the product lead was extremely empathetic and that could be felt in any task discussion, and his attitude made me want to do it instead of making it feel like busy work like it has in other companies.
dopamean 5y ago
I thought what allowed them to get to that many users was that they were on so many platforms and that they used a large team of remote engineers in Europe to do that.
- EVa5I7bHFq9mnYK 5y ago
  What allowed them to get to that many users was 1)mobile operators charged an arm and leg for text messages, especially in developing countries 2)whatsapp did not require registration with user name and password, as their competitors (Skype, ICQ and others).
bravetraveler 5y ago
Queue endless business majors wanting to copy them
rdsubhas 5y ago
My experience is, # of engineers is directly related to # of product people, which is in turn related to quantity (not quality) of ideas that the founders/leaders are desperate to try out. Higher quantity of ideas == More engineers. Higher quality of ideas == Fewer engineers. WhatsApp success is not about what WhatsApp did, it's about everything they didn't do. No desktop, no browser (until after reaaaaaaallly long), no account creation (your phone number is enough, thanks), no hidden ad engine, no hundreds of stupid features that product people stuff down in the name of experimentation. There are some leaders who want anything and everything: Throw 100 darts in the wall. Leave what sticks (don't touch or nurture it). Go back to throwing 100 more. The other guy does that? We want that too. Mobile? Yes. Desktop? Yes. Browser? Yes. Different affiliate channels? Yes. Custom features for B2B partners? Yes. Merchandizing? Yes. Build a mini Ad engine? Yes. Multi user profile management systems? Yes. Yes, yes, yes, every requirement is a go. The crucial difference is: What happens when you have a restricted pool of people, but insist on doing 100s of things? Some leaders simply treat this as an "org structure" problem, not a problem of focus. They slice and dice the org until the 30 people become split into teams of 2-3 with a manager for each, running into coordination hell on each step – just so that more quantity of features can be seen on paper. This is how most startups are, except for those rare few which don't "talk" about focusing, but actually focus.
- jeltz 5y ago
  As an end user I think WhatsApp is a terrible product for some the reasons you mentioned (tied to phone number, no desktop client) but I cannot deny that it was the smart thing to do from a business perspective.
- chpmrc 5y ago
  > no hundreds of stupid features that product people stuff down in the name of experimentation. Amen to that!
- tinus_hn 5y ago
  Ultimately though WhatsApp ended up with a billion users and 0 revenue.
thrower123 5y ago
Fifty engineers almost sounds like too many, if you aren't using most of them to develop internal frameworks and reinvent wheels. You'd have an Android team, an iOS team, a couple teams working on core APIs and infrastructure. Ideal team size is about five.
mikewarot 5y ago
The users are all handled through a common set of code, I don't see why its remarkable that it takes 50 engineers to keep things going. Customer support, I can see that eating manpower, but not keeping the code running on a few racks of servers.
short12 5y ago
That sounds about right, even slim I've honestly never understood how faang companies can have such incredible numbers of devs when the products basically move at a snails pace for years now
Pensacola 5y ago
Only? Sound like plenty.
steve_adams_86 5y ago
It crossed my mind while considering this that a figure like this doesn’t mean the same as it would have pre-cloud infrastructure. My team is fairly experienced but not so much in infrastructure. Even so, we build and deploy on Google cloud with fairly excellent scalability and ease. Ten years or longer ago I suspect the story would be different - the infrastructure and scale constraints could have pinned us at times. We would have needed to hire more to support that effort. And what I’m getting to is that yes, they had 50 engineers in their pay roll, but many more supporting various SaaS products and infrastructure they use and pay for. Not to say this isn’t impressive still, I think it’s a great thing and worth aspiring to. I love small teams.
- Jensson 5y ago
  Whatsapp used bare metal servers. It isn't hard to do, just that people no longer learn to do it. > Ten years or longer ago I suspect the story would be different Whatsapp was built 12 years ago.
  - steve_adams_86 5y ago
    I assumed they did their largest scale ups more recently, though. I could be wrong!
- toast0 5y ago
  > And what I’m getting to is that yes, they had 50 engineers in their pay roll, but many more supporting various SaaS products and infrastructure they use and pay for. I was at WhatsApp from 2011 to 2019. Hardware was outsourced to SoftLayer, of course. I think they had a sizable operation, but if you move that in house, you've got a team of 4ish for 24x7 colo work (we were only in one location for an embarassing amount of time), plus a networking person, and maybe a manager. We certainly got better service with SoftLayer's larger team including ease of getting more servers, but we would have had better visibility in house, so it's a tradeoff. Softlayer's staff was probably smaller than the combined staff if all of their major customers in-sourced, anyway. Other than that, we used multiple providers for SMS and voice verification; some of whom have a lot of staff. You need more staff for this, IMHO, because in addition to 24x7 coverage, you also need to arrange connectivity with carriers in all timezones and all languages. But some of the companies we used seemed pretty small, so I dunno. We were using Zendesk in our customer service system after growing out of Gmail and before in-sourcing it after aquisition, and Dyn for DNS (which doesn't take much staff to insource). What other SaaS do you think we used? Do you want to count the OS app stores and push services? Maybe payments while that lasted? G Suite for corporate email when forwarding to personal mail became too silly? We self-hosted our code repository and bug tracker.
dang 5y ago
Discussed in the past: Why WhatsApp Only Needs 50 Engineers for Its 900M Users - https://news.ycombinator.com/item?id=10225096 https://news.ycombinator.com/item?id=10225096 - Sept 2015 (248 comments)
johnisgood 5y ago
I could scale to billions of users as a one-man engineer, but I would need a marketing team. :D What I am trying to say is that I could make a WhatsApp-equivalent relatively quickly, alone, yes, but that is definitely not enough to gain billions of users.
- MomoXenosaga 5y ago
  They were at the right place at the right time. But people want to believe it is tech not luck... There is a webshop in my country that copied Amazon in the early 2000s and today they are beating Amazon.
JoeCoo7 5y ago
WhatsApp has to be one of those most easily scalable projects to work on. You have the freedom to divide chatrooms across different databases/shards. Same goes for notifications, where users can be divided since they're all isolated. The hardest part is just coordinating all the moving pieces in a way that can grow well, and that mostly just involves avoiding pitfalls in your design that might bottleneck.
throwaway1777 5y ago
And now WhatsApp has hundreds of engineers. Take that to mean whatever you want.
seoaeu 5y ago
It is silly to focus on number of engineers rather than on total cost. Any company should keep hiring more engineers until the additional salary expense exceeds the decreased operating cost/increased revenue that engineer generates
28194608 5y ago
Dear whatsapp devs please disable the feature which shows how much time and when to EVERYONE in the world. TY.
heywherelogingo 5y ago
This site is getting quite repetitive. I think the balance of News to things of interest has gotten quite out of whack. This article is not news, and is also in recent memory so not an interesting re-visit. Is HN average age decreasing? The content character has changed in recent years. Something's changed/changing.
- akkpxl 5y ago
  I thought I was the only one who felt this way. Almost every day I see the same "style" of articles that isn't news or that interesting. One thing I've noticed is almost every day I see an article explaining why remote work is better and/or how tech interviews are broken, both of which are popular opinions a lot of people on this site share. Maybe it's just me though
- bobberkarl 5y ago
  Started visiting hn 4/5 years ago, i agree. The content/discussion quality decreased substantially the past 2 years.
mkl95 5y ago
Ultimately, how many engineers a company has is not that relevant when it comes to scaling. A small team of product-oriented engineers with a devops mentality can easily develop, maintain, and continuously deploy a considerably complex application. A similarly complex project in the hands of a larger sales driven company often resembles a spaghetti ball, and is held together with rotting shell scripts and unproductive manual processes.
chatman 5y ago
This is misleading. Likely, those 50 engineers were standing on the shoulders of the giants, i.e. those open source maintainers who build wonderful value for everyone out there.
- ratww 5y ago
  Does this really matter? Companies with 100, 500, 1000, 5000, etc employees often stand exactly in the same position and use the same kinds of tools.
jokethrowaway 5y ago
While what you says about focus is important, I've seen plenty of focused startups die. WhatsApp had success because they executed well enough at the right time. I hate WhatsApp with passion because of its limits and that horrible web interface that needs your phone connected in order to work. I know plenty of people (me included) who stopped talking with some groups of people because they didn't tolerate WhatsApp. It's mediocre software at best, but because of the network effect and first mover advantage, that was enough.
canadianfella 5y ago
It’s a messaging app. Why would you need more than 50 engineers?

636 points by unripe_syntax 5y ago

Some of these points are a bit suspicious, and I bet don't tell "the whole story" the way they're intended. For instance, the statement about not investing in automation unless absolutely necessary or the point about hot-swapping Erlang code. Would be good to actually hear from one of these engineers for a discussion on the paint points of working on WhatsApp back then.

tdrdt 5y ago
Automation can be good, but will also be another system to support. I think this XKCD applies: https://xkcd.com/1205/ https://xkcd.com/1205/
- bryanrasmussen 5y ago
  If you shave off 1 second I have a hard time believing that 1 second will not just be swallowed up by the entropy of coffee breaks.
- bradhe 5y ago
  I'd like to see what their criteria for "absolutely necessary" is--I bet it doesn't fall on the right side of that table.

amelius 5y ago

Chat is the number 1 toy problem that is explained in every Erlang book out there. Therefore, honestly I'm not really surprised to see 50 engineers capable of building a product around it.

TacticalCoder 5y ago

What is amazing to me is that they did scale to 400 million monthly active users with less than 50 engineers... In 2014. That is: on hardware from seven years ago and older. Which is an eternity in tech. Since then we've seen incredible new hardware come out (EPYC 2 in 2018 for example). One can wonder: up to how many MAU could a service scale today, on today's hardware, with a small team? And what about tomorrow's hardware... This is fascinating.

hnarn 5y ago
Seven years can be an eternity in tech, but is it an eternity counting from 2021? I mean, AWS launched in 2006, Netflix expanded to Europe in 2012, Docker had their initial release in 2013 and so on, so none of these major events are that "new" anymore. Obviously things have changed (a lot) during the past seven years, but have they changed as much as they did between 2007 and 2014? What I'm saying is that for the general case of "scaling with few engineers", it seems to me that the tools available matter a lot more than the specifics of what hardware is available at the time. Launching an international service "from scratch" with a small team and a modest budget just wouldn't have been possible 20 years ago, but not because CPUs were too slow.

adrianlmm 5y ago

For a similar project Google would use 250 enginers, 200 of those would be diversity hires and the rest for the real work.

tiffanyh 5y ago

DuckDuckGo does around 35B searches per year with 160 employees. https://duckduckgo.com/traffic https://duckduckgo.com/traffic Not the same efficiency but it goes to show how certain domains and technology stacks can scale exceedingly well. Plus in DDG case I believe they outsource a lot of their tech to 3rd parties.

jokoon 5y ago

I remember reading they worked very hard to make their app available on many, many types of phones. Which is weird because they regularly announce they drop support for certain phone OS.

nnx 5y ago
It made sense years ago... before the iOS/Android duopoly we have now.
toast0 5y ago
I was there from 2011 to 2019. We had pretty clear numerical guidelines for continuing support of a platform or OS. If the numbers are close, things like upgradability or difficulty of support make a difference: if all the devices running OS 1.1 can actually be upgraded to 1.2 and 1.2 fixes important things, then dropping support for OS 1.1 can happen with more users still there. If builds need to be signed by the OS maker and they won't sign any more builds, user count doesn't matter (sorry Nokia S40 and iPhone 3G). We did make some longer lasting builds for platforms we were ending support for, but there needs to be an end. Older clients don't support newer features and we didn't want to have a network where that was the status quo. I don't know what the current expirations are, but mandatory updates a couple times a year helps keep the feature support similar.

kosolam 5y ago

WhatsApp is a spy app of Facebook nowadays

oblio 5y ago
That was its fate anyway. Some things just can't really make money unless they do it in scummy ways (or we haven't found a viable alternative or a way to protect them). Public instant messengers, public forums, public file hosting, public image hosting.
- shawabawa3 5y ago
  With 50 employees their "please pay $1/year" model might have actually made enough revenue
  - oblio 5y ago
    Nobody I know paid it even back then. That's why they used it, because it was free. It's like Microsoft Office these days. If your home license expires they keep threatening you they'll disable it. For years and years.
  - jmrm 5y ago
    I paid totally happy with the service they gave me, and I would pay monthly if that would mean having total privacy.
- alanfranz 5y ago
  Whatsapp used to charge 1 eur or usd per year per user. Assuming a 20% vat, that’s $800M in yearly revenue. If you spend $500k per engineer per year, you’re still left with $775M to handle infrastructure and all other expenses. Whatsapp would have been sustainable without fb. That’s the regret.
  - harpratap 5y ago
    You are very naively assuming it would actually reach 1B users if it was paid.

seanhandley 5y ago

I wonder if WhatsApp could be considered the most successful user-facing application with an Erlang backend.

polyomino 5y ago
What other candidates are there?
- peoplefromibiza 5y ago
  while WhatsApp is the largest I am aware of (it's one of the leading platforms Worldwide anyway), there are quite a few, sometimes in the form of Elixir: - Discord - Pinterest - Spotify - Klarna - Riot games (the messaging service) - PepsiCo (for their billion making e-commerce) - Toyota (for the connected platform)
- bradhe 5y ago
  Facebook Messenger, if I remember correctly, was also originally authored in Erlang.
  - mewpmewp2 5y ago
    Discord also if I remember correctly?
    - d3nj4l 5y ago
      Discord was written in Elixir! https://blog.discord.com/scaling-elixir-f9b8e1e7c29b https://blog.discord.com/scaling-elixir-f9b8e1e7c29b
      - mewpmewp2 5y ago
        Does it mean it's technically also based on Erlang?
        5y ago

bjarneh 5y ago

This article does very little to explain how they did that. It basically states the type of tech they used (Erlang, FreeBSD and SoftLayer), and something about not trying to over engineer stuff. The title also makes it sound like 500 engineers would make it easier to scale up WhatsApp, which does not make too much sense either.

Cthulhu_ 5y ago
The article is honestly kinda low on details, depth or new insights, but at least it links to some sources.
willejs 5y ago
Theres a high scalability post from years ago that has a bit more information. http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-to-nearly-500-million-users-11000-cores-an.html http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-...
- bjarneh 5y ago
  Thanks, it does seem like choosing a language that was made for doing things in parallel to begin with was a good idea. To be honest, I've always felt that programming in functional languages creates less bugs, but perhaps that's just my imagination.
  - dunefox 5y ago
    It certainly creates fewer bugs of a specific nature, depending on the language.
- vadfa 5y ago
  >19B messages in & 40B out per day whatsapp being a closed system I don't understand how this disparity is possible.
  - the_other 5y ago
    Most people are in group chats.

corobo 5y ago

Is there even a correlation between scale and engineers, other than scale = VC money = hire more cause we can afford it?

Ekaros 5y ago
I would expect there to be certain scaling. Specially when you move to more severs and more redundancy. Cloud solves part, but still not everything. If single person can support 10k or 100k users. You might still need more than that one to move to million or hundred million. Also some features like language support and such get more complicated when you want to widen userbase from just English speaking ones.
- corobo 5y ago
  > If single person can support 10k or 100k users I asked if there was a correlation here, using an if statement as an answer doesn't answer it haha. You're now asking the same question I was, just in-line

tdrdt 5y ago

"The simpler product makes it much easier to maintain and scale." This is good news for the developers, the product owners and the customers. I still can't understand why today people choose a (Javascript) stack of build systems, ton of dependencies, and all kinds of exotic tech that is the latest and most hyped. As a developer you need to support this in the future. It might be nice to build today but it will be a nightmare later. I have seen days go to waste because of Docker misconfigurations, problems with dependency versions, outdated dependencies not available anymore, bugs deep down in an unknown dependencies, and so on. Personally I want to build systems. I don't want to spend my day debugging all kinds of weird problems. The (old) WhatsApp teams sounds like a great workplace.

hasperdi 5y ago
These issues that you described can happen with any language, ask any programmer or system engineer with long enough experience they’ll have stories with weird problems. Regardless which language you pick, there will always be these kinds of issues.
- Cthulhu_ 5y ago
  > These issues that you described can happen with any language While you're not wrong, I do strongly feel it depends on the language and architecture chosen. I'm a Go developer these days, and there's a big mindset to e.g. avoid dependencies, and for those dependencies to avoid including even more dependencies, keeping things fairly lightweight and with a low 'attack surface'. I mean I personally wouldn't mind a stricter type system and native enum support and some other things, but for now, I enjoy how basic it is, whilst avoiding the footguns that C/C++ brings.
johnfn 5y ago
> Personally I want to build systems. Sure. And so do I. Specifically, I don't want to rebuild systems that have already been built for me which I can use as dependencies. And I don't want to spend all my time troubleshooting bugs, either - I'll take a well-maintained repository on GitHub with thousands of users bug-testing it for me over some janky thing some guy on another team threw together a few quarters ago before he quit.

Traster 5y ago

I think one thing to consider is that WhatsApp is probably by nature a scalable application. It is in the end, simply a client app that sends data to other client apps. So by nature, you scale in parallel very well, there are no 1-all communications. So unlike something like facebook where the data all needs to go to a central server and be polled by an unknown (but potentially arbitrary large) number of clients, in this case you have a huge number of point to point communications. It also means your server application can be largely stateless. I'm not saying that it's an easy problem to solve, but it is a much easier problem to solve than a lot of other applications.

crate_barre 5y ago
To add to this, not only is it a simpler problem, but they stood on the shoulders of giants that already solved it. They used ejabberd, a full-featured xmpp server written in Erlang. Imagine half your job already done for you. Granted, they heavily customized it by this point, but they did a great job picking the right off the shelf solution to begin with.
lampe3 5y ago
I fully agree here. Sending messages to a known (small) number of people is a solved problem for a long time. Email for example. With WhatsApp, it is even easier than email because WhatsApp is a closed system, and you can optimize. They used an uncommon setup? Okay cool. How does this help me? or any other company? I'm not sure what I should learn from this story? The only thing is: Feature creep is bad.
Tepix 5y ago
Why do you assume that WhatsApp uses point-to-point connections? My assumption is that most devices cannot talk to each other directly and require central servers.
- hbn 5y ago
  I could be wrong but I don't think he was saying "point-to-point communication" in the technical aspect of the term (i.e. peer-to-peer). Rather, just making the point that they're not storing tons of data on their server that has to be polled on demand by an unknown amount of clients forever. Even if there's a server between the users, once they receive a message and pass it onto whoever necessary, their job is done and they don't need to worry about that data any more.

M0r13n 5y ago

I'm honestly not that surprised about the size. In my experience, efficiency and the number of engineers have not necessarily correlated. For example, I used to work for a company with about 500 employees (relatively large by German standards) and now work for a competing company in the same industry with less than 150 employees. The core engineering team only consists of a dozen engineers. The output is noticeably larger. The number of new features and especially the stability is significantly better. It is noticeable that due to the small team, each project has one or two "heroes". They are responsible for most of the commits and know the system inside out. These "heroes" have been working on the same or similar systems for over 30 years in some cases and have experienced a lot in their careers. As such, they can bring experience and advice that is just out of reach for others. Given the choice, I would always prefer a small team over a large team.

yodsanklai 5y ago
> Given the choice, I would always prefer a small team over a large team. I think that's one of the conclusion of the "mythical man-month" book.
altacc 5y ago
Having worked in a range of companies, there is definitely an inverse correlation between company size and speed, especially for individual teams. At 500 people you're well into enterprise level organisation, where a lot of processes are put in place to minimize the amount of damage a single person can do to the organisation. That also puts a speed limit on those efficient and knowledgeable contributors. Some companies spend a lot of time & energy trying to implement operating models that try to overcome this but actually end up putting in place more decision processes & oversight. Sometimes the real solution is to reduce control and let the efficient teams lead the way. But then what would all that management do to fill their days...
- mywittyname 5y ago
  I think this is more correlated to maturity than size. It's just that size is highly correlated to maturity. I've been in large companies where I was the admin/owner of their AWS account. And I've worked at companies with 200 people where I have to submit tickets to get anything created in AWS. The major difference was how mature the product was.

topdancing 5y ago

You can very easily build your own WhatsApp server today by simply deploying https://www.ejabberd.im/ https://www.ejabberd.im/ on a box and using https://conversations.im/ https://conversations.im/ on an Android phone.

dopamean 5y ago
Comments like this always remind me of that old comment on here about how dropbox could easily be recreated with fsync or something.
- 1f60c 5y ago
  Here it is, for the uninitiated: https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
southerntofu 5y ago
Why is this being downvoted? Whatsapp is notoriously "just" a highly-optimized fork of ejabberd [0]. ejabberd is written in Erlang/OTP which enables quasi-horizontal scaling and amazing failure recovery, along with incredible uptime. What's different about Whatsapp compared to a traditional XMPP experience is: - it uses a single centralized server, without federating - it uses phone numbers as identifiers instead of nicknames - it has a tightly integrated (closed-source) client So you can not only "build your own WhatsApp", but also federate with others doing the same, which is pretty cool and prevents a single actor from abusing the whole network. [0] https://www.infoq.com/presentations/whatsapp-scalability/ https://www.infoq.com/presentations/whatsapp-scalability/
- AshamedCaptain 5y ago
  Doesn't anybody remember how literally Whatsapp's founder wrote on one of the ejabberd mailing lists with something like "I just installed the server, please help me configure authentication" or the like? Retrospectively that has always colored my perception of how relevant technical aspects are for a successful startup (i.e. not much). Marketing (and viral, user-capturing & monopolist practices) are (sadly) much more important.

LudwigNagasena 5y ago

Well, what number would one expect? Surely, the workforce doesn’t scale linearly with users.

zibzab 5y ago

Remember the fast-cheap-secure triangle of product development? Now guess which one they skipped

tim333 5y ago
I'm not sure the development was super fast. They also kept it fairly simple.
southerntofu 5y ago
The client is another story, but for the server they forked ejabberd which is a reliable, established XMPP server. So they probably had "fast-cheap-secure" on the server-side, because 6 years had been spent ironing out ejabberd bugs before the first version of Whatsapp came out. My understanding is that vulns in Whatsapp so far were client-side, but i'm interested if i missed some on the server side.
- Zash 5y ago
  They had some homebrew encryption and authentication before, see eg https://blog.thijsalkema.de/blog/2013/10/08/piercing-through-whatsapp-s-encryption/ https://blog.thijsalkema.de/blog/2013/10/08/piercing-through... https://blog.thijsalkema.de/blog/2013/10/08/piercing-through-whatsapps-encryption-2/ https://blog.thijsalkema.de/blog/2013/10/08/piercing-through...
  - southerntofu 5y ago
    > homebrew encryption What could possibly go wrong? :) Thanks for the links!

varispeed 5y ago

I bet these engineers got only a tiny tiny fraction of the billions of revenue. It always fascinates me how in the engineers' mind it reconciles that their work brings corporations they work for billions and yet they have to rent a tiny apartment for a substantial portion of their salary and otherwise lead a pretty average life. Then going to lavish offices witnessing how the money they could make use of is wasted on vanity.

southerntofu 5y ago
That's the founding principle of capitalism: private property. People living someplace don't own it, and people working some field don't own it. Some "owner" owns the land and means of production and extracts value from people doing the actual work. Yes, it's a deeply broken system. A slightly better system revolves around self-organized workers coop where everyone gets equal pay and there are no shares to hold (or everyone owns the same amount). An even better system abolishes money and private property so that people can live meaningful/useful lives without worrying about imaginary numbers ruining their entire existence.

xchip 5y ago

...the other 450 engineers are just for spying and tracking people

up6w6 5y ago

This just makes more clear that we didn't try enough creating an open-source instant messaging app that respects our privacy as it should be. It's probably much more difficult to do the same thing as WhatsApp did because of the dependency we created around it but I think we just need the right initiative to do so.

baby 5y ago
There are several hard problems linked to what people really mean when they say privacy.
snarf21 5y ago
It is hard to get the general public to care about privacy when they post pictures of every meal and share every thought and emotion they have with no filter.
- jodrellblank 5y ago
  It’s hard to get the general public to care about inferior things pushed on them by patronising people sneering at them. How is a photo of some food important to privacy enough to be the first thing you teach for? Seems more like punching down at people you see as inferior because photographing meals is “so lower class, right guys?”.

12ian34 5y ago

The many comments alluding to the technical challenge of building a chat app with 50 engineers being relatively easy are forgetting that scaling said chat app to 1 billion users is absolutely not easy and a function of more than just the engineering quality and technical challenge. There's MUCH more to a successful product than engineering...

tantalor 5y ago
... such as? Don't leave us hanging!
- dunefox 5y ago
  I'd imagine getting the users in the first place. That has nothing to do with engineering quality and scaling up, though.

jimbojet 5y ago

User count is fairly meaningless. How much was that in daily active / weekly active users? How many of those were bot / spammer accounts being “seasoned” to then sell to abusers?

shmatt 5y ago

There was a joke, or maybe anecdote, going around the Israeli high tech industry when Google purchased Waze When it came time to get an overview of the codebase being purchased. The Google Maps car navigation iOS team had over 200 engineers. The Waze iOS team had 2

lightning19 5y ago
off topic but I recently started working with an Israeli tech company and I'm amazed by the culture they have, what about Israel produces such high levels of entrepreneurship?
Jensson 5y ago
> The Google Maps car navigation iOS team had over 200 engineers. I don't really believe that. I worked at Google and the frontend teams were usually just a couple of engineers per target client, I doubt any projects had 200 entire engineers just for a single target client. Not even big projects like gmail has nearly that many. Edit: If I were to guess Google maps would have 200 engineers in total, or at least 200 figure would be the entire team of frontend engineers for every target client, and then many more working in other roles. 1 engineer per million users is a pretty good rule of thumb for Google products, and a large majority of those will work on backend systems.

JohnJamesRambo 5y ago

Remember this when people talk about Tether and how it can’t be real since it has X employees.

danrocks 5y ago
Tether can't be real because it's a scam, not because it has X employees.

streamofdigits 5y ago

Beware of survivorship bias. Team does X, does not achieve scale, nobody knows about it Team does X, achieves scale, X is the new sliced bread

papito 5y ago
The first team does not achieve scale because they are still building microservices. The word of the day is "lean".
- streamofdigits 5y ago
  intuitively I would agree, but the point of my comment was that intuition might be a poor guide if we want to understand what kind of team and software architectures "succeed". if you rewind some years microservices was also the word of the day (and probably there were some iconic projects picked to justify...)

unsungNovelty 5y ago

They not only used FreeBSD for their servers, their CEO at the time Jan Koum donated $1 million dollars to FreeBSD foundation too. Talk about giving back! The only thing they failed at was making money. I really wished they had. They knew what Whatsapp meant to the world. Look at WA now. https://freebsdfoundation.blogspot.com/2014/11/freebsd-foundation-announces-generous.html https://freebsdfoundation.blogspot.com/2014/11/freebsd-found...

moralestapia 5y ago
>The only thing they failed at was making money. Huh? They're all million/billionaires now. That part of the plan seems to have worked.
- btbuildem 5y ago
  Because they sold to FB. I think parent meant "they failed to make money on their own"
  - moralestapia 5y ago
    You probably don't know the story. Selling was part of the plan all along. Whatsapp didn't even try to make money on their own.
    - unsungNovelty 5y ago
      The founders hearts were in the right place. Brian Acton created Signal Foundation and Jan Koum like I said also have done good things. They also tried the paid version for a while if you remember. And they were also dead against ads on the app. Unless I missed something or if I am wrong somewhere (At which point I would love to be corrected), Selling was the plan all along doesn't seem like a right observation.
      - moralestapia 5y ago
        >The founders hearts were in the right place. Brian Acton created Signal Foundation and Jan Koum like I said also have done good things. I'm not saying they are bad people, and selling is not a bad thing either. I'm just pointing out that selling was the plan all along. The WhatsApp story that gets told regularly (like in this instance) is a story of how such few employees and such small infrastructure is enough to build a massive product. You know what that translates to? Low opex. If they wanted to, WhatsApp would've been quite profitable if they wanted but instead they chose to sell to the highest bidder. Again, that's not a bad thing, but also, I'm sure there were PLENTY of people that were interested in buying a billion user platform with 99% user retention. Why did they sold to Facebook? Only they know, but back then Facebook already had a tarnished reputation, so they definitely didn't do it "because of their mission and values".

papito 5y ago

Because complexity kills. Something that has yet to be learned by the new generation of developers. While disciples of FAANG create a gazillion microservices that they spend most of their time debugging in production ("observability"), smaller teams just get shit done. Since "monolith" has become a dirty word, it would be instructive to remember that some of the major companies out there are still light on distributed systems, which we all knew even back in the olden days of the internet were an absolute killer of efficiency and productivity. Yes, "microservices" is just another word for it - believe it or not, there was life before Node and Docker. Whatsapp, Dropbox, Instagram, StackOverflow - all existed or still exist as one easy-to-maintain application. But go ahead, "be like Google".

lumost 5y ago

My 2cents. Messaging is a weird app to focus on mau/dau. There have been many messaging protocols which scale to hundreds of millions of users, and there is prior art of large not particularly valuable chat systems like aim, gchat etc. No one is going to use a consumer messaging system which charges money or shows ads. Which brings to mind substantial questions on the business model others are using the messaging platform for e.g. data collection.

closeparen 5y ago
WhatsApp at this stage had a small signup fee.

pyuser583 5y ago

Scaling a single, discreet, product shouldn’t take a ton of engineers.

dano 5y ago

I looked at it this way. The thing most engineers, hero's or not, want to work on is new projects and to build products customers enjoy. The last thing they want to work on is maintenance. In my last personally owned company, small as we were, the engineering team and I documented how things worked as part of what we did every day. We found that by having the inner workings documented so that anyone in the team could effect a repair or update a piece of code freed us to work on new products and services. Just having the system continually documented freed our minds and kept us mostly happy (everyone has a challenge from time to time) and productive. The documentation, not that it really matters, was done in a MoinMoin wiki. It was written by engineers, for engineers. We didn't stop marketing or sales or anyone else from reading it, but the target audience was the engineering and operation staff thus no one had to over-explain the vocabulary of our architecture.

KennyFromIT 5y ago
How did you manage to keep the documentation from going stale or efforts being duplicated?
arthur_sav 5y ago
> Just having the system continually documented In my experience this is the hardest part. Can't count how many times my team started documenting and then slowly, for one or the other reason, the documentation became outdated. The cycle is: Meetings for communication -> becomes too cumbersome and time consuming -> implement policy to document things -> everyone start documenting (for a month or two) -> docs are not up-to-date because people forget/leave etc.. -> go back to meetings
- castlecrasher2 5y ago
  In my experience it's a culture thing. When I learned to love Jira/Confluence was when the product team conducted meetings directly from Jira and during task discussion almost always asked us if tasks and documentation were done. It could have been really annoying but the product lead was extremely empathetic and that could be felt in any task discussion, and his attitude made me want to do it instead of making it feel like busy work like it has in other companies.

dopamean 5y ago

I thought what allowed them to get to that many users was that they were on so many platforms and that they used a large team of remote engineers in Europe to do that.

EVa5I7bHFq9mnYK 5y ago
What allowed them to get to that many users was 1)mobile operators charged an arm and leg for text messages, especially in developing countries 2)whatsapp did not require registration with user name and password, as their competitors (Skype, ICQ and others).

bravetraveler 5y ago

Queue endless business majors wanting to copy them

rdsubhas 5y ago

My experience is, # of engineers is directly related to # of product people, which is in turn related to quantity (not quality) of ideas that the founders/leaders are desperate to try out. Higher quantity of ideas == More engineers. Higher quality of ideas == Fewer engineers. WhatsApp success is not about what WhatsApp did, it's about everything they didn't do. No desktop, no browser (until after reaaaaaaallly long), no account creation (your phone number is enough, thanks), no hidden ad engine, no hundreds of stupid features that product people stuff down in the name of experimentation. There are some leaders who want anything and everything: Throw 100 darts in the wall. Leave what sticks (don't touch or nurture it). Go back to throwing 100 more. The other guy does that? We want that too. Mobile? Yes. Desktop? Yes. Browser? Yes. Different affiliate channels? Yes. Custom features for B2B partners? Yes. Merchandizing? Yes. Build a mini Ad engine? Yes. Multi user profile management systems? Yes. Yes, yes, yes, every requirement is a go. The crucial difference is: What happens when you have a restricted pool of people, but insist on doing 100s of things? Some leaders simply treat this as an "org structure" problem, not a problem of focus. They slice and dice the org until the 30 people become split into teams of 2-3 with a manager for each, running into coordination hell on each step – just so that more quantity of features can be seen on paper. This is how most startups are, except for those rare few which don't "talk" about focusing, but actually focus.

jeltz 5y ago
As an end user I think WhatsApp is a terrible product for some the reasons you mentioned (tied to phone number, no desktop client) but I cannot deny that it was the smart thing to do from a business perspective.
chpmrc 5y ago
> no hundreds of stupid features that product people stuff down in the name of experimentation. Amen to that!
tinus_hn 5y ago
Ultimately though WhatsApp ended up with a billion users and 0 revenue.

thrower123 5y ago

Fifty engineers almost sounds like too many, if you aren't using most of them to develop internal frameworks and reinvent wheels. You'd have an Android team, an iOS team, a couple teams working on core APIs and infrastructure. Ideal team size is about five.

mikewarot 5y ago

The users are all handled through a common set of code, I don't see why its remarkable that it takes 50 engineers to keep things going. Customer support, I can see that eating manpower, but not keeping the code running on a few racks of servers.

short12 5y ago

That sounds about right, even slim I've honestly never understood how faang companies can have such incredible numbers of devs when the products basically move at a snails pace for years now

Pensacola 5y ago

Only? Sound like plenty.

steve_adams_86 5y ago

It crossed my mind while considering this that a figure like this doesn’t mean the same as it would have pre-cloud infrastructure. My team is fairly experienced but not so much in infrastructure. Even so, we build and deploy on Google cloud with fairly excellent scalability and ease. Ten years or longer ago I suspect the story would be different - the infrastructure and scale constraints could have pinned us at times. We would have needed to hire more to support that effort. And what I’m getting to is that yes, they had 50 engineers in their pay roll, but many more supporting various SaaS products and infrastructure they use and pay for. Not to say this isn’t impressive still, I think it’s a great thing and worth aspiring to. I love small teams.

Jensson 5y ago
Whatsapp used bare metal servers. It isn't hard to do, just that people no longer learn to do it. > Ten years or longer ago I suspect the story would be different Whatsapp was built 12 years ago.
- steve_adams_86 5y ago
  I assumed they did their largest scale ups more recently, though. I could be wrong!
toast0 5y ago
> And what I’m getting to is that yes, they had 50 engineers in their pay roll, but many more supporting various SaaS products and infrastructure they use and pay for. I was at WhatsApp from 2011 to 2019. Hardware was outsourced to SoftLayer, of course. I think they had a sizable operation, but if you move that in house, you've got a team of 4ish for 24x7 colo work (we were only in one location for an embarassing amount of time), plus a networking person, and maybe a manager. We certainly got better service with SoftLayer's larger team including ease of getting more servers, but we would have had better visibility in house, so it's a tradeoff. Softlayer's staff was probably smaller than the combined staff if all of their major customers in-sourced, anyway. Other than that, we used multiple providers for SMS and voice verification; some of whom have a lot of staff. You need more staff for this, IMHO, because in addition to 24x7 coverage, you also need to arrange connectivity with carriers in all timezones and all languages. But some of the companies we used seemed pretty small, so I dunno. We were using Zendesk in our customer service system after growing out of Gmail and before in-sourcing it after aquisition, and Dyn for DNS (which doesn't take much staff to insource). What other SaaS do you think we used? Do you want to count the OS app stores and push services? Maybe payments while that lasted? G Suite for corporate email when forwarding to personal mail became too silly? We self-hosted our code repository and bug tracker.

dang 5y ago

Discussed in the past: Why WhatsApp Only Needs 50 Engineers for Its 900M Users - https://news.ycombinator.com/item?id=10225096 https://news.ycombinator.com/item?id=10225096 - Sept 2015 (248 comments)

johnisgood 5y ago

I could scale to billions of users as a one-man engineer, but I would need a marketing team. :D What I am trying to say is that I could make a WhatsApp-equivalent relatively quickly, alone, yes, but that is definitely not enough to gain billions of users.

MomoXenosaga 5y ago
They were at the right place at the right time. But people want to believe it is tech not luck... There is a webshop in my country that copied Amazon in the early 2000s and today they are beating Amazon.

JoeCoo7 5y ago

WhatsApp has to be one of those most easily scalable projects to work on. You have the freedom to divide chatrooms across different databases/shards. Same goes for notifications, where users can be divided since they're all isolated. The hardest part is just coordinating all the moving pieces in a way that can grow well, and that mostly just involves avoiding pitfalls in your design that might bottleneck.

throwaway1777 5y ago

And now WhatsApp has hundreds of engineers. Take that to mean whatever you want.

seoaeu 5y ago

It is silly to focus on number of engineers rather than on total cost. Any company should keep hiring more engineers until the additional salary expense exceeds the decreased operating cost/increased revenue that engineer generates

28194608 5y ago

Dear whatsapp devs please disable the feature which shows how much time and when to EVERYONE in the world. TY.

heywherelogingo 5y ago

This site is getting quite repetitive. I think the balance of News to things of interest has gotten quite out of whack. This article is not news, and is also in recent memory so not an interesting re-visit. Is HN average age decreasing? The content character has changed in recent years. Something's changed/changing.

akkpxl 5y ago
I thought I was the only one who felt this way. Almost every day I see the same "style" of articles that isn't news or that interesting. One thing I've noticed is almost every day I see an article explaining why remote work is better and/or how tech interviews are broken, both of which are popular opinions a lot of people on this site share. Maybe it's just me though
bobberkarl 5y ago
Started visiting hn 4/5 years ago, i agree. The content/discussion quality decreased substantially the past 2 years.

mkl95 5y ago

Ultimately, how many engineers a company has is not that relevant when it comes to scaling. A small team of product-oriented engineers with a devops mentality can easily develop, maintain, and continuously deploy a considerably complex application. A similarly complex project in the hands of a larger sales driven company often resembles a spaghetti ball, and is held together with rotting shell scripts and unproductive manual processes.

chatman 5y ago

This is misleading. Likely, those 50 engineers were standing on the shoulders of the giants, i.e. those open source maintainers who build wonderful value for everyone out there.

ratww 5y ago
Does this really matter? Companies with 100, 500, 1000, 5000, etc employees often stand exactly in the same position and use the same kinds of tools.

jokethrowaway 5y ago

While what you says about focus is important, I've seen plenty of focused startups die. WhatsApp had success because they executed well enough at the right time. I hate WhatsApp with passion because of its limits and that horrible web interface that needs your phone connected in order to work. I know plenty of people (me included) who stopped talking with some groups of people because they didn't tolerate WhatsApp. It's mediocre software at best, but because of the network effect and first mover advantage, that was enough.

canadianfella 5y ago

It’s a messaging app. Why would you need more than 50 engineers?