27 ms·

Why WhatsApp Only Needs 50 Engineers for Its 900M Users (2015)

233 points by pieterr 4y ago

xrayarx 4y ago
Title should include “2015”.
BaudouinVH 4y ago
The Guardian web site has/used to have a warning displayed for articles of a certain age. This would apply here. The article is from 2015. Close to irrelevant, alas.
- vladsanchez 4y ago
  If they're still powered by the same Erlang codebase, it's STILL VERY RELEVANT. Read https://wiredelta.com/how-was-whatsapp-developed/ https://wiredelta.com/how-was-whatsapp-developed/ and let me know. ;)
  - jterrys 4y ago
    Seems like an uptick in Erlang articles recently. It's interesting how some users seem scared of the language, despite it being so pervasive. I know Discord as an example heavily relies on it.
    - foepys 4y ago
      There was a wave of Elixir articles a few years ago when it was a hot topic. People were always fascinated by Erlang (or rather the underlying BEAM) but it never really broke into the mainstream, probably because it is so different.
- PeterisP 4y ago
  It's still very relevant because if it was possible to provide a billion-user scale service with 50 engineers with 2015 tech, then it's an indicator to others that it is possible to do the same today.
  - threeseed 4y ago
    You can do this today with 1 engineer and a cloud provider. It's pretty trivial these days. The hard part is making this a successful business. Because the minute you try and monetise it with advertising you will quickly find out that building a competitive ad platform is where the real challenge is.
    - peoplefromibiza 4y ago
      > You can do this today with 1 engineer and a cloud provider. It's pretty trivial these days No, you can't. If it really is trivial today, it was in 2015: cloud providers already existed, smartphones already existed, programming tools were already more than mature, anybody could run a few instances of ejabberd and build an app to talk to them. WhatsApp image upload used a simple PHP page... It wasn't exactly rocket science, it was simply very well executed.
    - bsaul 4y ago
      you would be surprised by how much work an e2e chat application with the quality of whatsapp requires. Doing a crappy chat app, yes sure that's easy. But one that works as reliably, with the general performance of whatsapp ? Even with today's tech, that's super hard.
- peoplefromibiza 4y ago
  > The article is from 2015. Close to irrelevant, alas. so things become irrelevant after 7 years now? will this comment be irrelevant in 7 years?
  - deleted 4y ago
    [deleted]
- TigerTeamX 4y ago
  I'd say even more relevant. Why? Because 7 years ago our scaling/technology was worse than what we potentially can do now. If you want you can scale with a small team if you focus on the actual problems instead of overengineering
ChrisRR 4y ago
I'm always more impressed that sites like twitter and facebook need so many engineers for what are fundamentally fairly simple platforms that haven't hugely changed in years
- Beaver117 4y ago
  Ads, sales, new partnerships with companies, etc WhatsApp doesn't have any of those
  - bierjunge 4y ago
    They have at least sales and partnerships: https://business.whatsapp.com/ https://business.whatsapp.com/
    - IceWreck 4y ago
      They do now. The article is from 2015.
    - darkwater 4y ago
      Nowadays that it's Meta. Back in the day it was just a end-user chat platform, and pretty simple.
  - jillesvangurp 4y ago
    They used to have the 1 dollar per year payment for users. With the amount of users they had, that probably covered all of their cost and then some.
- GoToRO 4y ago
  All those people and the X button to close a picture preview is not working now for quite some time.
- fsociety 4y ago
  Ignoring the fact that Facebook/Meta also includes several other businesses, the amount of work required for their adtech alone is non-trivial and the AI play is going to be crucial for that. Plus having the global outreach requires a fair bit of work around compliance and privacy.
deleted 4y ago
[deleted]
assimpleaspossi 4y ago
>Mahdavi joined WhatsApp about two years ago, after the startup was up and running, and its approach to engineering was unlike any he had seen---in part because it used Erlang and a computer operating system called FreeBSD, but also because it strove to keep its operation so simple. "It was a completely different way of building a high-scale infrastructure," he said on Monday. "It was an eye-opener to see the minimalistic approach to solving ... just the problems that needed to be solved."
- ignoramous 4y ago
  Disappointed that both the founders didn't blog much about their running of WhatsApp before acquisition. There's a video or two like Jan Koum at YC Startup School and Rick Reed at Erlang Factory, but that's about it. u/toast0 is the only early WhatsApp employee who's active on HN, may be they'll chime in about what made WhatsApp eng unique.
  - ksec 4y ago
    There are quite a few articles and blog post about Pre Facebook Whatsapp [1] on the web. So it isn't exactly a secret. [1] http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
  - 1024core 4y ago
    Rick Reed was a rock star engineer at Yahoo. Jan Koum was more concerned about getting his shiny BMW dinged and more famous for badly parking the said BMW. (just kidding, Jan, please don't bash my Prius!) Both were very good engineers who managed to create something amazing with a laser-like focus on the product.
  - toast0 4y ago
    Heyo. To address a couple things elsewhere in the thread. WhatsApp circa acquisition was not text only messaging. We had multimedia messaging (video, images, audio), I think since before my time. I don't remember when end to end encryption happened (we were working on it for a long time, and it went through a lot of not user visible testing before we announced it and started turning off plain text messaging), but before that, the multimedia servers also did transcoding; post encryption, we had to do transcoding client side. If you think back to the variety of client platforms supported at the time, video codec support was all over the place. We did have infrastructure to receive payments (apple and google in app payments and paypal), although requiring payment was very selective. I don't think there's published details on that, so I won't go into specifics, but you can't really require payment in places where $1/year is a significant burden or where it's difficult to pay a US based country; and you don't want to require payment in places where people will nope out and use any of the many alternatives. Otoh, my spouse paid for the 5? year plan just from prompting, without even asking me if she could get it free, and she usually doesn't do phone base payments without asking me for help. On profit and loss, I understand why GAAP include stock based compensation, but it's weird to say there was a giant loss because of it. From what I've seen in public numbers, revenue was a bit more than expenses, and that's what we were told internally as well. After acquisition, I had much less visibility into the company financials; I did see daily verification expenses as part of my job, and sometimes saw our SoftLayer bills, but not our headcount expenses or any accounting for server resources in Facebook Infrastructure. Real time voice calls launched As to why things went right. Limited scope and clear product vision helped. You could answer almost any design question by opening the platform SMS/MMS app; our app should look as much like that as possible, because that's what users know and expect. Dedicated platform teams buidling in the platform SDK was the right choice for that, although it does leave the real issue where it's hard to transfer your user data between platforms because each platform designed their own local storage databases. Largely experienced workforce, with great autonomy and responsibility. On the server side, different services were mostly independent and often managed with a team of one or two; we'd do some amount of cross-training, but while I was on vacation, nobody did verification server development, just emergency fixes; and similar for other teams of one. But since I didn't need to coordinate with others, I could push verification server changes multiple times a day as needed (sometimes a few times a day). Chat always had the largest team, if nothing else, mostly all the separate services had to do something with chat, too. Although originally things were very separate, running things some things through chat made state management and authentication simpler. It greatly helped that Erlang is the right fit for a chat service, and that Jan and Brian choice ejabberd to start with when people were using the early WhatsApp as a cludge to chat (originally, WhatsApp was just a short public text 'status' you could see for your contacts, they pivoted to chat later). About half the early server team had used FreeBSD at Yahoo, and zero had any experience with Erlang. It helped that the SoftLayer servers (mostly SuperMicro, although some Lenovo post IBM acquisition) were very stable. This fed into the stability of FreeBSD and the operability of Erlang. Whenever we shut down chat servers, we'd find some clients with chat connections open for 45 days (mostly Nokia S60, which had a stable networking stack, but no push services, so we had to stay connected). Ocassionally, we'd need to do BEAM updates or FreeBSD kernel updates to address issues, but mostly servers were running for months or years uninterrupted. This is only possible with quality software and hardware. Both FreeBSD and BEAM/Erlang/OTP are quite approachable for local patching as well. They don't have a lot of churn, so patches don't need a lot of changes between releases, and things are well organized. We didn't have a ton of patches, but we did run things towards the limits. Not quite the same limits that Netflix explores though. We never did more than 2x10G at Softlayer, but chat didn't need that much bandwidth (ran out of CPU first), and for the most part MMS would be real close to disk bandwidth limits before network limits; and more MMS servers gave us larger storage capacity as well as more network and more CPU (TLS isn't free). Edit: I've heard from SoftLayer that by not using most of their services (including their load balancers, ugh), we stayed in the sweet spot of stability; we did have issues with LAN stability from time to time though; I used to joke that we were their network monitoring team, but they did beef things up there towards the end of our time at SoftLayer, I started getting their incident alerts before we noticed and reported problems. We didn't use any sort of service orchestration until Facebook. I'm old and grumpy now, but I hate all these layers of stuff that hides things. We never needed to split a physical server into multiple jobs, so just running FreeBSD on bare metal was good enough. Erlang's dist and pg2 with some augmentation here and there worked for finding the current active servers. Another thing is WhatsApp fits an offline first model quite well. Chats and MMS queue easily, so does contact synchronization and status updates. Twitter and Facebook feeds where it's loaded interactively and has to (more or less) find your contacts and then get all their recent posts and then sort them with a short deadline is hard. We didn't really have to solve a problem like that, because those processes were background processes and didn't have the same response time needs (some other design decisions also make it easier). Pushing all the state to the client makes things easier for the server, IMHO it also makes the user experience better. And we didn't need to store things forever on the server, which reduces cost. Gosh, that was a wall of text. Hope it's useful. Edit: oh yeah, we all enjoyed being under the radar. There was a lot of distraction at Yahoo and later at Facebook being constantly in the news for this or that thing that was out of engineering's control for the most part. I rembember most of the US coverage of the merger announcement having to explain what WhatsApp even was.
deleted 4y ago
[deleted]
bsaul 4y ago
heard the count is now close to 2000 , thanks to meta. Is that correct ?
afeezaziz 4y ago
Let's go back to the era where we need only a small amount of engineers to build stuffs.
- bsaul 4y ago
  i'm wondering if elon's stunt with twitter will start a new trend.
brendamn 4y ago
Using WhatsApp as an example of a lean engineering org should almost be banned at this point. WhatsApp had a high performing engineering team that used basically the perfect set of tools to build their application (which also had a narrow feature scope; plaintext messaging). Even with hindsight there is very little you could do to improve on how they executed. Just because WhatsApp scaled to almost half a billion users with a small engineering team doesn't mean that's the standard, or even achievable, for almost all teams.
- BoorishBears 4y ago
  I say the opposite, it should be done more often. The problem is if you decide that you will need 1000 person org in 5 years, you will have 2000 person org in 5 years. If you challenge yourself to have a 100 person org, you might end up with 1000 people anyways, but at least you're giving yourself a fair chance. Way too often I see engineering for headcount rather than scale. Building out systems thinking "we'll hire X experts" instead of "we're Y experts, so let's use Y` that's inline with our in house skillset"
  - senttoschool 4y ago
    But Whatsapp lost $140m and made only $10m in revenue in 2014. It wasn't a sustainable business.
    - peoplefromibiza 4y ago
      In the six months ending June 30, 2014, WhatsApp brought in $15.921 million in revenue, but had a net loss of $232.5 million. However, $206.5 million of that loss was for share-based compensation expenses and issuance of common stock below fair value. Its net cash used in operating expenses during the first half of 2014 was $13.5 million, which sounds much more reasonable. Essentially, due to WhatsApp’s quickly rising valuation, it used share-based compensation to attract top talent. Eventually, the $22 billion acquisition by Facebook would largely make the “expenses” of issuing that stock moot. This wasn’t cash that WhatsApp was burning, but paper money it was doling out.
college_physics 4y ago
People try to find flaws or special circumstances when the order of magnitude itself is just mindbogling. Seven orders!! 1000000000/100 = 10^7 This is a feature of software as an economic good which, particularly when further leveraged in open source ecosystems, makes it fundamentally different that any other example. Imagine 50 "bricks and mortar" engineers trying to provide anything to 900M users.
- senttoschool 4y ago
  But they lost $140m in 2014 and made only $10m in revenue. They would have had to shut down their service if they didn't find product market fit or if Meta/someone else didn't buy them. When you need to start making money, you suddenly need to a lot more engineers. Now you need developers to make sales tools, compliance, moderation, finance, A/B test engineers for growth ideas, niche features for big customers, engineers to bring down cloud costs, etc. It doesn't make their engineering less impressive but I think we need to understand the full picture. At the end of the day, WhatsApp needed to make money to keep their service going.
  - peoplefromibiza 4y ago
    > But they lost $140m in 2014 and made only $10m in revenue you posted the same comment two times already. same response https://news.ycombinator.com/item?id=34566572 https://news.ycombinator.com/item?id=34566572
    - senttoschool 4y ago
      So it was still a $26.5m net loss for 6 months? Or $53m net loss for 12 months?
      - B8MGHCBekDuRi 4y ago
        For the year ending December 31, 2013, WhatsApp had $10.2 million in revenue and a net loss of $138.146 million. Net cash used in operating during this period was only $9.9 million, while share-based compensation amounted to $98.8 million They only spent $9.9 million over $10.2 million in revenues for operations. WhatsApp’s goal is still growth, rather than monetization. Mark Zuckerberg and WhatsApp CEO Jan Koum said when the acquisition was made in February that ads aren’t the right way to earn money on messaging They were investing on growth, an investment that lead to a $16 billion acquisition from Facebook. I would love to replicate their same "unsustainable business"
NayamAmarshe 4y ago
Telegram has like 30 people and is more feature rich, faster in development, smoother in operation and the stuff they provide is more mind-blowing than whatever Meta's team has ever come up with.
- sgt 4y ago
  What is Telegram's real time chat feature built in. Also Erlang?
  - deleted 4y ago
    [deleted]
  - pingiun 4y ago
    I believe it's C++
    - sgt 4y ago
      According to HN that is madness. You simply cannot write services in C++! /s
- bsaul 4y ago
  telegram isn't e2e (at least by default) and it doesn't support all the java phones whatsapp does ( not sure if whatsapp still do though). But most of all, it was created much much later, after all the patterns and techniques have already being ironed out by the first wave of chat systems.
  - gorbypark 4y ago
    I might be wrong, but I believe e2e in WhatsApp came after the Meta acquisition. There was some sort of encryption as far back as 2012, but I don't think it was end-to-end. Only iOS and Android had support for it, but not any of the other clients. The partnership with Whisper Systems and "true" e2e came in 2016 (~2 years after the acquisition).
  - NayamAmarshe 4y ago
hello2023 4y ago
whatsapp requires to access to contacts on the phopne. otherwise it just shows a bunch of phone numbers and I never get who I'm talking to in the groups. And it shares my phone number to everyone. Such a feature hasn't been changed and now I know why
- bamboozled 4y ago
  I hate WhatsApp for this reason, but it’s so damn prolific!
- mkl 4y ago
  > And it shares my phone number to everyone. Who is "everyone"? Anyone that searches your name, or all your contacts? (I've never used WhatsApp.)
  - deleted 4y ago
    [deleted]
  - vinay427 4y ago
    As far as I'm aware: 1. People in a conversation with you (including a group conversation you are added to, which I believe you can limit to require your consent) can see your phone number 2. People who already have your number in their contacts can see that your number is registered with WhatsApp
- daydream 4y ago
  If you don’t give WhatsApp access to your contacts it will show people’s screen names in chats, not just their number. WhatsApp only uploads phone numbers from your contacts. No other information. This is all documented here: https://faq.whatsapp.com/1191526044909364/ https://faq.whatsapp.com/1191526044909364/
onion2k 4y ago
There's an important lesson in the WhatsApp story: It's scope that increases headcount, not users. WhatsApp could scale to 900M users because the app they build didn't do a lot. It was for sharing plaintext messages between groups of people. That's quite hard at scale but you can solve it with 50 people as they demonstrated. When they started adding more features, that's when they needed more people. If you keep the scope of your app small you don't need many people to build it.
- Lio 4y ago
  I agree and would expand that argument to say that add that head count doesn't scale linearly. That is adding people to a project means that you have to add even more people to deal with the management complexity. I guess that theoretically it's possible to have a productive very large organisation but it's really hard to do that compared with the simplicity of a smaller team.
  - pieterr 4y ago
    That is adding people to a project means that you have to add even more people to deal with the management complexity. Fred Brooks - The Mythical Man-Month (1975)
    - Lio 4y ago
      Yep that's exactly what I had in mind. It's like a fundamental law of project management that for some reason still gets ignored a lot today.
  - ghaff 4y ago
    Exactly. You can obviously do things with a small team but, if you want to go beyond that, you start to accrete a lot of headcount that's to some degree just a function of having a larger organization that can't work on the basis of every knows everyone and what their role is.
    - naijaboiler 4y ago
boeingUH60 4y ago
Random fact: WhatsApp co-founder Jan Koum is one of the few entrepreneurs who seem to have just retired to a lavish life after cashing out. He hasn’t announced any public projects or startups he’s working on, unlike the other co-founder Acton. But then, he’s built 4 mega yachts [1] and spent $400 million+ on mansions [2]. 1- https://www.superyachtfan.com/yacht/mogambo/owner/ https://www.superyachtfan.com/yacht/mogambo/owner/ 2- https://www.dirt.com/gallery/moguls/tech/jan-koum-house-beverly-hills-whatsapp-1203604615/ https://www.dirt.com/gallery/moguls/tech/jan-koum-house-beve...
- fsloth 4y ago
  It's fine to recognize you're built out as a one trick pony and then enjoy the trappings of success. I don't mean this in a disrespectful manner - building and selling WhatsApp was an astounding feat.
  - ghaff 4y ago
    You're being downvoted. But, while I personally wouldn't have gone the lavish lifestyle route and would probably dabble in things I found interesting, it's not unreasonable and indicated a certain self-awareness to realize you've hit the lottery and you may not want to devote your life to trying to hit the lottery again.
    - slfnflctd 4y ago
      Absolutely valid choice for some of us. Life has taught me that I'm not really all that special in any particular capacity, and more importantly to be okay with it. Being able to cover all my future expenses in a 3 or 4 bedroom place with a high quality security system would be winning at life. I'd definitely want to pursue some hobbies, but I wouldn't expect any of them to make money or change the world. I could at least donate to nonprofits for the latter, though.
      - robocat 4y ago
        > with a high quality security system Jeepers. I am guessing your other priorities in your life make it so you need to live in an unsafe area? I am middle aged, and I have never lived anywhere in my hometown Christchurch that had an active security system. In more that one place the front door or back door was never locked. Probably because I just have never owned much worth stealing! Correction: my parents had a security system, but they stopped using it and it stopped working maybe 15 years ago? I currently live in a particularly safe little area of my city, although I wouldn’t mind having a secret lockup for the few things that can’t be insured. Most rural or small-town New Zealand is even safer than living in my city. Above said, I do believe thievery is increasing in Christchurch. My friends in Woolston had a small shed ram-raided (failed because the shed builder went overboard on big posts), and those friends have needed to add security cameras to help prevent thieves casing their home. Anecdotally theft in Lyttelton is on the rise.
jhoechtl 4y ago
They don't throw out a ton of new features but essentially keep running a very good system?
openplatypus 4y ago
Well if you delegate data analytics, ad and metadata gulping to criminal entity that is Facebook, sure your operations are easier to handle.
unixhero 4y ago
Freebsd
shmde 4y ago
Another tech blog you would find interesting. https://zerodha.tech/blog/hello-world/ https://zerodha.tech/blog/hello-world/ Tldr: How 30 member tech team formed over seven years built India’s largest stock broker.
alberth 4y ago
Jason Fried (Basecamp) had a recent post on this topic of company size. Essentially, they have the same revenue & product as Asana, but with only 1/20th the employees. https://world.hey.com/jason/on-company-size-8095488d https://world.hey.com/jason/on-company-size-8095488d
notepalf 4y ago
building on top of established technologies (XMPP) helps
shudza 4y ago
My wild guess (without opening the link of course) would be that they didn't overengineer their product.
smrtinsert 4y ago
> You had this little island that was Erlang, and it was hard to build enough boats back to the island to make everything hook in," says Facebook vice president of engineering Jay Parikh. This doesn't ring true for me, especially in an era where services so easily communicate via http
ThinkBeat 4y ago
This article is from 2015. Any idea how many engineers they have now? Total headcount would also be interesting PR, Sales, Support, HR, etc etc,
deleted 4y ago
[deleted]
antirez 4y ago
I believe the Telegram team was still very small until recently even if they had many more features.
ilrwbwrkhv 4y ago
This is all a symptom of vc funding rounds increasing in size. Whatsapp raised 250,000 seed. Now seeds are over a few millions at least. That in turn makes companies spend like crazy and not focus on efficiency and so on.

233 points by pieterr 4y ago

Title should include “2015”.

The Guardian web site has/used to have a warning displayed for articles of a certain age. This would apply here. The article is from 2015. Close to irrelevant, alas.

vladsanchez 4y ago
If they're still powered by the same Erlang codebase, it's STILL VERY RELEVANT. Read https://wiredelta.com/how-was-whatsapp-developed/ https://wiredelta.com/how-was-whatsapp-developed/ and let me know. ;)
- jterrys 4y ago
  Seems like an uptick in Erlang articles recently. It's interesting how some users seem scared of the language, despite it being so pervasive. I know Discord as an example heavily relies on it.
  - foepys 4y ago
    There was a wave of Elixir articles a few years ago when it was a hot topic. People were always fascinated by Erlang (or rather the underlying BEAM) but it never really broke into the mainstream, probably because it is so different.
PeterisP 4y ago
It's still very relevant because if it was possible to provide a billion-user scale service with 50 engineers with 2015 tech, then it's an indicator to others that it is possible to do the same today.
- threeseed 4y ago
  You can do this today with 1 engineer and a cloud provider. It's pretty trivial these days. The hard part is making this a successful business. Because the minute you try and monetise it with advertising you will quickly find out that building a competitive ad platform is where the real challenge is.
  - peoplefromibiza 4y ago
    > You can do this today with 1 engineer and a cloud provider. It's pretty trivial these days No, you can't. If it really is trivial today, it was in 2015: cloud providers already existed, smartphones already existed, programming tools were already more than mature, anybody could run a few instances of ejabberd and build an app to talk to them. WhatsApp image upload used a simple PHP page... It wasn't exactly rocket science, it was simply very well executed.
  - bsaul 4y ago
    you would be surprised by how much work an e2e chat application with the quality of whatsapp requires. Doing a crappy chat app, yes sure that's easy. But one that works as reliably, with the general performance of whatsapp ? Even with today's tech, that's super hard.
peoplefromibiza 4y ago
> The article is from 2015. Close to irrelevant, alas. so things become irrelevant after 7 years now? will this comment be irrelevant in 7 years?
- deleted 4y ago
  [deleted]
TigerTeamX 4y ago
I'd say even more relevant. Why? Because 7 years ago our scaling/technology was worse than what we potentially can do now. If you want you can scale with a small team if you focus on the actual problems instead of overengineering

ChrisRR 4y ago

I'm always more impressed that sites like twitter and facebook need so many engineers for what are fundamentally fairly simple platforms that haven't hugely changed in years

Beaver117 4y ago
Ads, sales, new partnerships with companies, etc WhatsApp doesn't have any of those
- bierjunge 4y ago
  They have at least sales and partnerships: https://business.whatsapp.com/ https://business.whatsapp.com/
  - IceWreck 4y ago
    They do now. The article is from 2015.
  - darkwater 4y ago
    Nowadays that it's Meta. Back in the day it was just a end-user chat platform, and pretty simple.
- jillesvangurp 4y ago
  They used to have the 1 dollar per year payment for users. With the amount of users they had, that probably covered all of their cost and then some.
GoToRO 4y ago
All those people and the X button to close a picture preview is not working now for quite some time.
fsociety 4y ago
Ignoring the fact that Facebook/Meta also includes several other businesses, the amount of work required for their adtech alone is non-trivial and the AI play is going to be crucial for that. Plus having the global outreach requires a fair bit of work around compliance and privacy.

deleted 4y ago

[deleted]

assimpleaspossi 4y ago

>Mahdavi joined WhatsApp about two years ago, after the startup was up and running, and its approach to engineering was unlike any he had seen---in part because it used Erlang and a computer operating system called FreeBSD, but also because it strove to keep its operation so simple. "It was a completely different way of building a high-scale infrastructure," he said on Monday. "It was an eye-opener to see the minimalistic approach to solving ... just the problems that needed to be solved."

ignoramous 4y ago
Disappointed that both the founders didn't blog much about their running of WhatsApp before acquisition. There's a video or two like Jan Koum at YC Startup School and Rick Reed at Erlang Factory, but that's about it. u/toast0 is the only early WhatsApp employee who's active on HN, may be they'll chime in about what made WhatsApp eng unique.
- ksec 4y ago
  There are quite a few articles and blog post about Pre Facebook Whatsapp [1] on the web. So it isn't exactly a secret. [1] http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
- 1024core 4y ago
  Rick Reed was a rock star engineer at Yahoo. Jan Koum was more concerned about getting his shiny BMW dinged and more famous for badly parking the said BMW. (just kidding, Jan, please don't bash my Prius!) Both were very good engineers who managed to create something amazing with a laser-like focus on the product.
- toast0 4y ago
  Heyo. To address a couple things elsewhere in the thread. WhatsApp circa acquisition was not text only messaging. We had multimedia messaging (video, images, audio), I think since before my time. I don't remember when end to end encryption happened (we were working on it for a long time, and it went through a lot of not user visible testing before we announced it and started turning off plain text messaging), but before that, the multimedia servers also did transcoding; post encryption, we had to do transcoding client side. If you think back to the variety of client platforms supported at the time, video codec support was all over the place. We did have infrastructure to receive payments (apple and google in app payments and paypal), although requiring payment was very selective. I don't think there's published details on that, so I won't go into specifics, but you can't really require payment in places where $1/year is a significant burden or where it's difficult to pay a US based country; and you don't want to require payment in places where people will nope out and use any of the many alternatives. Otoh, my spouse paid for the 5? year plan just from prompting, without even asking me if she could get it free, and she usually doesn't do phone base payments without asking me for help. On profit and loss, I understand why GAAP include stock based compensation, but it's weird to say there was a giant loss because of it. From what I've seen in public numbers, revenue was a bit more than expenses, and that's what we were told internally as well. After acquisition, I had much less visibility into the company financials; I did see daily verification expenses as part of my job, and sometimes saw our SoftLayer bills, but not our headcount expenses or any accounting for server resources in Facebook Infrastructure. Real time voice calls launched As to why things went right. Limited scope and clear product vision helped. You could answer almost any design question by opening the platform SMS/MMS app; our app should look as much like that as possible, because that's what users know and expect. Dedicated platform teams buidling in the platform SDK was the right choice for that, although it does leave the real issue where it's hard to transfer your user data between platforms because each platform designed their own local storage databases. Largely experienced workforce, with great autonomy and responsibility. On the server side, different services were mostly independent and often managed with a team of one or two; we'd do some amount of cross-training, but while I was on vacation, nobody did verification server development, just emergency fixes; and similar for other teams of one. But since I didn't need to coordinate with others, I could push verification server changes multiple times a day as needed (sometimes a few times a day). Chat always had the largest team, if nothing else, mostly all the separate services had to do something with chat, too. Although originally things were very separate, running things some things through chat made state management and authentication simpler. It greatly helped that Erlang is the right fit for a chat service, and that Jan and Brian choice ejabberd to start with when people were using the early WhatsApp as a cludge to chat (originally, WhatsApp was just a short public text 'status' you could see for your contacts, they pivoted to chat later). About half the early server team had used FreeBSD at Yahoo, and zero had any experience with Erlang. It helped that the SoftLayer servers (mostly SuperMicro, although some Lenovo post IBM acquisition) were very stable. This fed into the stability of FreeBSD and the operability of Erlang. Whenever we shut down chat servers, we'd find some clients with chat connections open for 45 days (mostly Nokia S60, which had a stable networking stack, but no push services, so we had to stay connected). Ocassionally, we'd need to do BEAM updates or FreeBSD kernel updates to address issues, but mostly servers were running for months or years uninterrupted. This is only possible with quality software and hardware. Both FreeBSD and BEAM/Erlang/OTP are quite approachable for local patching as well. They don't have a lot of churn, so patches don't need a lot of changes between releases, and things are well organized. We didn't have a ton of patches, but we did run things towards the limits. Not quite the same limits that Netflix explores though. We never did more than 2x10G at Softlayer, but chat didn't need that much bandwidth (ran out of CPU first), and for the most part MMS would be real close to disk bandwidth limits before network limits; and more MMS servers gave us larger storage capacity as well as more network and more CPU (TLS isn't free). Edit: I've heard from SoftLayer that by not using most of their services (including their load balancers, ugh), we stayed in the sweet spot of stability; we did have issues with LAN stability from time to time though; I used to joke that we were their network monitoring team, but they did beef things up there towards the end of our time at SoftLayer, I started getting their incident alerts before we noticed and reported problems. We didn't use any sort of service orchestration until Facebook. I'm old and grumpy now, but I hate all these layers of stuff that hides things. We never needed to split a physical server into multiple jobs, so just running FreeBSD on bare metal was good enough. Erlang's dist and pg2 with some augmentation here and there worked for finding the current active servers. Another thing is WhatsApp fits an offline first model quite well. Chats and MMS queue easily, so does contact synchronization and status updates. Twitter and Facebook feeds where it's loaded interactively and has to (more or less) find your contacts and then get all their recent posts and then sort them with a short deadline is hard. We didn't really have to solve a problem like that, because those processes were background processes and didn't have the same response time needs (some other design decisions also make it easier). Pushing all the state to the client makes things easier for the server, IMHO it also makes the user experience better. And we didn't need to store things forever on the server, which reduces cost. Gosh, that was a wall of text. Hope it's useful. Edit: oh yeah, we all enjoyed being under the radar. There was a lot of distraction at Yahoo and later at Facebook being constantly in the news for this or that thing that was out of engineering's control for the most part. I rembember most of the US coverage of the merger announcement having to explain what WhatsApp even was.

deleted 4y ago

[deleted]

bsaul 4y ago

heard the count is now close to 2000 , thanks to meta. Is that correct ?

afeezaziz 4y ago

Let's go back to the era where we need only a small amount of engineers to build stuffs.

bsaul 4y ago
i'm wondering if elon's stunt with twitter will start a new trend.

brendamn 4y ago

Using WhatsApp as an example of a lean engineering org should almost be banned at this point. WhatsApp had a high performing engineering team that used basically the perfect set of tools to build their application (which also had a narrow feature scope; plaintext messaging). Even with hindsight there is very little you could do to improve on how they executed. Just because WhatsApp scaled to almost half a billion users with a small engineering team doesn't mean that's the standard, or even achievable, for almost all teams.

BoorishBears 4y ago
I say the opposite, it should be done more often. The problem is if you decide that you will need 1000 person org in 5 years, you will have 2000 person org in 5 years. If you challenge yourself to have a 100 person org, you might end up with 1000 people anyways, but at least you're giving yourself a fair chance. Way too often I see engineering for headcount rather than scale. Building out systems thinking "we'll hire X experts" instead of "we're Y experts, so let's use Y` that's inline with our in house skillset"
- senttoschool 4y ago
  But Whatsapp lost $140m and made only $10m in revenue in 2014. It wasn't a sustainable business.
  - peoplefromibiza 4y ago
    In the six months ending June 30, 2014, WhatsApp brought in $15.921 million in revenue, but had a net loss of $232.5 million. However, $206.5 million of that loss was for share-based compensation expenses and issuance of common stock below fair value. Its net cash used in operating expenses during the first half of 2014 was $13.5 million, which sounds much more reasonable. Essentially, due to WhatsApp’s quickly rising valuation, it used share-based compensation to attract top talent. Eventually, the $22 billion acquisition by Facebook would largely make the “expenses” of issuing that stock moot. This wasn’t cash that WhatsApp was burning, but paper money it was doling out.

college_physics 4y ago

People try to find flaws or special circumstances when the order of magnitude itself is just mindbogling. Seven orders!! 1000000000/100 = 10^7 This is a feature of software as an economic good which, particularly when further leveraged in open source ecosystems, makes it fundamentally different that any other example. Imagine 50 "bricks and mortar" engineers trying to provide anything to 900M users.

senttoschool 4y ago
But they lost $140m in 2014 and made only $10m in revenue. They would have had to shut down their service if they didn't find product market fit or if Meta/someone else didn't buy them. When you need to start making money, you suddenly need to a lot more engineers. Now you need developers to make sales tools, compliance, moderation, finance, A/B test engineers for growth ideas, niche features for big customers, engineers to bring down cloud costs, etc. It doesn't make their engineering less impressive but I think we need to understand the full picture. At the end of the day, WhatsApp needed to make money to keep their service going.
- peoplefromibiza 4y ago
  > But they lost $140m in 2014 and made only $10m in revenue you posted the same comment two times already. same response https://news.ycombinator.com/item?id=34566572 https://news.ycombinator.com/item?id=34566572
  - senttoschool 4y ago
    So it was still a $26.5m net loss for 6 months? Or $53m net loss for 12 months?
    - B8MGHCBekDuRi 4y ago
      For the year ending December 31, 2013, WhatsApp had $10.2 million in revenue and a net loss of $138.146 million. Net cash used in operating during this period was only $9.9 million, while share-based compensation amounted to $98.8 million They only spent $9.9 million over $10.2 million in revenues for operations. WhatsApp’s goal is still growth, rather than monetization. Mark Zuckerberg and WhatsApp CEO Jan Koum said when the acquisition was made in February that ads aren’t the right way to earn money on messaging They were investing on growth, an investment that lead to a $16 billion acquisition from Facebook. I would love to replicate their same "unsustainable business"

NayamAmarshe 4y ago

Telegram has like 30 people and is more feature rich, faster in development, smoother in operation and the stuff they provide is more mind-blowing than whatever Meta's team has ever come up with.

sgt 4y ago
What is Telegram's real time chat feature built in. Also Erlang?
- deleted 4y ago
  [deleted]
- pingiun 4y ago
  I believe it's C++
  - sgt 4y ago
    According to HN that is madness. You simply cannot write services in C++! /s
bsaul 4y ago
telegram isn't e2e (at least by default) and it doesn't support all the java phones whatsapp does ( not sure if whatsapp still do though). But most of all, it was created much much later, after all the patterns and techniques have already being ironed out by the first wave of chat systems.
- gorbypark 4y ago
  I might be wrong, but I believe e2e in WhatsApp came after the Meta acquisition. There was some sort of encryption as far back as 2012, but I don't think it was end-to-end. Only iOS and Android had support for it, but not any of the other clients. The partnership with Whisper Systems and "true" e2e came in 2016 (~2 years after the acquisition).
- NayamAmarshe 4y ago

hello2023 4y ago

whatsapp requires to access to contacts on the phopne. otherwise it just shows a bunch of phone numbers and I never get who I'm talking to in the groups. And it shares my phone number to everyone. Such a feature hasn't been changed and now I know why

bamboozled 4y ago
I hate WhatsApp for this reason, but it’s so damn prolific!
mkl 4y ago
> And it shares my phone number to everyone. Who is "everyone"? Anyone that searches your name, or all your contacts? (I've never used WhatsApp.)
- deleted 4y ago
  [deleted]
- vinay427 4y ago
  As far as I'm aware: 1. People in a conversation with you (including a group conversation you are added to, which I believe you can limit to require your consent) can see your phone number 2. People who already have your number in their contacts can see that your number is registered with WhatsApp
daydream 4y ago
If you don’t give WhatsApp access to your contacts it will show people’s screen names in chats, not just their number. WhatsApp only uploads phone numbers from your contacts. No other information. This is all documented here: https://faq.whatsapp.com/1191526044909364/ https://faq.whatsapp.com/1191526044909364/

onion2k 4y ago

There's an important lesson in the WhatsApp story: It's scope that increases headcount, not users. WhatsApp could scale to 900M users because the app they build didn't do a lot. It was for sharing plaintext messages between groups of people. That's quite hard at scale but you can solve it with 50 people as they demonstrated. When they started adding more features, that's when they needed more people. If you keep the scope of your app small you don't need many people to build it.

Lio 4y ago
I agree and would expand that argument to say that add that head count doesn't scale linearly. That is adding people to a project means that you have to add even more people to deal with the management complexity. I guess that theoretically it's possible to have a productive very large organisation but it's really hard to do that compared with the simplicity of a smaller team.
- pieterr 4y ago
  That is adding people to a project means that you have to add even more people to deal with the management complexity. Fred Brooks - The Mythical Man-Month (1975)
  - Lio 4y ago
    Yep that's exactly what I had in mind. It's like a fundamental law of project management that for some reason still gets ignored a lot today.
- ghaff 4y ago
  Exactly. You can obviously do things with a small team but, if you want to go beyond that, you start to accrete a lot of headcount that's to some degree just a function of having a larger organization that can't work on the basis of every knows everyone and what their role is.
  - naijaboiler 4y ago

boeingUH60 4y ago

Random fact: WhatsApp co-founder Jan Koum is one of the few entrepreneurs who seem to have just retired to a lavish life after cashing out. He hasn’t announced any public projects or startups he’s working on, unlike the other co-founder Acton. But then, he’s built 4 mega yachts [1] and spent $400 million+ on mansions [2]. 1- https://www.superyachtfan.com/yacht/mogambo/owner/ https://www.superyachtfan.com/yacht/mogambo/owner/ 2- https://www.dirt.com/gallery/moguls/tech/jan-koum-house-beverly-hills-whatsapp-1203604615/ https://www.dirt.com/gallery/moguls/tech/jan-koum-house-beve...

fsloth 4y ago
It's fine to recognize you're built out as a one trick pony and then enjoy the trappings of success. I don't mean this in a disrespectful manner - building and selling WhatsApp was an astounding feat.
- ghaff 4y ago
  You're being downvoted. But, while I personally wouldn't have gone the lavish lifestyle route and would probably dabble in things I found interesting, it's not unreasonable and indicated a certain self-awareness to realize you've hit the lottery and you may not want to devote your life to trying to hit the lottery again.
  - slfnflctd 4y ago
    Absolutely valid choice for some of us. Life has taught me that I'm not really all that special in any particular capacity, and more importantly to be okay with it. Being able to cover all my future expenses in a 3 or 4 bedroom place with a high quality security system would be winning at life. I'd definitely want to pursue some hobbies, but I wouldn't expect any of them to make money or change the world. I could at least donate to nonprofits for the latter, though.
    - robocat 4y ago
      > with a high quality security system Jeepers. I am guessing your other priorities in your life make it so you need to live in an unsafe area? I am middle aged, and I have never lived anywhere in my hometown Christchurch that had an active security system. In more that one place the front door or back door was never locked. Probably because I just have never owned much worth stealing! Correction: my parents had a security system, but they stopped using it and it stopped working maybe 15 years ago? I currently live in a particularly safe little area of my city, although I wouldn’t mind having a secret lockup for the few things that can’t be insured. Most rural or small-town New Zealand is even safer than living in my city. Above said, I do believe thievery is increasing in Christchurch. My friends in Woolston had a small shed ram-raided (failed because the shed builder went overboard on big posts), and those friends have needed to add security cameras to help prevent thieves casing their home. Anecdotally theft in Lyttelton is on the rise.

jhoechtl 4y ago

They don't throw out a ton of new features but essentially keep running a very good system?

openplatypus 4y ago

Well if you delegate data analytics, ad and metadata gulping to criminal entity that is Facebook, sure your operations are easier to handle.

unixhero 4y ago

Freebsd

shmde 4y ago

Another tech blog you would find interesting. https://zerodha.tech/blog/hello-world/ https://zerodha.tech/blog/hello-world/ Tldr: How 30 member tech team formed over seven years built India’s largest stock broker.

alberth 4y ago

Jason Fried (Basecamp) had a recent post on this topic of company size. Essentially, they have the same revenue & product as Asana, but with only 1/20th the employees. https://world.hey.com/jason/on-company-size-8095488d https://world.hey.com/jason/on-company-size-8095488d

notepalf 4y ago

building on top of established technologies (XMPP) helps

shudza 4y ago

My wild guess (without opening the link of course) would be that they didn't overengineer their product.

smrtinsert 4y ago

> You had this little island that was Erlang, and it was hard to build enough boats back to the island to make everything hook in," says Facebook vice president of engineering Jay Parikh. This doesn't ring true for me, especially in an era where services so easily communicate via http

ThinkBeat 4y ago

This article is from 2015. Any idea how many engineers they have now? Total headcount would also be interesting PR, Sales, Support, HR, etc etc,

deleted 4y ago

[deleted]

antirez 4y ago

I believe the Telegram team was still very small until recently even if they had many more features.

ilrwbwrkhv 4y ago

This is all a symptom of vc funding rounds increasing in size. Whatsapp raised 250,000 seed. Now seeds are over a few millions at least. That in turn makes companies spend like crazy and not focus on efficiency and so on.