5 ms·

>Mahdavi joined WhatsApp about two years ago, after the startup was up and running, and its approach to engineering was unlike any he had seen---in part because

>Mahdavi joined WhatsApp about two years ago, after the startup was up and running, and its approach to engineering was unlike any he had seen---in part because it used Erlang and a computer operating system called FreeBSD, but also because it strove to keep its operation so simple. "It was a completely different way of building a high-scale infrastructure," he said on Monday. "It was an eye-opener to see the minimalistic approach to solving ... just the problems that needed to be solved."

ignoramous 4y ago
Disappointed that both the founders didn't blog much about their running of WhatsApp before acquisition. There's a video or two like Jan Koum at YC Startup School and Rick Reed at Erlang Factory, but that's about it. u/toast0 is the only early WhatsApp employee who's active on HN, may be they'll chime in about what made WhatsApp eng unique.
- ksec 4y ago
  There are quite a few articles and blog post about Pre Facebook Whatsapp [1] on the web. So it isn't exactly a secret. [1] http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
- 1024core 4y ago
  Rick Reed was a rock star engineer at Yahoo. Jan Koum was more concerned about getting his shiny BMW dinged and more famous for badly parking the said BMW. (just kidding, Jan, please don't bash my Prius!) Both were very good engineers who managed to create something amazing with a laser-like focus on the product.
- toast0 4y ago
  Heyo. To address a couple things elsewhere in the thread. WhatsApp circa acquisition was not text only messaging. We had multimedia messaging (video, images, audio), I think since before my time. I don't remember when end to end encryption happened (we were working on it for a long time, and it went through a lot of not user visible testing before we announced it and started turning off plain text messaging), but before that, the multimedia servers also did transcoding; post encryption, we had to do transcoding client side. If you think back to the variety of client platforms supported at the time, video codec support was all over the place. We did have infrastructure to receive payments (apple and google in app payments and paypal), although requiring payment was very selective. I don't think there's published details on that, so I won't go into specifics, but you can't really require payment in places where $1/year is a significant burden or where it's difficult to pay a US based country; and you don't want to require payment in places where people will nope out and use any of the many alternatives. Otoh, my spouse paid for the 5? year plan just from prompting, without even asking me if she could get it free, and she usually doesn't do phone base payments without asking me for help. On profit and loss, I understand why GAAP include stock based compensation, but it's weird to say there was a giant loss because of it. From what I've seen in public numbers, revenue was a bit more than expenses, and that's what we were told internally as well. After acquisition, I had much less visibility into the company financials; I did see daily verification expenses as part of my job, and sometimes saw our SoftLayer bills, but not our headcount expenses or any accounting for server resources in Facebook Infrastructure. Real time voice calls launched As to why things went right. Limited scope and clear product vision helped. You could answer almost any design question by opening the platform SMS/MMS app; our app should look as much like that as possible, because that's what users know and expect. Dedicated platform teams buidling in the platform SDK was the right choice for that, although it does leave the real issue where it's hard to transfer your user data between platforms because each platform designed their own local storage databases. Largely experienced workforce, with great autonomy and responsibility. On the server side, different services were mostly independent and often managed with a team of one or two; we'd do some amount of cross-training, but while I was on vacation, nobody did verification server development, just emergency fixes; and similar for other teams of one. But since I didn't need to coordinate with others, I could push verification server changes multiple times a day as needed (sometimes a few times a day). Chat always had the largest team, if nothing else, mostly all the separate services had to do something with chat, too. Although originally things were very separate, running things some things through chat made state management and authentication simpler. It greatly helped that Erlang is the right fit for a chat service, and that Jan and Brian choice ejabberd to start with when people were using the early WhatsApp as a cludge to chat (originally, WhatsApp was just a short public text 'status' you could see for your contacts, they pivoted to chat later). About half the early server team had used FreeBSD at Yahoo, and zero had any experience with Erlang. It helped that the SoftLayer servers (mostly SuperMicro, although some Lenovo post IBM acquisition) were very stable. This fed into the stability of FreeBSD and the operability of Erlang. Whenever we shut down chat servers, we'd find some clients with chat connections open for 45 days (mostly Nokia S60, which had a stable networking stack, but no push services, so we had to stay connected). Ocassionally, we'd need to do BEAM updates or FreeBSD kernel updates to address issues, but mostly servers were running for months or years uninterrupted. This is only possible with quality software and hardware. Both FreeBSD and BEAM/Erlang/OTP are quite approachable for local patching as well. They don't have a lot of churn, so patches don't need a lot of changes between releases, and things are well organized. We didn't have a ton of patches, but we did run things towards the limits. Not quite the same limits that Netflix explores though. We never did more than 2x10G at Softlayer, but chat didn't need that much bandwidth (ran out of CPU first), and for the most part MMS would be real close to disk bandwidth limits before network limits; and more MMS servers gave us larger storage capacity as well as more network and more CPU (TLS isn't free). Edit: I've heard from SoftLayer that by not using most of their services (including their load balancers, ugh), we stayed in the sweet spot of stability; we did have issues with LAN stability from time to time though; I used to joke that we were their network monitoring team, but they did beef things up there towards the end of our time at SoftLayer, I started getting their incident alerts before we noticed and reported problems. We didn't use any sort of service orchestration until Facebook. I'm old and grumpy now, but I hate all these layers of stuff that hides things. We never needed to split a physical server into multiple jobs, so just running FreeBSD on bare metal was good enough. Erlang's dist and pg2 with some augmentation here and there worked for finding the current active servers. Another thing is WhatsApp fits an offline first model quite well. Chats and MMS queue easily, so does contact synchronization and status updates. Twitter and Facebook feeds where it's loaded interactively and has to (more or less) find your contacts and then get all their recent posts and then sort them with a short deadline is hard. We didn't really have to solve a problem like that, because those processes were background processes and didn't have the same response time needs (some other design decisions also make it easier). Pushing all the state to the client makes things easier for the server, IMHO it also makes the user experience better. And we didn't need to store things forever on the server, which reduces cost. Gosh, that was a wall of text. Hope it's useful. Edit: oh yeah, we all enjoyed being under the radar. There was a lot of distraction at Yahoo and later at Facebook being constantly in the news for this or that thing that was out of engineering's control for the most part. I rembember most of the US coverage of the merger announcement having to explain what WhatsApp even was.
  - tiffanyh 4y ago
    Always love reading posts by toast0. toast0, if you ever wrote a book about your experience & lessons learned in how to build a great product from the technical side - I'd definitely buy it for myself and gift it to friends. Just say'n :)
    - toast0 4y ago
      Awww. Sadly, I don't have nearly the interest required to write a book. The biggest reason I would is because there's a Pullitzer winning author who stole my name and writes awful fiction with amusing titles. I bet I could sell a lot of books because of name confusion. ;) And it would be payback for the anxiety I got when I heard that Terry Gross was going to interview me later in the day on Fresh Air and I wws totally unprepared. Luckily, it was already recorded and they played a clip and it wasn't me. ;) For user facing product design my advice is simple: think about what I would do... and then do something else ;) but I'll accept my advice/thoughts for user invisible stuff could be useful.
  - klelatti 4y ago
    Thanks for a terrific and informative response! > About half the early server team had used FreeBSD at Yahoo, and zero had any experience with Erlang. Can you say anything more about the choice of Erlang and experience with it? It seems pretty remarkable that you built such a successful business / service with no experience of Erlang.
    - toast0 4y ago
      Hire smart people, give them Erlang books, tada. Erlang is basically a perfect fit for a chat server. ejabberd is out there and there's several high profile use cases where someone needed a chat server, picked ejabberd and became smitten with Erlang. Facebook did it before WhatsApp, but they rebuilt chat in C++ because they couldn't hire Erlang people? and the two people working on it wanted to have some time to do other things. Riot Games did it more recently. The nice thing about Erlang is it's really easy to make changes to your system; you don't need to move traffic and restart a server, you can just hotload the code changes. So, they started with ejabberd, and by the time I joined it was completely different. Erlang also has tremendous observability. Each Erlang process has detailed information available (heap size, message queue length, reduction count (cpu use, kind of), you can view the message queue if you want, etc). The BEAM is also observable, it's even got DTrace hooks. There's a way to build where you get stats on lock contention (lock counting), which pretty quickly helps you find the root of scaling issues. Once you find lock contention, almost always the answer is find a way to distribute the work over more locks so there's less contention; same thing you do when a process's message queue backs up. It is a small community, so if you're blessed with needing to scale, you are a bit on your own, but you probably could reach out to Erlang Solutions, if you get really stuck. Erlang will scale much farther than the community thinks though. WhatsApp routinely ran mnesia databases with way more data than people thought possible (just don't use disc_only_copies for anything other than the schema table) and dist clusters way larger than people thought possible (pg2 and global locks in general got iffy at times, but I think the new pg is better, and we didn't use global locks except through pg2). There is a general trend where there's no explicit limit on things, but the system will not work well if you go over the implicit limit, and you won't have much guidance. I hear WhatsApp now uses proper Erlang application packaging, wkth releases and relups, etc. But when I was there, we just did hotloading and it was good enough.
ksec 4y ago
> and a computer operating system called FreeBSD, I dont know why but I cant stop laughing when I read this line.

ignoramous 4y ago

Disappointed that both the founders didn't blog much about their running of WhatsApp before acquisition. There's a video or two like Jan Koum at YC Startup School and Rick Reed at Erlang Factory, but that's about it. u/toast0 is the only early WhatsApp employee who's active on HN, may be they'll chime in about what made WhatsApp eng unique.

ksec 4y ago
There are quite a few articles and blog post about Pre Facebook Whatsapp [1] on the web. So it isn't exactly a secret. [1] http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
1024core 4y ago
Rick Reed was a rock star engineer at Yahoo. Jan Koum was more concerned about getting his shiny BMW dinged and more famous for badly parking the said BMW. (just kidding, Jan, please don't bash my Prius!) Both were very good engineers who managed to create something amazing with a laser-like focus on the product.
toast0 4y ago
Heyo. To address a couple things elsewhere in the thread. WhatsApp circa acquisition was not text only messaging. We had multimedia messaging (video, images, audio), I think since before my time. I don't remember when end to end encryption happened (we were working on it for a long time, and it went through a lot of not user visible testing before we announced it and started turning off plain text messaging), but before that, the multimedia servers also did transcoding; post encryption, we had to do transcoding client side. If you think back to the variety of client platforms supported at the time, video codec support was all over the place. We did have infrastructure to receive payments (apple and google in app payments and paypal), although requiring payment was very selective. I don't think there's published details on that, so I won't go into specifics, but you can't really require payment in places where $1/year is a significant burden or where it's difficult to pay a US based country; and you don't want to require payment in places where people will nope out and use any of the many alternatives. Otoh, my spouse paid for the 5? year plan just from prompting, without even asking me if she could get it free, and she usually doesn't do phone base payments without asking me for help. On profit and loss, I understand why GAAP include stock based compensation, but it's weird to say there was a giant loss because of it. From what I've seen in public numbers, revenue was a bit more than expenses, and that's what we were told internally as well. After acquisition, I had much less visibility into the company financials; I did see daily verification expenses as part of my job, and sometimes saw our SoftLayer bills, but not our headcount expenses or any accounting for server resources in Facebook Infrastructure. Real time voice calls launched As to why things went right. Limited scope and clear product vision helped. You could answer almost any design question by opening the platform SMS/MMS app; our app should look as much like that as possible, because that's what users know and expect. Dedicated platform teams buidling in the platform SDK was the right choice for that, although it does leave the real issue where it's hard to transfer your user data between platforms because each platform designed their own local storage databases. Largely experienced workforce, with great autonomy and responsibility. On the server side, different services were mostly independent and often managed with a team of one or two; we'd do some amount of cross-training, but while I was on vacation, nobody did verification server development, just emergency fixes; and similar for other teams of one. But since I didn't need to coordinate with others, I could push verification server changes multiple times a day as needed (sometimes a few times a day). Chat always had the largest team, if nothing else, mostly all the separate services had to do something with chat, too. Although originally things were very separate, running things some things through chat made state management and authentication simpler. It greatly helped that Erlang is the right fit for a chat service, and that Jan and Brian choice ejabberd to start with when people were using the early WhatsApp as a cludge to chat (originally, WhatsApp was just a short public text 'status' you could see for your contacts, they pivoted to chat later). About half the early server team had used FreeBSD at Yahoo, and zero had any experience with Erlang. It helped that the SoftLayer servers (mostly SuperMicro, although some Lenovo post IBM acquisition) were very stable. This fed into the stability of FreeBSD and the operability of Erlang. Whenever we shut down chat servers, we'd find some clients with chat connections open for 45 days (mostly Nokia S60, which had a stable networking stack, but no push services, so we had to stay connected). Ocassionally, we'd need to do BEAM updates or FreeBSD kernel updates to address issues, but mostly servers were running for months or years uninterrupted. This is only possible with quality software and hardware. Both FreeBSD and BEAM/Erlang/OTP are quite approachable for local patching as well. They don't have a lot of churn, so patches don't need a lot of changes between releases, and things are well organized. We didn't have a ton of patches, but we did run things towards the limits. Not quite the same limits that Netflix explores though. We never did more than 2x10G at Softlayer, but chat didn't need that much bandwidth (ran out of CPU first), and for the most part MMS would be real close to disk bandwidth limits before network limits; and more MMS servers gave us larger storage capacity as well as more network and more CPU (TLS isn't free). Edit: I've heard from SoftLayer that by not using most of their services (including their load balancers, ugh), we stayed in the sweet spot of stability; we did have issues with LAN stability from time to time though; I used to joke that we were their network monitoring team, but they did beef things up there towards the end of our time at SoftLayer, I started getting their incident alerts before we noticed and reported problems. We didn't use any sort of service orchestration until Facebook. I'm old and grumpy now, but I hate all these layers of stuff that hides things. We never needed to split a physical server into multiple jobs, so just running FreeBSD on bare metal was good enough. Erlang's dist and pg2 with some augmentation here and there worked for finding the current active servers. Another thing is WhatsApp fits an offline first model quite well. Chats and MMS queue easily, so does contact synchronization and status updates. Twitter and Facebook feeds where it's loaded interactively and has to (more or less) find your contacts and then get all their recent posts and then sort them with a short deadline is hard. We didn't really have to solve a problem like that, because those processes were background processes and didn't have the same response time needs (some other design decisions also make it easier). Pushing all the state to the client makes things easier for the server, IMHO it also makes the user experience better. And we didn't need to store things forever on the server, which reduces cost. Gosh, that was a wall of text. Hope it's useful. Edit: oh yeah, we all enjoyed being under the radar. There was a lot of distraction at Yahoo and later at Facebook being constantly in the news for this or that thing that was out of engineering's control for the most part. I rembember most of the US coverage of the merger announcement having to explain what WhatsApp even was.
- tiffanyh 4y ago
  Always love reading posts by toast0. toast0, if you ever wrote a book about your experience & lessons learned in how to build a great product from the technical side - I'd definitely buy it for myself and gift it to friends. Just say'n :)
  - toast0 4y ago
    Awww. Sadly, I don't have nearly the interest required to write a book. The biggest reason I would is because there's a Pullitzer winning author who stole my name and writes awful fiction with amusing titles. I bet I could sell a lot of books because of name confusion. ;) And it would be payback for the anxiety I got when I heard that Terry Gross was going to interview me later in the day on Fresh Air and I wws totally unprepared. Luckily, it was already recorded and they played a clip and it wasn't me. ;) For user facing product design my advice is simple: think about what I would do... and then do something else ;) but I'll accept my advice/thoughts for user invisible stuff could be useful.
- klelatti 4y ago
  Thanks for a terrific and informative response! > About half the early server team had used FreeBSD at Yahoo, and zero had any experience with Erlang. Can you say anything more about the choice of Erlang and experience with it? It seems pretty remarkable that you built such a successful business / service with no experience of Erlang.
  - toast0 4y ago
    Hire smart people, give them Erlang books, tada. Erlang is basically a perfect fit for a chat server. ejabberd is out there and there's several high profile use cases where someone needed a chat server, picked ejabberd and became smitten with Erlang. Facebook did it before WhatsApp, but they rebuilt chat in C++ because they couldn't hire Erlang people? and the two people working on it wanted to have some time to do other things. Riot Games did it more recently. The nice thing about Erlang is it's really easy to make changes to your system; you don't need to move traffic and restart a server, you can just hotload the code changes. So, they started with ejabberd, and by the time I joined it was completely different. Erlang also has tremendous observability. Each Erlang process has detailed information available (heap size, message queue length, reduction count (cpu use, kind of), you can view the message queue if you want, etc). The BEAM is also observable, it's even got DTrace hooks. There's a way to build where you get stats on lock contention (lock counting), which pretty quickly helps you find the root of scaling issues. Once you find lock contention, almost always the answer is find a way to distribute the work over more locks so there's less contention; same thing you do when a process's message queue backs up. It is a small community, so if you're blessed with needing to scale, you are a bit on your own, but you probably could reach out to Erlang Solutions, if you get really stuck. Erlang will scale much farther than the community thinks though. WhatsApp routinely ran mnesia databases with way more data than people thought possible (just don't use disc_only_copies for anything other than the schema table) and dist clusters way larger than people thought possible (pg2 and global locks in general got iffy at times, but I think the new pg is better, and we didn't use global locks except through pg2). There is a general trend where there's no explicit limit on things, but the system will not work well if you go over the implicit limit, and you won't have much guidance. I hear WhatsApp now uses proper Erlang application packaging, wkth releases and relups, etc. But when I was there, we just did hotloading and it was good enough.

ksec 4y ago

> and a computer operating system called FreeBSD, I dont know why but I cant stop laughing when I read this line.