5 ms·

It wasn't a great article from wired. I feel the language choice is probably the least contributing factor in terms of what's interesting at that scale. Yes,

It wasn't a great article from wired. I feel the language choice is probably the least contributing factor in terms of what's interesting at that scale. Yes, it's an eclectic choice, but I find systems/component architecture much more interesting than language and program architecture in scaling considerations. A bit more info here: http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-to-nearly-500-million-users-11000-cores-an.html http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-... With regard to fun challenges to solve, from that article: "Lots of problems related to scale as you might imagine. Problems with flapping connections, queues getting so long they delay high priority operations, flapping of timers, code that worked just fine at one traffic level breaking badly at higher traffic levels, high priority messages not getting serviced under high load, operations blocking other operations in unexpected ways, failures causing resources issues, and so on. These things just happen and have to be worked through no matter what system you are using."

sidcool 11y ago
I agree, Wired didn't cover the subject with the kind of detail programmers would expect.
- forgetsusername 11y ago
  >with the kind of detail programmers would expect. Wired isn't writing articles for programmers. It's a pop-science magazine writing tech articles for the mainstream.
austenallred 11y ago
I posted this on /r/programming earlier, and there was a fantastic response from /u/MyTribeCalledQuest. It's here https://www.reddit.com/r/programming/comments/3l2spe/why_whatsapp_only_needs_50_engineers_for_its_900m/cv2x4xt https://www.reddit.com/r/programming/comments/3l2spe/why_wha... In its entirety: I can explain a bit more in depth than the author did. Erlang uses a model known as the actor model. This means that each part of your program is a completely independent entity like a mailbox, known as an actor. For one actor to communicate with another actor, it simply puts a message in the intended actor's mailbox. Since each actor is independent, this means that we can spawn as many actors as we want to read messages from a single mailbox, and do what is requested. In terms of what WhatsApp is doing, this means that they can have (potentially infinite) conversations going on between users all at the same time. If any part of their pipeline is working too slowly (there are more messages being put in the mailbox than taken out) they can just spawn additional actors to read mail (at any time they want, even programmatically). Additionally, as alluded to by the author, in Erlang you can do something known as "hot loading," which means spawning newer versions of actors (when the implementation changes) while still running the old actors so that there is never any loss of service. Haskell is similar to Erlang in its ability to scale. In fact, there is a library in Haskell known as HdpH-RS that enables fault-tolerant (the programmer does not have to worry about intermediate failure at all) distributed computing/memory (tested up to 1400 cores). The main reason for this is that Haskell is strictly-typed, meaning all types must be defined (or derivable) at compile time, and immutable, meaning that variables cannot change their values after they are defined. Due to all of this strictness, you never have to worry about errors found in languages like C such as segfaults, or improper use of a function since you can enforce usage with types. For instance a function that should only accept unit vectors would use a type UnitVector instead of Vector (Note that this approach is generally a bad idea in C/C++/Java etc). Essentially, if your program compiles, then it will run correctly (generally the only real errors you run into are logical ones -- the algorithm itself being incorrect). This makes it really easy to refactor code later on. The reason Facebook can handle urgent tasks so quickly is mostly due to Haskell's generics, which are like templates from C++ or generics in Java. The main difference however is that Haskell lets you create generic mixins for types known as type classes. This means that you can write extremely general code to handle almost anything in the high level. Then, whenever you need to handle something new, you just have to choose which mixins you want and just add them to your type by defining the interface. For example, I could make a Multipliable type class, I would require the programmer to only define the multiplication operator for his type, and then it would automatically create things like pow or sqr.
- fanf2 11y ago
  That comment's description of Erlang is wrong. In Erlang, message queues are tied to processes. You can't have multiple processes reading from the same mailbox without writing your own message broker process to implement a multi-reader mailbox.
rdtsc 11y ago
I don't know I found the article decent _for_ being on wired. Didn't expect BEAM VM internals there... > I feel the language choice is probably the least contributing factor in terms of what's interesting at that scale. However, as they say "right from the horses' mouths", WhatsApp engineers have said that Erlang and FreeBSD are critical tools for their success. One can accuse them of lying, of course, but why? Here is an interview with Eugene Fooksman: https://pdincau.wordpress.com/2013/03/27/an-interview-with-e.. https://pdincau.wordpress.com/2013/03/27/an-interview-with-e.... --- ...and the consensus in our team is that it is largely because of Erlang. We’re managing to serve huge amount of connections from single front-end server... --- There is the "That A Billion With A 'B'" talk by Rick Reed at Erlang Factory (already posted somewhere) in here. Here is another older directly from their website: https://blog.whatsapp.com/196/1-million-is-so-2011 https://blog.whatsapp.com/196/1-million-is-so-2011 Clearly states Erlang and FreeBSD are important. > These things just happen and have to be worked through no matter what system you are using. Yeah but that is a bit too generalizing. It is like saying "well the language is Turing Complete" argument, yet it is but one tool is much better than another. Surely Twitter runs on JVM, Google doesn't use Erlang etc etc. So you can do it many other ways. And, at least in this case, they certainly picked the right tools, as we can see from their success. The number of users * amount of messages processed / # of engineers is a not a bad metric to see how good a tool is. And Erlang is a great tool here. (Another example is when you use the smartphone to access the Internet, chances are 50% of the time that connection is set up by Erlang. It is there in the background doing its job, not crashing, setting up even an order of magnitude larger amount of data than we see from WhatsApp).
- mpdehaan2 11y ago
  "One can accuse them of lying, of course, but why?" I never said that.
acconsta 11y ago
>I feel the language choice is probably the least contributing factor in terms of what's interesting at that scale Erlang and BEAM were designed for highly concurrent messaging applications. Surely that's of some importance.

sidcool 11y ago

I agree, Wired didn't cover the subject with the kind of detail programmers would expect.

forgetsusername 11y ago
>with the kind of detail programmers would expect. Wired isn't writing articles for programmers. It's a pop-science magazine writing tech articles for the mainstream.

austenallred 11y ago

I posted this on /r/programming earlier, and there was a fantastic response from /u/MyTribeCalledQuest. It's here https://www.reddit.com/r/programming/comments/3l2spe/why_whatsapp_only_needs_50_engineers_for_its_900m/cv2x4xt https://www.reddit.com/r/programming/comments/3l2spe/why_wha... In its entirety: I can explain a bit more in depth than the author did. Erlang uses a model known as the actor model. This means that each part of your program is a completely independent entity like a mailbox, known as an actor. For one actor to communicate with another actor, it simply puts a message in the intended actor's mailbox. Since each actor is independent, this means that we can spawn as many actors as we want to read messages from a single mailbox, and do what is requested. In terms of what WhatsApp is doing, this means that they can have (potentially infinite) conversations going on between users all at the same time. If any part of their pipeline is working too slowly (there are more messages being put in the mailbox than taken out) they can just spawn additional actors to read mail (at any time they want, even programmatically). Additionally, as alluded to by the author, in Erlang you can do something known as "hot loading," which means spawning newer versions of actors (when the implementation changes) while still running the old actors so that there is never any loss of service. Haskell is similar to Erlang in its ability to scale. In fact, there is a library in Haskell known as HdpH-RS that enables fault-tolerant (the programmer does not have to worry about intermediate failure at all) distributed computing/memory (tested up to 1400 cores). The main reason for this is that Haskell is strictly-typed, meaning all types must be defined (or derivable) at compile time, and immutable, meaning that variables cannot change their values after they are defined. Due to all of this strictness, you never have to worry about errors found in languages like C such as segfaults, or improper use of a function since you can enforce usage with types. For instance a function that should only accept unit vectors would use a type UnitVector instead of Vector (Note that this approach is generally a bad idea in C/C++/Java etc). Essentially, if your program compiles, then it will run correctly (generally the only real errors you run into are logical ones -- the algorithm itself being incorrect). This makes it really easy to refactor code later on. The reason Facebook can handle urgent tasks so quickly is mostly due to Haskell's generics, which are like templates from C++ or generics in Java. The main difference however is that Haskell lets you create generic mixins for types known as type classes. This means that you can write extremely general code to handle almost anything in the high level. Then, whenever you need to handle something new, you just have to choose which mixins you want and just add them to your type by defining the interface. For example, I could make a Multipliable type class, I would require the programmer to only define the multiplication operator for his type, and then it would automatically create things like pow or sqr.

fanf2 11y ago
That comment's description of Erlang is wrong. In Erlang, message queues are tied to processes. You can't have multiple processes reading from the same mailbox without writing your own message broker process to implement a multi-reader mailbox.

rdtsc 11y ago

I don't know I found the article decent _for_ being on wired. Didn't expect BEAM VM internals there... > I feel the language choice is probably the least contributing factor in terms of what's interesting at that scale. However, as they say "right from the horses' mouths", WhatsApp engineers have said that Erlang and FreeBSD are critical tools for their success. One can accuse them of lying, of course, but why? Here is an interview with Eugene Fooksman: https://pdincau.wordpress.com/2013/03/27/an-interview-with-e.. https://pdincau.wordpress.com/2013/03/27/an-interview-with-e.... --- ...and the consensus in our team is that it is largely because of Erlang. We’re managing to serve huge amount of connections from single front-end server... --- There is the "That A Billion With A 'B'" talk by Rick Reed at Erlang Factory (already posted somewhere) in here. Here is another older directly from their website: https://blog.whatsapp.com/196/1-million-is-so-2011 https://blog.whatsapp.com/196/1-million-is-so-2011 Clearly states Erlang and FreeBSD are important. > These things just happen and have to be worked through no matter what system you are using. Yeah but that is a bit too generalizing. It is like saying "well the language is Turing Complete" argument, yet it is but one tool is much better than another. Surely Twitter runs on JVM, Google doesn't use Erlang etc etc. So you can do it many other ways. And, at least in this case, they certainly picked the right tools, as we can see from their success. The number of users * amount of messages processed / # of engineers is a not a bad metric to see how good a tool is. And Erlang is a great tool here. (Another example is when you use the smartphone to access the Internet, chances are 50% of the time that connection is set up by Erlang. It is there in the background doing its job, not crashing, setting up even an order of magnitude larger amount of data than we see from WhatsApp).

mpdehaan2 11y ago
"One can accuse them of lying, of course, but why?" I never said that.

acconsta 11y ago

>I feel the language choice is probably the least contributing factor in terms of what's interesting at that scale Erlang and BEAM were designed for highly concurrent messaging applications. Surely that's of some importance.