7 ms·
Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD users [video]
- general_failure 12y agoNot sure the title is correct. After all, the people use WhatsApp through Linux (Android), Symbian or whatever and not FreeBSD.
- adamnemecek 12y agoThe servers were (are?) FreeBSD.
- tachion 12y agoThe servers are FreeBSD 9.x. After Facebook acquisition they've tried migrating it all to Linux but quickly discovered that Linux cant cope with that amount of traffic and they decided the FreeBSD is the tool to be used here. That lead to 'famous' Facebook job ad looking for Linux kernel engineers to help them bring the Linux TCP/IP capabilities in par with FreeBSD. Currently they're testing FreeBSD 10.1-RELEASE for their next platform OS upgrade.
- agumonkey 12y agoPretty telling technical story and also flattering for the FreeBSD team.
- atmosx 12y agoDo you have any idea why Facebook would rather develop a TCP/IP Stack for Linux to match FreeBSD's, instead of further evolving FreeBSD's stack? Licensing issues?
- justincormack 12y agoHow could licensing issues be a problem? I think people prefer a monoculture, even though the security idea of having multiple software stacks is appealing, thoroughly testing issues on two OS stacks is a lot more work, especially at Facebook scale. Presumably they are tied into Linux for a lot of their internal infrastructure.
- Perdition 12y agoThe main security threat these days isn't viruses or worms so a monoculture makes no real difference. In fact running multiple operating systems is more likely to lead to a misconfiguration issue which allows an attacker into your network.
- justincormack 12y agoIf someone getting into your network is your threat model then, well, you have already been owned. If you want to have resilient services where you can swap them out if there are security issues, a monoculture is not a win.
- MasterScrat 12y agoI would rather guess it is for consistency reasons. People are more familiar with Linux than FreeBSD. Similarly one of the reasons Facebook relies on PHP is because a lot of developers are familiar with it.
- justincormack 12y agoI know lots of people who are more familiar with FreeBSD who are working on Linux systems. The Linux monoculture is just massively dominant.
- tachion 12y agoWhile I have no actual knowledge why they made that decision, its not very hard to guess, and the possible answers are around their own staff familiarity with FreeBSD, the pros of having the platform simplified by using single OS (Linux in their case, since for sure they've more servers with Linux already than they've acquired with WhatsApp) and all possible security and financial implications following these factors. However, it seems like at least for know they might be staying with FreeBSD a bit longer, what's a great PR/recognition win for the FreeBSD itself. I really doubt its for 'licencing issues' because (religious wars aside) the BSD license is way more attractive to businesses than GPL one. Said all that - I might be terribly wrong about everything I said, because its not an insider knowledge ;)
- sciurus 12y agoFacebook is almost certainly further evolving Linux's existing TCP/IP stack, not developing a new one.
- wslh 12y ago> Linux cant cope with that amount of traffic I like FreeBSD but I don't think we can say that Linux can't cope with that amount of traffic while Google and Facebook use Linux. There is a missing piece here that's not about network traffic.
- justincormack 12y agoI think it is amount of traffic per server. WhatsApp didnt have a lot of servers and maybe reengineering for massive scale out on Linux doesnt make sense versus staying with what works.
- nonane 12y agoAre there more details available about this? What specifically about the Linux TCP/IP stack is weaker than BSD's stack? Where does the weakness lie?
- jlouis 12y agoIt doesn't have to be a weaker stack per se. It is also a lot about knowledge. WhatsApp vested much time into understanding FreeBSD. It may be if you vested the equal amount of time on Linux with knowledgable people, then you would see roughly the same performance. If I should take a guess, the advantage of the FreeBSD stack is that it is simpler and easier to understand. Hence, it is easier to tune for performance.
- ps0ps 12y agoThis is not true. Facebook has never attempted to migrate them to Linux. If WhatsApp wants to convert to Linux, it is their decision. More than likely they will stay on FreeBSD as that is what they know how to run and a good way to kill a product is to force them to do things instead of focusing on building their product.
- sandGorgon 12y agohttp://bsd.slashdot.org/story/14/08/06/1731218/facebook-seeks-devs-to-make-linux-network-stack-as-good-as-freebsds http://bsd.slashdot.org/story/14/08/06/1731218/facebook-seek...
- toast0 12y agoA job posting in August indicates a failed migration attempt that couldn't have started until October, because that's when the transaction closed?
- andor 12y agoYes, of course, and Whatsapp users care very much about what's going on behind the scenes ;-) I'm not a mainframe user because I book flights, or a Windows user because i tried Bing.
- tachion 12y agoIt takes very little sense of humor to detect the fact that the title is a nerdy joke ;)
- justincormack 12y agoNetflix is the other FreeBSD based service that many people dont know about. While the control plane is the much hyped Linux on AWS stuff, the actual streams are FreeBSD boxes in ISPs[1]. [1] https://openconnect.itp.netflix.com/software/index.html https://openconnect.itp.netflix.com/software/index.html
- tachion 12y agoThere's also Juniper, EMC2, Apple, Yahoo (back in the day, at least), Sony PlayStation and many, many others.
- justincormack 12y agoI would only really count Juniper in that list, building on and with FreeBSD. Apple take some bits but their product is not really FreeBSD (alas) and they dont seem to give back. AFAIK EMC and Yahoo are ex-users now, or largely.
- feld 12y agoEMC Isilon still uses FreeBSD. There are employees who are also FreeBSD developers. They're working hard on giving back changes so they can jump to a much newer FreeBSD release. Yahoo's FreeBSD isn't dead; YBSD! still exists too.
- adamnemecek 12y agoAlso HN :-)
- perlgeek 12y agoThis feels like mixing of levels to me. The Internet traffic of nearly every user will run through a Cisco or Juniper router. Does that make us all IOS or JunOS users?
- rdtsc 12y agoIt will and if Junos or ios stop working so will the internet to a large extent. It speaks to reliability and scalability aspects. You could say the same about erlang for example, it used by WhatsApp and in many internet-cell network gateways.
- wallflower 12y agoTo me, the more fascinating fact is that Erlang enabled their massive ability to scale. http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
- rudiger 12y agoNot just Erlang/BEAM optimization, but the team's background in high performance, C-based systems on FreeBSD enabled them to achieve massive scale.
- Ixiaus 12y agoWhat's more fascinating is that erlang underpins many more staples of the internet than most geeks even know. WhatsApp is just one 19B example.
- jambo 12y agoYep. Especially when you consider that things like RabbitMQ, Riak, ejabberd, etc. are written in Erlang and take advantage of OTP.
- pjmlp 12y agoNot only the Internet. A few German game studios are using it on their backends.
- marktangotango 12y agoPhysics in Erlang? Game scripting engine implemented in Erlang? What types of games are we talking about here?
- pjmlp 12y agoI wrote backend. Physics and game scripting belong on the client side, usually. http://de.slideshare.net/wooga/erlang-the-big-switch-in-social-games http://de.slideshare.net/wooga/erlang-the-big-switch-in-soci... http://sdtimes.com/enterprises-can-learn-from-game-development/ http://sdtimes.com/enterprises-can-learn-from-game-developme...
- praveenster 12y agoSince Whatsapp doesn't have client facing web page, but only api's used by the client apps, are their scalability requirements different from those that have a web interface as well?
- deleted 12y ago[deleted]
- Demiurge 12y agomessages are probably much smaller, but more numerous and latency has to be 'realtime', but also client-server-client juggling. so, to me this means they need to respond faster, more often, and deal with more sharing than a website.
- dschiptsov 12y agoShowing FreeBSD/Erlang stuff that just works and cope with substantial amounts of traffic is not politically correct, because it will cause a severe pain of cognitive dissonance to the Docker/JVM/Node/Cloud crowds.. Lets chant together: JVM is the best VM in the world with millions of man-hours invested in optimizations. Servlet/JSP are mature, battle-tested, enterprise-grade, type-safe solutions. NodeJS is the only way to develop high-scalable internet services. Everything must be a containerized micro-service, orchestrated by Docker/CoreOS/systemd.
- pjmlp 12y agoIt is worse than that, where you would offshore development like on the other cases? :)
- jambo 12y agoI found the pragmatism and sense of deferring until the least responsible moment—especially the speaker's view of what that moment is—very interesting. In the Q&A "What do you use for configuration management?" "<slight laugh> `make`", going on to say they don't have enough node types or a large enough cluster to do automation yet.
- glasz 12y agojust throwing myself in here: Starve [1] also runs on FreeBSD. [1] https://www.starve.io https://www.starve.io
- sandGorgon 12y agoif one HAD to choose a non-erlang stack (for whatever reason), what would you choose ? Go, Nodejs or Java - I'm mainly concerned about clustering, because I think that is what gave the Whatsapp devs their ability to scale(without having to do too much re-engineering). Or can the clustering be moved to something like the queuing layer ?
- pjmlp 12y agoJava, of course. There are lots of JVM implementations to choose from, many with high quality code generation (both AOT and JIT) and GC implementations. It has VM and cluster monitoring tools only comparable by .NET offerings. The majority of big data solutions are built on top of it. There are lots of libraries to choose from. If Java 8, the language, isn't your thing, there are lots of modern languages to choose from.
- sandGorgon 12y agoI wsa hoping you wouldnt say that :( Java high performance clustering is... intimidating. It is too enterprisey. From my (little) research, it seems that one would need to build this ground up on top of Netty or Undertow.
- marktangotango 12y agoCare to elaborate? Load balancing to a bunch of http servers, sticky sessions, or store the session to a db, or the best option is no session data, memcached, its all pretty straight forward?
- sandGorgon 12y agoCorrect me if I'm wrong, but one of the biggest scaleups in FreeBSD seemed to be zero-copy sockets. This should be solved in Linux when kdbus is turned on (replacing the current IPC : D-Bus).