7 ms·
Facebook Chat Architecture (Erlang)
- mcs 16y agoI wonder if they have been experimenting with Node.JS.
- kennu 16y agoNode.js is nice for evented I/O, but does it solve any of the other problems Erlang is good at, such as distributed parallel processing? How do you scale a Node.js-based service horizontally over many servers?
- axod 16y agoYou could write facebook chat in pretty much any half decent language. The question is how competent the developers are rather than what tool they decide to use. It's not rocket science, it's mainly plumbing - moving data around.
- nivertech 16y agoif you have unlimited resources and huge datacenters - maybe you can. Try to calculate have many nodes will you need to implement COMET-like chat system for 500M users, especially when you using Python or Ruby with some async IO evented framework.
- axod 16y agoI don't think that's the case. Facebook are the ones with 10,000 servers (probably a lot more now idk). Some people are often too quick to try and emulate people/products by picking out some magical quality, using it, and expecting identical results. It's just like women who latch onto fashion tips/diet tips from celebs - eg "Oooo Angelina Jolie used the X diet. If I use that, I can be like her". Similarly, some techies think "Ooo facebook used Erlang. If I use Erlang, I won't have any problems scaling". It's certainly more in how you approach things, and how you architect things than which particular tool you choose to use. There is no magic solution. FWIW, I run Mibbit which handles a good number of users and messages per second on a few servers. Not facebook numbers yet, but not small either. I think we do a few billion messages a month.
- nivertech 16y agoI don't give a shit about Facebook! I use Erlang since Ericsson using it since 80-ies. It's already 12 years since Ericsson open-sourced it. All those evented frameworks are single-threaded. All my servers have at least 16 cores and in few years they'll have 256 cores and more. Good luck handling 256 instances of node.js per server. Good luck connecting all your nodes into distributed cluster. The only real competition for Erlang in this case is Jetty/Netty with Java or Scala. Erlang/OTP is just a tool. You still need to work very hard to build a system like Facebook Chat!
- riprock 16y agoThere's also C++ implementations like boost::asio which would work and scale just fine too. There was a HN post discussing Alex Payne's blog on how node.js is "scaling in the small." http://news.ycombinator.com/item?id=1552908 http://news.ycombinator.com/item?id=1552908 http://al3x.net/2010/07/27/node.html http://al3x.net/2010/07/27/node.html
- axod 16y agoI'm not sure what your point here is. Facebook chat doesn't use CPU. Why would it need 16 cores? It's more than likely IO bound. It's just moving boring data around. If you're using serious CPU power to write a chat backend, you're doing something really badly or using a crappy tool. Your "But I need more CPU cores!!!" argument is moot.
- nivertech 16y agoI don't need more CPUs. But can you find a single core machine nova days? Most of the servers are 16 cores, which in case of evented framework will be heavily underutilized.
- axod 16y ago> "which in case of evented framework will be heavily underutilized." I'm not following your logic here. If something doesn't need 16 CPUs, why is splitting it over 16 cores a good idea? You're going to be adding a lot of inefficiency, complexity etc for no gain. Most of mine are dual or possibly 4 core. One core for OS stuff, other processes etc, one core to do networking IO for me. As I say, networking IO shouldn't be using CPU. If you're doing long or blocking jobs, pass it over to some other thread. (That's a far better way to utilize multiple cores than splitting the whole networking IO over multiple cores).
- deleted 16y ago[deleted]
- jaxtapose 16y agoHow much do you know about Erlang? Because, anybody competent with the tools of distributed systems would know that Erlang is one of the best tools for the job. You could spend a decade, or more, building the base level functionality that comes stock standard in Erlang. If you're not using the right tools, for the right job, you're incompetent.
- revoltingx 16y agoUm, no. Perhaps at the beginning you will mainly be moving around but as use cases get more complex you will find yourself processing more and more data. (i.e. filtering info, generating hashes, gathering statistics, etc.) Not only that, but i bet that most of the data is on RAM cache, and not bound by network io. That is, initial data load time is minimal and the only new data is whatever the user typed. You can log this in the background so the user doesn't wait for the data to be saved to disk or to a network. So, a decently designed app like this will probably not be much network bound and will be cpu/memory bound. With Erlang and a multicore machine you can really get the bang out of your buck like this.
- puzza007 16y agoI'm not sure that would provide any advantages over Erlang. There's a nice discussion here: http://erlang.2086793.n4.nabble.com/node-js-compared-to-erlang-td2953290.html http://erlang.2086793.n4.nabble.com/node-js-compared-to-erla...
- okaramian 16y agoThere is also the issue of Facebook chat being rolled out in 2008, which I'm assuming Node.js didn't exist by. Node.js would need to provide a significant gain in order to justify refactoring that system.
- iregdtoreply 16y agoInteresting. I didn't know they used Erlang. Still, the quality of Facebook chat is pretty poor.
- JonnieCache 16y agoThe XMPP gateway works just fine for me, as reliable as any other centralized IM service I've used heavily. Info on how to connect to it here: http://www.facebook.com/sitetour/chat.php http://www.facebook.com/sitetour/chat.php
- mambodog 16y agoI agree, I use it with iChat, the service is fine. It's the browser-based client that's weak.
- criticurious 16y agoNice feature.. but when i clicked at the link and then clicked on 'Other' (bottom center), it says 'Use SSL/TLS: no'. I wonder if the chat messages are sent unencrypted ?
- JonnieCache 16y agoPersonally I trust the endpoint less than the route in this case.
- tyrmored 16y agoThank you! I had no idea this existed.
- metabrew 16y agoInteresting to see that they mention loving the hot code reloading/upgrades, but also that they don't use the OTP release/upgrade functionality. I wonder what their process is.. Just coping over new .beams and loading them is fine for functional changes only, but you need a system to run code-upgrade hooks if you want to change the state being passed around.
- mononcqc 16y agoAs far as I know, you don't need releases and upgrade functionality to make use of the code-upgrade hooks. It simply works without them. Releases and upgrades do make it simpler to synchronize what needs to be taken down in what order, restarted, left alone, etc. But simple updates are still doable without that.
- metabrew 16y agoYes you can trigger updates just by doing this in most cases: sys:suspend(Pid), {module, my_module} = code:load_file(my_module), sys:change_code(Pid, my_module, undefined, [], 10000), sys:resume(Pid); But you still need a way to do that for running processes, something that release_handler does for you with real OTP releases by finding which pids are running for each module, by looking at supervisors and so on.
- jlouis 16y agoRight, the OTP upgrade system is best when you have few but very controlled upgrades of your application. If you alter the production system once every week, it is better to do something with lighter weight.
- metabrew 16y agoJust watched the video of the talk, "the reason we didn't use the otp release process is that the documentation was very obscure at the time..".
- revoltingx 16y agoErlang is pretty great. It makes node.js look like a toy. I don't use facebook much, but it looks like the chat application was pretty much hacked together and has been able to scale quite well even then. I've used it to build a small multiplayer game and so far I couldn't be happier. Here's a picture of how the architecture works: http://4.bp.blogspot.com/_Vtq6sCdzyKs/TK8Fooqqt9I/AAAAAAAAAGY/AdcMq7zxYUA/s1600/solomon_single_server.png http://4.bp.blogspot.com/_Vtq6sCdzyKs/TK8Fooqqt9I/AAAAAAAAAG...
- koski 16y agoI wonder how much it has changed since the April 2009.
- JonnieCache 16y agoOn page 13 they quote peak inbound traffic as 1gb/second. Would this be a gigabit or gigabyte? Usually I would just assume gigabit but in the case of facebook i dont feel so hasty...
- nivertech 16y agoI guess it's Gb (Gigabit). Since the only traffic Facebook Chat has is presence, which is handled using separate C++ servers. Out of 500M users (350M active users) not many of them use chat, maybe because by design it feels disconnected from the rest of the site.
- ichverstehe 16y agoDo you have any sources to back up that claim? Among my friends Facebook Chat has entirely replaced MSN Messenger.
- nivertech 16y agoanything can replace MSN Messenger ;)
- JonnieCache 16y agoExactly. And everyone in europe used to use MSN messenger. Now I expect they use FB chat. My non tech friends either use FB chat, or they dont use IM.
- ichverstehe 16y agoMSN Messenger had been sitting on the IM throne in Denmark for a couple of years until Facebook Chat came along.
- cageface 16y agoThe nature of contemporary web applications seems to be getting closer and closer to Erlang's original problem domain: lots of clients, continuously pulling down small updates, with a need for consistent and low latency. Maybe it's finally Erlang's turn to shine?
- timdorr 16y agoThis was on here a week or two ago: http://blog.mysyncpad.com/post/2073441622/node-js-vs-erlang-syncpads-experience http://blog.mysyncpad.com/post/2073441622/node-js-vs-erlang-... Erlang's definitely got a speed and concurrency advantage. More people should take the time to learn about it, as it fits a lot of use cases.
- admp 16y agoFor those who are perfectly comfortable with viewing PDFs directly, here's a link: www.erlang-factory.com/upload/presentations/31/EugeneLetuchy-ErlangatFacebook.pdf
- mikedmiked 16y agoAlways appreciated. Thanks.
- Aloisius 16y agoI'm confused as to why one would build a service in a language that is so difficult to hire engineers for.
- tgriesser 16y agoIt makes sense if building a service in that language means that you need to hire far fewer engineers to maintain because it is much better suited to handle the particular demands of that application, as erlang seems to be in this case
- SkyMarshal 16y agoYou don't have to hire engineers with N years of experience in Erlang, rather hire great engineers who are programming language polyglots, and even if they don't already know Erlang they'll get up to speed on it fast, as well as any other new and useful language that comes along.
- DrJosiah 16y agoI wonder if FB would be willing to post some of their load numbers... Like... how many sessions are created per day, persisted from day to day, how many messages sent per chat session, how long windows stay open, how many restores, etc.