8 ms·

How Facebook uses Erlang for real-time chat

37 points by gosuri 17y ago

melvinram 17y ago
I haven't used Erlang but I wonder if people will start blaming Erlang as "not stable" since Facebook chat is so unstable... the same way people said Ruby on Rails couldn't scale because Twitter was haven't problems scaling.
- mononcqc 17y ago
  I hope not, because it's hard to blame only Erlang. First of all, they rolled their own epoll-based server for the whole chat system. Then, instead of using a 'sane' comet/long-poll implementation based on JSONP (to avoid limitations of same-domain requests), it seems that they used a scheme where the chat nodes have a fake subdomain like 'http://N.subdomain.facebook.com/... http://N.subdomain.facebook.com/.... I don't know how efficient this is supposed to be, but I haven't seen that too often. Of course, it's wholly possible their Erlang implementation of the chat system is the reason why it fails. Personally, I have never had much trouble with their chat, although I haven't used it much either.
  - lukatmyshu 17y ago
    The reason that they use N.subdomain.facebook.com is so they can avoid the default "2 http requests per domain" rule in older browsers.
    - mononcqc 17y ago
      I know that. What I don't get is why they chose that over JSONP, which solves the exact same problem by using <script> tags and a callback function. Their solution seems to work well, anyway.
      - catch23 17y ago
        The jsonp method causes older browsers to use a ton of memory if you leave the browser window open for a long period of time. You'll see that there's a whole bunch of script tags in the dom that aren't cleaned up. Given the way people use facebook & how many old browsers still exist, this is probably the reason facebook decided against it.
        axod 17y ago
        jsonp also causes unsightly "loading" indicators in most browsers.
      - lukatmyshu 17y ago
        Plus w/ jsonp can't get the response from POST requests and you don't get notification that your page didn't load. So your web-stack needs to be able to handle sending any errors via jsonp as well. In any case, they don't solve the same problem. Jsonp solves the problem of talking to another domain (PostMessage does the same thing). Channels solve the problem of the browser limiting the number of concurrent HTTP requests you can have open to one domain.
- rjurney 17y ago
  Given how easy it is to scale an Erlang app to provide incredible reliability in concurrent systems - as that is what the entire language is optimized for - this is pretty unlikely.
  - axod 17y ago
    Is chat a 'concurrent system'? :/
    - rjurney 17y ago
      Yeah.
- gosuri 17y ago
  I believe its the Frontend implementation of the Chat that's unstable. Meebo has done facebook chat implementation pretty well.
- sjs382 17y ago
  I thought the same thing. :)
- warfangle 17y ago
  Then again, the way Twitter overcame their scaling issues was to.. uh, well.. move off from Ruby for a portion of their stack.
aditya 17y ago
Er... why didn't they just use XMPP? I haven't used facebook chat much, but the times that I have used it, it has proven to be a lot more unreliable compared to gTalk.
- mononcqc 17y ago
  Having evaluated some XMPP solutions (over BOSH), one of the problems I have had with it was about the openness of the protocol. You need to tweak the server a whole lot if you want to close down what your users can do and from where they can do it; it becomes complex to synchronize all the data from the backend of the site: who ignores who, privacy settings, friend lists, etc. Another problem that I had encountered was about BOSH. From what I had read in the RFCs, the BOSH session is based on random seed that you hash over and over in SHA (say 200 times). The server then just has to SHA your message to find the hash of the previous message. This, coupled with the basic session identifiers effectively makes it relatively complex to have the chat working over multiple windows or tabs. There might be other reasons, but those are the obvious ones to me.
suhail 17y ago
Isn't this a bit old and why does it go to posterous when it's taken from facebook.com?
- tdmackey 17y ago
  It's very old, and the posterous owner thinks its fine to blatantly copy entire articles by merely providing "From: http://www.facebook.com/note.php?note_id=14218138919 http://www.facebook.com/note.php?note_id=14218138919 as the only indication that it wasn't original.
dylanz 17y ago
That was a fantastic read. This is indeed old news in terms of technology choices, but it was great to hear how they rolled the product out. I almost didn't read this article due to the "isn't this a bit old" comments here, but, am very glad I did.
axod 17y ago
FWIW, Their Facebook chat seems to be constantly wrong. Unless it's just my account. "50 users online", then you click on it, and it changes its mind and says 7 users.
aw3c2 17y ago
blogspam, actual URL is http://www.facebook.com/note.php?note_id=14218138919 http://www.facebook.com/note.php?note_id=14218138919

37 points by gosuri 17y ago

I haven't used Erlang but I wonder if people will start blaming Erlang as "not stable" since Facebook chat is so unstable... the same way people said Ruby on Rails couldn't scale because Twitter was haven't problems scaling.

mononcqc 17y ago
I hope not, because it's hard to blame only Erlang. First of all, they rolled their own epoll-based server for the whole chat system. Then, instead of using a 'sane' comet/long-poll implementation based on JSONP (to avoid limitations of same-domain requests), it seems that they used a scheme where the chat nodes have a fake subdomain like 'http://N.subdomain.facebook.com/... http://N.subdomain.facebook.com/.... I don't know how efficient this is supposed to be, but I haven't seen that too often. Of course, it's wholly possible their Erlang implementation of the chat system is the reason why it fails. Personally, I have never had much trouble with their chat, although I haven't used it much either.
- lukatmyshu 17y ago
  The reason that they use N.subdomain.facebook.com is so they can avoid the default "2 http requests per domain" rule in older browsers.
  - mononcqc 17y ago
    I know that. What I don't get is why they chose that over JSONP, which solves the exact same problem by using <script> tags and a callback function. Their solution seems to work well, anyway.
    - catch23 17y ago
      The jsonp method causes older browsers to use a ton of memory if you leave the browser window open for a long period of time. You'll see that there's a whole bunch of script tags in the dom that aren't cleaned up. Given the way people use facebook & how many old browsers still exist, this is probably the reason facebook decided against it.
      - axod 17y ago
        jsonp also causes unsightly "loading" indicators in most browsers.
    - lukatmyshu 17y ago
      Plus w/ jsonp can't get the response from POST requests and you don't get notification that your page didn't load. So your web-stack needs to be able to handle sending any errors via jsonp as well. In any case, they don't solve the same problem. Jsonp solves the problem of talking to another domain (PostMessage does the same thing). Channels solve the problem of the browser limiting the number of concurrent HTTP requests you can have open to one domain.
rjurney 17y ago
Given how easy it is to scale an Erlang app to provide incredible reliability in concurrent systems - as that is what the entire language is optimized for - this is pretty unlikely.
- axod 17y ago
  Is chat a 'concurrent system'? :/
  - rjurney 17y ago
    Yeah.
gosuri 17y ago
I believe its the Frontend implementation of the Chat that's unstable. Meebo has done facebook chat implementation pretty well.
sjs382 17y ago
I thought the same thing. :)
warfangle 17y ago
Then again, the way Twitter overcame their scaling issues was to.. uh, well.. move off from Ruby for a portion of their stack.

aditya 17y ago

Er... why didn't they just use XMPP? I haven't used facebook chat much, but the times that I have used it, it has proven to be a lot more unreliable compared to gTalk.

mononcqc 17y ago
Having evaluated some XMPP solutions (over BOSH), one of the problems I have had with it was about the openness of the protocol. You need to tweak the server a whole lot if you want to close down what your users can do and from where they can do it; it becomes complex to synchronize all the data from the backend of the site: who ignores who, privacy settings, friend lists, etc. Another problem that I had encountered was about BOSH. From what I had read in the RFCs, the BOSH session is based on random seed that you hash over and over in SHA (say 200 times). The server then just has to SHA your message to find the hash of the previous message. This, coupled with the basic session identifiers effectively makes it relatively complex to have the chat working over multiple windows or tabs. There might be other reasons, but those are the obvious ones to me.

suhail 17y ago

Isn't this a bit old and why does it go to posterous when it's taken from facebook.com?

tdmackey 17y ago
It's very old, and the posterous owner thinks its fine to blatantly copy entire articles by merely providing "From: http://www.facebook.com/note.php?note_id=14218138919 http://www.facebook.com/note.php?note_id=14218138919 as the only indication that it wasn't original.

dylanz 17y ago

That was a fantastic read. This is indeed old news in terms of technology choices, but it was great to hear how they rolled the product out. I almost didn't read this article due to the "isn't this a bit old" comments here, but, am very glad I did.

axod 17y ago

FWIW, Their Facebook chat seems to be constantly wrong. Unless it's just my account. "50 users online", then you click on it, and it changes its mind and says 7 users.

aw3c2 17y ago

blogspam, actual URL is http://www.facebook.com/note.php?note_id=14218138919 http://www.facebook.com/note.php?note_id=14218138919