10 ms·

Rails in Realtime - How LayerVault updates every page in a second

94 points by kellysutton 14y ago

gingerlime 14y ago
Nice post with lots of cool stuff. Any particular reason you're using delayed_job for async jobs + memcache for caching and not resque/redis? Not saying anything bad about the former, just wonder if there are pros/cons you considered when choosing between the two?
- kellysutton 14y ago
  When first building LayerVault around this time last year, the simplicity of delayed_job appealed to me. Not having to set up and maintain a Redis application was a bonus. The simple '.delay' syntax sealed the deal for me. That being said, the day where we need to separate our queueing system will come. But delayed_job/MySQL has brought us much further than I'd thought possible.
  - davidw 14y ago
    I think this is a fantastic illustration of how it's more important to get something 'pretty good' out in front of customers, even if you could have used the latest/coolest/hippest things. You're illustrating how you can go back and improve it later, when you're clearer on what needs doing.
mikeryan 14y ago
Really interesting article. But it the final line seems slightly disingenuous Building a realtime web application does not require you to begin with a realtime framework. Rails is used for CRUD operations but Node/Socket.io/Jquery are your 'realtime' framework. Its just simple, proprietary (and not in a bad way) and does exactly the job you need.
- kellysutton 14y ago
  Very true. I see some folks assume that to get that realtime performance, you need to start with a "realtime" framework. We probably will hit a ceiling with Rails (once we decide to take the time to update down from 500ms to 50ms, perhaps) that we might not run into using something built for the job. LayerVault wasn't built to be realtime first and foremost. No disingenuity intended.
- 1qaz2wsx3edc 14y ago
  As soon as the author wrote "socket.io" the gears in my brain ground to a stop. Use observers, pusher, and a client side MVC. Don't serialize records deeply, your client side controller or framework should handle the updates nicely. IMO use `after_commit` http://rails-bestpractices.com/posts/695-use-after_commit http://rails-bestpractices.com/posts/695-use-after_commit . Just let that data flow, brawhlings.
  - drivebyacct2 14y ago
    Why oh why is pusher better than a websocket here? Especially since pusher is a third party service and it's probably using (or is able to use) websockets as a transport for that push anyway? Sure, you can use Sever-Sent-Events, but a Websocket would be sufficient too (and probably works in more places, no less).
    - damncabbage 14y ago
      Especially since pusher is a third party service and it's probably using (or is able to use) websockets as a transport for that push anyway? Pusher.com's page title is "HTML5 WebSocket Powered Realtime Messaging Service", no less.
    - 1qaz2wsx3edc 14y ago
      Let me clarify, you can use pusher, or something like firehose.io. Pusher uses a web socket. Pusher is cheaper when comparing against running our own dumb pub/sub server. Rails 4 will support SSE, however we're not riding that edge as there were existing solutions to the problem. - I think this covers most realtime problems without having to code that much, let alone a secondary server/service to handle the problem, we outsourced it, were happy.
      - drivebyacct2 14y ago
        Hm, I guess I'm blessed with a library for it, but I'm using my own WebSocket server with no sweat right now.
    - 14y ago
mthomas 14y ago
Can I ask that all product blogs provide a link to the actual site? Not everyone knows what layervault is and it would be nice to browse to the site without having to manipulate the address bar.
deleted 14y ago
[deleted]
adrianpike 14y ago
Cool article, thanks for the insight into your stack. One thing that threw me out of the gate, though, was lumping in Backbone.js in with Meteor & Ember. Backbone at it's heart is just a way of providing some structure to your JS models - it doesn't really have anywhere near the level of complexity that Ember, Meteor, and some of the other frameworks have, and it still doesn't provide anything for server-side persistence, synchronization, access control, etc, so transitioning away from Rails to Backbone wouldn't really be an option. :) Big thanks for dropping the link to cache_digests too, not sure how I've managed to miss out on that so far!
mhartl 14y ago
Just a quick heads-up in case any LayerVault peeps are reading: the post rather badly misspells "David Heinemeier Hansson".
- kellysutton 14y ago
  Whoops. Fixed and gave you a credit in the post footer.
  - mhartl 14y ago
    Excellent.
ludwigvan 14y ago
Here is a recent video of DHH, discussing some of the techniques and issues mentioned in this article, such as backbone vs rails, pjax, Russian doll caching: http://www.youtube.com/watch?v=FkLVl3gpJP4 http://www.youtube.com/watch?v=FkLVl3gpJP4
davemo 14y ago
As someone relatively new to Rails who is working on a "near" realtime system this was great, thanks for the article. Wanted to point out that you can simplify your "folder count updating" code by using a relatively unused jQuery function: ".load" [1]. $.get($section.attr('data-url')) .success(function (response) { $section.html(response); }); would become $section.load($section.data('url')); [1] http://jqapi.com/#p=load http://jqapi.com/#p=load
23david 14y ago
Congrats for successfully getting your rails site to load faster, but it's still slow. Honestly, what you're doing at this point is delaying the big rewrite. That's not a bad thing, but it will still be necessary at some point to offload your real-time features to systems that are intended for those kinds of things from the ground up. Rails was fundamentally designed to optimize for programmer productivity over speed of execution, with the idea that people generally cost more than hardware. Unfortunately once you get to serious scale the hardware necessary to keep things quick starts costing a lot and operationally it becomes a big headache and stuff just doesn't work. It's going to take a seismic shift for rails to suddenly outperform lightweight 'real-time' frameworks running on faster scripting languages. We need to be honest and aware of both the benefits as well as the limitations of our tools.
- batista 14y ago
  >Congrats for successfully getting your rails site to load faster, but it's still slow. Looks pretty snappy to me. >Honestly, what you're doing at this point is delaying the big rewrite. That sounds brilliant! As in: http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html
  - 23david 14y ago
    That's definitely a classic article. It just doesn't apply here. Hard to explain why without going into a ton of details, but basically: Layervault in 2012 != Netscape in 1996 Netscape's crufty spaghetti C code != Layervault's elegant but slow Rails code
alexquez 14y ago
Hey guys you mentioned that you have the observer send a message to the room. How does the Rails code send a message to the node server?
- matthuggins 14y ago
  This is along the same lines as a question I asked here too. Hopefully we'll see at least one of ours answered, as I'm curious as well. :)
matthuggins 14y ago
You made two points that sound contradictory without further explanation, if you don't mind providing more insight. First, you mentioned that you want to stick with Rails without duplicating views, etc. Second, you mention that you're using Socket.IO (i.e.: Node.JS as opposed to Ruby/Rails). How are you integrating Socket.IO with your Rails app? Do you have a separate Node.JS server that is just glossed over in your post, or do you use something like execjs to integrate Socket.IO directly into your Rails app? Thanks for sharing!

94 points by kellysutton 14y ago

Nice post with lots of cool stuff. Any particular reason you're using delayed_job for async jobs + memcache for caching and not resque/redis? Not saying anything bad about the former, just wonder if there are pros/cons you considered when choosing between the two?

kellysutton 14y ago
When first building LayerVault around this time last year, the simplicity of delayed_job appealed to me. Not having to set up and maintain a Redis application was a bonus. The simple '.delay' syntax sealed the deal for me. That being said, the day where we need to separate our queueing system will come. But delayed_job/MySQL has brought us much further than I'd thought possible.
- davidw 14y ago
  I think this is a fantastic illustration of how it's more important to get something 'pretty good' out in front of customers, even if you could have used the latest/coolest/hippest things. You're illustrating how you can go back and improve it later, when you're clearer on what needs doing.

mikeryan 14y ago

Really interesting article. But it the final line seems slightly disingenuous Building a realtime web application does not require you to begin with a realtime framework. Rails is used for CRUD operations but Node/Socket.io/Jquery are your 'realtime' framework. Its just simple, proprietary (and not in a bad way) and does exactly the job you need.

kellysutton 14y ago
Very true. I see some folks assume that to get that realtime performance, you need to start with a "realtime" framework. We probably will hit a ceiling with Rails (once we decide to take the time to update down from 500ms to 50ms, perhaps) that we might not run into using something built for the job. LayerVault wasn't built to be realtime first and foremost. No disingenuity intended.
1qaz2wsx3edc 14y ago
As soon as the author wrote "socket.io" the gears in my brain ground to a stop. Use observers, pusher, and a client side MVC. Don't serialize records deeply, your client side controller or framework should handle the updates nicely. IMO use `after_commit` http://rails-bestpractices.com/posts/695-use-after_commit http://rails-bestpractices.com/posts/695-use-after_commit . Just let that data flow, brawhlings.
- drivebyacct2 14y ago
  Why oh why is pusher better than a websocket here? Especially since pusher is a third party service and it's probably using (or is able to use) websockets as a transport for that push anyway? Sure, you can use Sever-Sent-Events, but a Websocket would be sufficient too (and probably works in more places, no less).
  - damncabbage 14y ago
    Especially since pusher is a third party service and it's probably using (or is able to use) websockets as a transport for that push anyway? Pusher.com's page title is "HTML5 WebSocket Powered Realtime Messaging Service", no less.
  - 1qaz2wsx3edc 14y ago
    Let me clarify, you can use pusher, or something like firehose.io. Pusher uses a web socket. Pusher is cheaper when comparing against running our own dumb pub/sub server. Rails 4 will support SSE, however we're not riding that edge as there were existing solutions to the problem. - I think this covers most realtime problems without having to code that much, let alone a secondary server/service to handle the problem, we outsourced it, were happy.
    - drivebyacct2 14y ago
      Hm, I guess I'm blessed with a library for it, but I'm using my own WebSocket server with no sweat right now.
  - 14y ago

mthomas 14y ago

Can I ask that all product blogs provide a link to the actual site? Not everyone knows what layervault is and it would be nice to browse to the site without having to manipulate the address bar.

deleted 14y ago

[deleted]

adrianpike 14y ago

Cool article, thanks for the insight into your stack. One thing that threw me out of the gate, though, was lumping in Backbone.js in with Meteor & Ember. Backbone at it's heart is just a way of providing some structure to your JS models - it doesn't really have anywhere near the level of complexity that Ember, Meteor, and some of the other frameworks have, and it still doesn't provide anything for server-side persistence, synchronization, access control, etc, so transitioning away from Rails to Backbone wouldn't really be an option. :) Big thanks for dropping the link to cache_digests too, not sure how I've managed to miss out on that so far!

mhartl 14y ago

Just a quick heads-up in case any LayerVault peeps are reading: the post rather badly misspells "David Heinemeier Hansson".

kellysutton 14y ago
Whoops. Fixed and gave you a credit in the post footer.
- mhartl 14y ago
  Excellent.

ludwigvan 14y ago

Here is a recent video of DHH, discussing some of the techniques and issues mentioned in this article, such as backbone vs rails, pjax, Russian doll caching: http://www.youtube.com/watch?v=FkLVl3gpJP4 http://www.youtube.com/watch?v=FkLVl3gpJP4

davemo 14y ago

As someone relatively new to Rails who is working on a "near" realtime system this was great, thanks for the article. Wanted to point out that you can simplify your "folder count updating" code by using a relatively unused jQuery function: ".load" [1]. $.get($section.attr('data-url')) .success(function (response) { $section.html(response); }); would become $section.load($section.data('url')); [1] http://jqapi.com/#p=load http://jqapi.com/#p=load

23david 14y ago

Congrats for successfully getting your rails site to load faster, but it's still slow. Honestly, what you're doing at this point is delaying the big rewrite. That's not a bad thing, but it will still be necessary at some point to offload your real-time features to systems that are intended for those kinds of things from the ground up. Rails was fundamentally designed to optimize for programmer productivity over speed of execution, with the idea that people generally cost more than hardware. Unfortunately once you get to serious scale the hardware necessary to keep things quick starts costing a lot and operationally it becomes a big headache and stuff just doesn't work. It's going to take a seismic shift for rails to suddenly outperform lightweight 'real-time' frameworks running on faster scripting languages. We need to be honest and aware of both the benefits as well as the limitations of our tools.

batista 14y ago
>Congrats for successfully getting your rails site to load faster, but it's still slow. Looks pretty snappy to me. >Honestly, what you're doing at this point is delaying the big rewrite. That sounds brilliant! As in: http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html
- 23david 14y ago
  That's definitely a classic article. It just doesn't apply here. Hard to explain why without going into a ton of details, but basically: Layervault in 2012 != Netscape in 1996 Netscape's crufty spaghetti C code != Layervault's elegant but slow Rails code

alexquez 14y ago

Hey guys you mentioned that you have the observer send a message to the room. How does the Rails code send a message to the node server?

matthuggins 14y ago
This is along the same lines as a question I asked here too. Hopefully we'll see at least one of ours answered, as I'm curious as well. :)

matthuggins 14y ago

You made two points that sound contradictory without further explanation, if you don't mind providing more insight. First, you mentioned that you want to stick with Rails without duplicating views, etc. Second, you mention that you're using Socket.IO (i.e.: Node.JS as opposed to Ruby/Rails). How are you integrating Socket.IO with your Rails app? Do you have a separate Node.JS server that is just glossed over in your post, or do you use something like execjs to integrate Socket.IO directly into your Rails app? Thanks for sharing!