5 ms·

Ginger Tech Stack (Realtime + Django)

93 points by ipmb 14y ago

clivestaples 14y ago
Very interesting. I was agonizing over the weekend about which stack to use for a new project. I know Rails really well, but Express/Node.js is so hard to ignore these days. This solution seems like the best of both worlds, but I'm still not sure how it works: In the described scenario, Django loads to initial page with the Backbone.js/Socket.IO code and Django continues to handle all post/put/deletes while Socket.IO delivers all the "get" requests via web sockets?
- ipmb 14y ago
  Yeah, I think you've got it. 1. Initial page load dumps in as much data as possible to get things started. 2. All writes go through the Django API (we just look for a success/fail response). 3. All new/real-time data gets pushed to the client via socket.io. This includes data that you yourself created. #1 is kind of a given, but #2 & #3 are more personal preference/app-specific I think. We could have done #2 through socket.io, but as noted in the post, Python is our primary language and we wanted to push as much of the logic to it as we could. At first we had content creators reading responses from the API with all other team members getting their updates from socket.io. This was problematic because creators would get content twice (once over the update channel for the team and once from the API). It resulted in some strange race conditions and a lot of "have I already received this message?" code. In the end, it was easiest for us to just always defer to the data from the socket (which was faster as well).
  - clivestaples 14y ago
    This makes a lot of sense and I look forward to trying this out in my own project. Thanks.
  - amackera 14y ago
    What is your system for pub/sub from the client's perpective? Do you have matching models on Backbone & Django? I guess what I'm interested in is how to know what users to route what data to via socket.io. Great stack! We use something very similar.
    - ipmb 14y ago
      The client passes a unique key with a short TTL to the Node.js server that maps them to the proper user and teams for the duration of their session.
- danneu 14y ago
  I'm too am comfortable with Rails and the Ruby ecosystem but jumped into Express recently with Socket.io and MongoDB and think you should too. My curiosity was shortly satisfied and I decided to stick with synchronous code instead dealing with callbacks. For a moment there, I was high with "I'll use Node with everything!". But the experienced inspired me to play with Ruby's EventMachine, Cramp, Goliath, and implementations like socketio-ruby on Cramp. I also took some time out of my life to really dive in to Javascript and experience some new tools.
zaidos 14y ago
This stack has no soul.
- whalesalad 14y ago
  This made me chuckle. Unfortunately most of the people on HN will probably take it the wrong way.
- younata 14y ago
  I get that you want to make a joke. I also think you need to understand it's not a joke. The fact that there exists "kick a ginger day" [0] is proof that it's not a joke. Abuse is not funny, ever. [0] http://laist.com/2009/11/24/kick_a_ginger_day_spawned_at_least.php http://laist.com/2009/11/24/kick_a_ginger_day_spawned_at_lea...
frankwiles 14y ago
Looks great, congrats on the launch Pete!
- ipmb 14y ago
  Thanks Frank! I certainly can't take the credit though. Almost everyone on our team (http://lincolnloop.com/about/ http://lincolnloop.com/about/) has committed to the project repo.
Alexandervn 14y ago
Have you considered Mongrel2 or ZeroMQ for your stack?
- ipmb 14y ago
  Not much more than a cursory look. It sounds like they aim to solve some of the technical challenges we encountered though.
deleted 14y ago
[deleted]
neilk 14y ago
I saw the Ginger video demo, where is any real-time functionality needed?
- lopatin 14y ago
  There were real time discussion updates in the video
- ipmb 14y ago
  As the product evolved, the real-time needs (surprisingly) became less than we originally expected. I think this will change as we continue to evolve. That being said, after the initial page load, almost everything happens without a page refresh or full page load. You see new messages (with favicon notification), votes, unread counts, etc. all in real-time. I think Ginger would feel a lot klunkier without the real-time bits.
bdr 14y ago
Did you try tornadio2?
- ipmb 14y ago
  AFAIK it was released after we settled on Node.js.
  - garindra 14y ago
    We use Tornadio2 along with Pika (with the IOLoop sharing) for an internal project and it's been great.
- sciurus 14y ago
  This is the first time I've heard of tornadio2; it turns out there is a list of other socket.io server implementations at https://github.com/learnboost/socket.io/wiki/ https://github.com/learnboost/socket.io/wiki/
waterside81 14y ago
What hit me about this article is that I employ almost an identical stack and have never really stepped back and taken a look at just how many pieces are required to make a non-trivial web app. When you're working with these tools day-in, day-out, you get almost institutionalized and forget the complexities that you've absorbed over the years. A side note: it's a testament to the great work the open-source community has put in over the past decade (two decades?) or so that so many pieces can come together and work relatively seamlessly.
swah 14y ago
Is using a queue for calling Node.js the better/proper thing to do here? I thought this was the exact use case of Thrift. Thanks for the post, I wish more companies would describe their stack at least in a high-level way, so we can know what is working "in production".

93 points by ipmb 14y ago

Very interesting. I was agonizing over the weekend about which stack to use for a new project. I know Rails really well, but Express/Node.js is so hard to ignore these days. This solution seems like the best of both worlds, but I'm still not sure how it works: In the described scenario, Django loads to initial page with the Backbone.js/Socket.IO code and Django continues to handle all post/put/deletes while Socket.IO delivers all the "get" requests via web sockets?

ipmb 14y ago
Yeah, I think you've got it. 1. Initial page load dumps in as much data as possible to get things started. 2. All writes go through the Django API (we just look for a success/fail response). 3. All new/real-time data gets pushed to the client via socket.io. This includes data that you yourself created. #1 is kind of a given, but #2 & #3 are more personal preference/app-specific I think. We could have done #2 through socket.io, but as noted in the post, Python is our primary language and we wanted to push as much of the logic to it as we could. At first we had content creators reading responses from the API with all other team members getting their updates from socket.io. This was problematic because creators would get content twice (once over the update channel for the team and once from the API). It resulted in some strange race conditions and a lot of "have I already received this message?" code. In the end, it was easiest for us to just always defer to the data from the socket (which was faster as well).
- clivestaples 14y ago
  This makes a lot of sense and I look forward to trying this out in my own project. Thanks.
- amackera 14y ago
  What is your system for pub/sub from the client's perpective? Do you have matching models on Backbone & Django? I guess what I'm interested in is how to know what users to route what data to via socket.io. Great stack! We use something very similar.
  - ipmb 14y ago
    The client passes a unique key with a short TTL to the Node.js server that maps them to the proper user and teams for the duration of their session.
danneu 14y ago
I'm too am comfortable with Rails and the Ruby ecosystem but jumped into Express recently with Socket.io and MongoDB and think you should too. My curiosity was shortly satisfied and I decided to stick with synchronous code instead dealing with callbacks. For a moment there, I was high with "I'll use Node with everything!". But the experienced inspired me to play with Ruby's EventMachine, Cramp, Goliath, and implementations like socketio-ruby on Cramp. I also took some time out of my life to really dive in to Javascript and experience some new tools.

zaidos 14y ago

This stack has no soul.

whalesalad 14y ago
This made me chuckle. Unfortunately most of the people on HN will probably take it the wrong way.
younata 14y ago
I get that you want to make a joke. I also think you need to understand it's not a joke. The fact that there exists "kick a ginger day" [0] is proof that it's not a joke. Abuse is not funny, ever. [0] http://laist.com/2009/11/24/kick_a_ginger_day_spawned_at_least.php http://laist.com/2009/11/24/kick_a_ginger_day_spawned_at_lea...

frankwiles 14y ago

Looks great, congrats on the launch Pete!

ipmb 14y ago
Thanks Frank! I certainly can't take the credit though. Almost everyone on our team (http://lincolnloop.com/about/ http://lincolnloop.com/about/) has committed to the project repo.

Alexandervn 14y ago

Have you considered Mongrel2 or ZeroMQ for your stack?

ipmb 14y ago
Not much more than a cursory look. It sounds like they aim to solve some of the technical challenges we encountered though.

deleted 14y ago

[deleted]

neilk 14y ago

I saw the Ginger video demo, where is any real-time functionality needed?

lopatin 14y ago
There were real time discussion updates in the video
ipmb 14y ago
As the product evolved, the real-time needs (surprisingly) became less than we originally expected. I think this will change as we continue to evolve. That being said, after the initial page load, almost everything happens without a page refresh or full page load. You see new messages (with favicon notification), votes, unread counts, etc. all in real-time. I think Ginger would feel a lot klunkier without the real-time bits.

bdr 14y ago

Did you try tornadio2?

ipmb 14y ago
AFAIK it was released after we settled on Node.js.
- garindra 14y ago
  We use Tornadio2 along with Pika (with the IOLoop sharing) for an internal project and it's been great.
sciurus 14y ago
This is the first time I've heard of tornadio2; it turns out there is a list of other socket.io server implementations at https://github.com/learnboost/socket.io/wiki/ https://github.com/learnboost/socket.io/wiki/

waterside81 14y ago

What hit me about this article is that I employ almost an identical stack and have never really stepped back and taken a look at just how many pieces are required to make a non-trivial web app. When you're working with these tools day-in, day-out, you get almost institutionalized and forget the complexities that you've absorbed over the years. A side note: it's a testament to the great work the open-source community has put in over the past decade (two decades?) or so that so many pieces can come together and work relatively seamlessly.

swah 14y ago

Is using a queue for calling Node.js the better/proper thing to do here? I thought this was the exact use case of Thrift. Thanks for the post, I wish more companies would describe their stack at least in a high-level way, so we can know what is working "in production".