9 ms·

10 000 concurrent real-time connections to Django

186 points by mYk 14y ago

kapilvt 14y ago
why do people persist in mangling django into doing websocket persitent concurrency when there are better py or otherwise well suited tools to such tasks...
- LeafStorm 14y ago
  Because it's a popular, well-written framework with a huge volume of existing code, and integrating WebSockets into Django means you don't have to maintain two codebases if you want to have a realtime Django app. Also, it's fun.
  - kapilvt 14y ago
    how much of that existing code has any application in a websocket context.. if it uses the orm layer throw it out.. if it uses the template layer.. optionally throw it out for efficiency (jinja and friends).. what are you really left with... routing.. trivial.. a familiar api with the guts out.. overrated and a hobgoblin of consistency.
    - mYk 14y ago
      This system doesn't replace Django; it complements it. You could build 95% of an application with the traditional request-response model and add the 5% of real-time featurs with a system similar to my demo. (Of course, given your opinions on Django, I don't recommend you build anything with it.)
      - eyepulp 14y ago
        Well said =) It's incredibly frustrating to so easily build the entire app with a "traditional" django stack and then be faced with solving for a whole new stack just to avoid wasteful XHR-polling for some simple server-side event driven UI updates. We've implemented some SSE based solutions lately with gevent & nginx in front of django, and it's been great to keep it all in the django family. Maybe switching to py3k has some value after all... =)
        notdonspaulding 14y ago
        I concur.
    - jvdongen 14y ago
      In addition to templating, orm and routing there is also highly integrated i18n toolchain/workflow, built-in csrf protection, good session handling, well integrated caching and many more of such 'details'. All those things by themselves could be considered trivial and could be gotten from many individual libraries - the level of integration and polish you encounter with Django is anything but trivial though and takes real time and effort. I've recently had to make a choice for a Python based application platform/environment and have chosen Django (again) despite having no use for the ORM and ORM-using contrib modules. Simply because all the other things are there and work together beautifully without me spending any time on code that is not directly related to my goals.
    - etchalon 14y ago
      Trivial or not, working within an existing library provides a level of maintainability you just don't get when you toss in something like Tornado to do the tiny bit of your app that requires a persistent connection and websockets. Even if all I got was django's router and views, it's save me the awful headache of having to graft Tornado and Django together, coding under two entirely different library sets.
    - bayesianhorse 14y ago
      In my opinion there is nothing wrong with doing simple Django ORM calls even with a tornado/gevent websocket. Async purists will cry foul, but if all you want is a realtime feature, you can block for a few milliseconds without getting into trouble. In the long run they might build tulip-awareness into the ORM. For that matter I start Tornado servers with management commands just for convenience...
      - twism 14y ago
        Or you could do this ... https://gist.github.com/anonymous/5190528 https://gist.github.com/anonymous/5190528 This blocks in a separate thread leaving the event loop to handle other tasks.
  - mYk 14y ago
    Indeed, these are the exact reasons!
- _fs 14y ago
  What would you recommend as the best tools for accomplishing this?
stefantalpalaru 14y ago
The C10K problem[1] is about real clients accessing real web servers, not about writing short strings over WebSocket. I'll associate "C10K" and "Django" when I'll see a dynamic web page being served using the template system, the standard middleware and the ORM (with the default behavior of creating and destroying a database connection for each request). [1] http://www.kegel.com/c10k.html http://www.kegel.com/c10k.html
- mYk 14y ago
  The C10k problem was originally about serving static files -- quoting the page you linked to: "take four kilobytes from the disk and send them to the network". The demo could certainly be extended to send larger amounts of data -- left as an exercise for the reader. I'm not sure to understand your second paragraph -- if I used this system in a real application, I would serve the pages with the traditional handler (with template rendering, middleware, etc.) and then exchange messages over the websocket. These are different roles. Regarding database connections, the default behavior isn't the one you're describing any more; I implemented persistent connections in Django a few weeks ago.
  - stefantalpalaru 14y ago
    Static files are no longer a problem thanks to Varnish or nginx. And this is Django, after all, so please focus on dynamic content performance. I'm glad to hear that Django managed to get persistent db connections after years of "just use PgPool/PgBouncer". Keep up the good work!
    - mYk 14y ago
      Yes, I'm just referring to C10k because this demo uses a technique originally created to solve C10k. Reaching 10 000 connections wasn't difficult in this case; it was just a matter of tuning a few system parameters. Exploring the APIs and studying how they can fit together was much more interesting, and sometimes challenging.
      - 14y ago
eliben 14y ago
Very cool, well done.
bayesianhorse 14y ago
I dabbled in realtime connections for a while now, and this is certainly a sexy option to avoid the fragmentation happening in the Python async world.
- hoov 14y ago
  Well, I think that was Guido's goal in creating Tulip.
est 14y ago
For Python2.X, I think gevent.wsgi.WSGIServer could handle C10k or more, no?
- fafhrd91 14y ago
  Potentially it is slower than tulip, It does magic stack slicing. also it works only on X86 cpu.
- lucian1900 14y ago
  Or just Twisted.
- hoov 14y ago
  In my experience -- yes. I've pushed it further than that. In practice, debugging is an unpleasant experience. If you're doing anything slightly out of the ordinary, you should take care.

186 points by mYk 14y ago

why do people persist in mangling django into doing websocket persitent concurrency when there are better py or otherwise well suited tools to such tasks...

LeafStorm 14y ago
Because it's a popular, well-written framework with a huge volume of existing code, and integrating WebSockets into Django means you don't have to maintain two codebases if you want to have a realtime Django app. Also, it's fun.
- kapilvt 14y ago
  how much of that existing code has any application in a websocket context.. if it uses the orm layer throw it out.. if it uses the template layer.. optionally throw it out for efficiency (jinja and friends).. what are you really left with... routing.. trivial.. a familiar api with the guts out.. overrated and a hobgoblin of consistency.
  - mYk 14y ago
    This system doesn't replace Django; it complements it. You could build 95% of an application with the traditional request-response model and add the 5% of real-time featurs with a system similar to my demo. (Of course, given your opinions on Django, I don't recommend you build anything with it.)
    - eyepulp 14y ago
      Well said =) It's incredibly frustrating to so easily build the entire app with a "traditional" django stack and then be faced with solving for a whole new stack just to avoid wasteful XHR-polling for some simple server-side event driven UI updates. We've implemented some SSE based solutions lately with gevent & nginx in front of django, and it's been great to keep it all in the django family. Maybe switching to py3k has some value after all... =)
      - notdonspaulding 14y ago
        I concur.
  - jvdongen 14y ago
    In addition to templating, orm and routing there is also highly integrated i18n toolchain/workflow, built-in csrf protection, good session handling, well integrated caching and many more of such 'details'. All those things by themselves could be considered trivial and could be gotten from many individual libraries - the level of integration and polish you encounter with Django is anything but trivial though and takes real time and effort. I've recently had to make a choice for a Python based application platform/environment and have chosen Django (again) despite having no use for the ORM and ORM-using contrib modules. Simply because all the other things are there and work together beautifully without me spending any time on code that is not directly related to my goals.
  - etchalon 14y ago
    Trivial or not, working within an existing library provides a level of maintainability you just don't get when you toss in something like Tornado to do the tiny bit of your app that requires a persistent connection and websockets. Even if all I got was django's router and views, it's save me the awful headache of having to graft Tornado and Django together, coding under two entirely different library sets.
  - bayesianhorse 14y ago
    In my opinion there is nothing wrong with doing simple Django ORM calls even with a tornado/gevent websocket. Async purists will cry foul, but if all you want is a realtime feature, you can block for a few milliseconds without getting into trouble. In the long run they might build tulip-awareness into the ORM. For that matter I start Tornado servers with management commands just for convenience...
    - twism 14y ago
      Or you could do this ... https://gist.github.com/anonymous/5190528 https://gist.github.com/anonymous/5190528 This blocks in a separate thread leaving the event loop to handle other tasks.
- mYk 14y ago
  Indeed, these are the exact reasons!
_fs 14y ago
What would you recommend as the best tools for accomplishing this?

stefantalpalaru 14y ago

The C10K problem[1] is about real clients accessing real web servers, not about writing short strings over WebSocket. I'll associate "C10K" and "Django" when I'll see a dynamic web page being served using the template system, the standard middleware and the ORM (with the default behavior of creating and destroying a database connection for each request). [1] http://www.kegel.com/c10k.html http://www.kegel.com/c10k.html

mYk 14y ago
The C10k problem was originally about serving static files -- quoting the page you linked to: "take four kilobytes from the disk and send them to the network". The demo could certainly be extended to send larger amounts of data -- left as an exercise for the reader. I'm not sure to understand your second paragraph -- if I used this system in a real application, I would serve the pages with the traditional handler (with template rendering, middleware, etc.) and then exchange messages over the websocket. These are different roles. Regarding database connections, the default behavior isn't the one you're describing any more; I implemented persistent connections in Django a few weeks ago.
- stefantalpalaru 14y ago
  Static files are no longer a problem thanks to Varnish or nginx. And this is Django, after all, so please focus on dynamic content performance. I'm glad to hear that Django managed to get persistent db connections after years of "just use PgPool/PgBouncer". Keep up the good work!
  - mYk 14y ago
    Yes, I'm just referring to C10k because this demo uses a technique originally created to solve C10k. Reaching 10 000 connections wasn't difficult in this case; it was just a matter of tuning a few system parameters. Exploring the APIs and studying how they can fit together was much more interesting, and sometimes challenging.
    - 14y ago

eliben 14y ago

Very cool, well done.

bayesianhorse 14y ago

I dabbled in realtime connections for a while now, and this is certainly a sexy option to avoid the fragmentation happening in the Python async world.

hoov 14y ago
Well, I think that was Guido's goal in creating Tulip.

est 14y ago

For Python2.X, I think gevent.wsgi.WSGIServer could handle C10k or more, no?

fafhrd91 14y ago
Potentially it is slower than tulip, It does magic stack slicing. also it works only on X86 cpu.
lucian1900 14y ago
Or just Twisted.
hoov 14y ago
In my experience -- yes. I've pushed it further than that. In practice, debugging is an unpleasant experience. If you're doing anything slightly out of the ordinary, you should take care.