4 ms·

The C10K problem[1] is about real clients accessing real web servers, not about writing short strings over WebSocket. I'll associate "C10K" and "Django" when I

The C10K problem[1] is about real clients accessing real web servers, not about writing short strings over WebSocket. I'll associate "C10K" and "Django" when I'll see a dynamic web page being served using the template system, the standard middleware and the ORM (with the default behavior of creating and destroying a database connection for each request). [1] http://www.kegel.com/c10k.html http://www.kegel.com/c10k.html

mYk 14y ago
The C10k problem was originally about serving static files -- quoting the page you linked to: "take four kilobytes from the disk and send them to the network". The demo could certainly be extended to send larger amounts of data -- left as an exercise for the reader. I'm not sure to understand your second paragraph -- if I used this system in a real application, I would serve the pages with the traditional handler (with template rendering, middleware, etc.) and then exchange messages over the websocket. These are different roles. Regarding database connections, the default behavior isn't the one you're describing any more; I implemented persistent connections in Django a few weeks ago.
- stefantalpalaru 14y ago
  Static files are no longer a problem thanks to Varnish or nginx. And this is Django, after all, so please focus on dynamic content performance. I'm glad to hear that Django managed to get persistent db connections after years of "just use PgPool/PgBouncer". Keep up the good work!
  - mYk 14y ago
    Yes, I'm just referring to C10k because this demo uses a technique originally created to solve C10k. Reaching 10 000 connections wasn't difficult in this case; it was just a matter of tuning a few system parameters. Exploring the APIs and studying how they can fit together was much more interesting, and sometimes challenging.
    - calvinx 14y ago
      Awesome stuff!
  - raverbashing 14y ago
    Well, about PgBouncer, I believe this tells more of PostgreSQL than of Django + psql libraries (After all it will still open and close connections for Django but keeps them open for psql) Now, for serving 10k connections with template libraries and middlewares and ORM, out of the box? Serving dynamic content for each connection? Impossible =) Not without some kind of caching (but you can do that with the help of a middleware)
    - stefantalpalaru 14y ago
      It's not PostgreSQL's fault that Django closes the connection after each request explicitly (see close_connection() for what was done before 1.6): [1] https://github.com/django/django/blob/master/django/db/__init__.py https://github.com/django/django/blob/master/django/db/__ini...
      - raverbashing 14y ago
        I understand that, and I'm not saying that there aren't several issues in Django (but some are getting better) What I'm saying is that with PgBouncer you have Django <> PgBouncer <> Postgresql, right? So what PgBouncer is doing (dealing with several open/close connections) maybe could be done better inside PostgreSQL
- StavrosK 14y ago
  By "implemented", do you mean "contributed to trunk", or "used it in my application"? If the former, thanks, if the latter, I was under the impression that it was on by default in 1.5?
  - stefantalpalaru 14y ago
    Looks like it will be available in the next (1.6?) release[1]. [1] https://docs.djangoproject.com/en/dev/ref/databases/#persistent-connections https://docs.djangoproject.com/en/dev/ref/databases/#persist...
    - StavrosK 14y ago
      Oh right, 1.5 and earlier. I misread that, thank you.
vampirechicken 14y ago
Why would you insist on creating and destroying a db connection on each request? We stopped doing that decades ago.
- jaytaylor 14y ago
  Django hasn't stopped insisting on that... v1.5 might have pooling, but it's taken them a long time to get connection pooling in.
  - obviouslygreen 14y ago
    Forgive my ignorance, but isn't this the point of something like pgpool? I didn't think connecting to a connection pool would be as expensive as connecting to the database itself (if it is, then my next question would be: What's the point of a connection pool? Because clearly I've missed something one way or the other).
    - jaytaylor 14y ago
      Yes, but pgpool isn't always an option. If you use Heroku's Postgres service, you do not have sufficient privileges to run the commands to attach to software like pgpool. The only way we were able to get connection pooling working in django 1.3/1.4 was to use django-dbpool[0]. It works okay, but is still pretty sketchy compared to the connection pooling libraries available for the JVM like BoneCP or C3P0. [0] https://github.com/gmcguire/django-db-pool https://github.com/gmcguire/django-db-pool
      - talmirza 14y ago
        You can still us pg_pool with heroku.
      - obviouslygreen 14y ago
        Understood, thank you for the clarification.
    - stefantalpalaru 14y ago

mYk 14y ago

The C10k problem was originally about serving static files -- quoting the page you linked to: "take four kilobytes from the disk and send them to the network". The demo could certainly be extended to send larger amounts of data -- left as an exercise for the reader. I'm not sure to understand your second paragraph -- if I used this system in a real application, I would serve the pages with the traditional handler (with template rendering, middleware, etc.) and then exchange messages over the websocket. These are different roles. Regarding database connections, the default behavior isn't the one you're describing any more; I implemented persistent connections in Django a few weeks ago.

stefantalpalaru 14y ago
Static files are no longer a problem thanks to Varnish or nginx. And this is Django, after all, so please focus on dynamic content performance. I'm glad to hear that Django managed to get persistent db connections after years of "just use PgPool/PgBouncer". Keep up the good work!
- mYk 14y ago
  Yes, I'm just referring to C10k because this demo uses a technique originally created to solve C10k. Reaching 10 000 connections wasn't difficult in this case; it was just a matter of tuning a few system parameters. Exploring the APIs and studying how they can fit together was much more interesting, and sometimes challenging.
  - calvinx 14y ago
    Awesome stuff!
- raverbashing 14y ago
  Well, about PgBouncer, I believe this tells more of PostgreSQL than of Django + psql libraries (After all it will still open and close connections for Django but keeps them open for psql) Now, for serving 10k connections with template libraries and middlewares and ORM, out of the box? Serving dynamic content for each connection? Impossible =) Not without some kind of caching (but you can do that with the help of a middleware)
  - stefantalpalaru 14y ago
    It's not PostgreSQL's fault that Django closes the connection after each request explicitly (see close_connection() for what was done before 1.6): [1] https://github.com/django/django/blob/master/django/db/__init__.py https://github.com/django/django/blob/master/django/db/__ini...
    - raverbashing 14y ago
      I understand that, and I'm not saying that there aren't several issues in Django (but some are getting better) What I'm saying is that with PgBouncer you have Django <> PgBouncer <> Postgresql, right? So what PgBouncer is doing (dealing with several open/close connections) maybe could be done better inside PostgreSQL
StavrosK 14y ago
By "implemented", do you mean "contributed to trunk", or "used it in my application"? If the former, thanks, if the latter, I was under the impression that it was on by default in 1.5?
- stefantalpalaru 14y ago
  Looks like it will be available in the next (1.6?) release[1]. [1] https://docs.djangoproject.com/en/dev/ref/databases/#persistent-connections https://docs.djangoproject.com/en/dev/ref/databases/#persist...
  - StavrosK 14y ago
    Oh right, 1.5 and earlier. I misread that, thank you.

vampirechicken 14y ago

Why would you insist on creating and destroying a db connection on each request? We stopped doing that decades ago.

jaytaylor 14y ago
Django hasn't stopped insisting on that... v1.5 might have pooling, but it's taken them a long time to get connection pooling in.
- obviouslygreen 14y ago
  Forgive my ignorance, but isn't this the point of something like pgpool? I didn't think connecting to a connection pool would be as expensive as connecting to the database itself (if it is, then my next question would be: What's the point of a connection pool? Because clearly I've missed something one way or the other).
  - jaytaylor 14y ago
    Yes, but pgpool isn't always an option. If you use Heroku's Postgres service, you do not have sufficient privileges to run the commands to attach to software like pgpool. The only way we were able to get connection pooling working in django 1.3/1.4 was to use django-dbpool[0]. It works okay, but is still pretty sketchy compared to the connection pooling libraries available for the JVM like BoneCP or C3P0. [0] https://github.com/gmcguire/django-db-pool https://github.com/gmcguire/django-db-pool
    - talmirza 14y ago
      You can still us pg_pool with heroku.
    - obviouslygreen 14y ago
      Understood, thank you for the clarification.
  - stefantalpalaru 14y ago