3 ms·

I am considering this stack off the shelf in my next big project: - uWSGI - performs better than gunicorn and has support for async apps using gevent - ngini

I am considering this stack off the shelf in my next big project: - uWSGI - performs better than gunicorn and has support for async apps using gevent - nginix - front end server - pyramid - web framework - mongodb - database - mongoengine - mongodb and python mapper - zeromq - messaging and communication - jinja2 - for template engine - gevent - for async processing - gevent-zeromq - to make zeromq non-blocking and gevent compatible - socket-io - JS lib for realtime communication I still need to develop robust session management. I considered various options and came to conclusion if I want something fast, truly distributed and not using sticky session I should come up with my own session manager demon hosted on each node. I would use ZeroMQ to communicate to it.

shazow 15y ago
Have you considered using Beaker with one of these backend extensions? https://github.com/didip/beaker_extensions https://github.com/didip/beaker_extensions I personally use Redis, but you could shove your sessions into Mongodb since you're already using it. Otherwise my preferred stack is very similar to yours, except I use Mako for templates and PostgreSQL/Redis for backend storage.
- antimora 15y ago
  Thanks. Actually I did consider Beaker it didn't fulfill my requirement. I wanted to replicate sessions across nodes actively and async, and also I wanted to persist to MongoDB async. So here is how I was considering to build: - sessions updated and validated by a demon process per node. - each validation and update will be one call via ZeroMQ's Req/Rep pattern. With each call I can validate session and reset timestamp. - after each validation I asynchly will replicate session across other nodes via ZeroMQ's Pub/Sub ( I don't care about extra memory) - sessions will also persist in MongoDB (async), just in case each node is restarted, thus preserving session. Btw, I only want to valid/invalidate a session token and keep authorization information. Any other small values I could simply keep in the cookies encrypted, e.g. User ID. Though I could keep session information in cookie as well, but this allows sessions to live forever and it's not good if I want to kick some user out.
  - bigethan 15y ago
    What's the use case that raises these requirements? Or, more politely, what problem are you solving with this setup?
    - antimora 15y ago
      Primarily I wanted it to be really fast, be able to invalidate session, and be available on any node via active replication. If I used Beaker, I had to use memcached which is not replicated (by default setup), and each validation would have required 2 round trips to memchached (validation, update stamp). It's easier to write my own with ZeroMQ and I can build with custom logic at demon level.
      - espeed 15y ago
        Awhile back I looked into Mongo for sessions, but it isn't really designed to do this. Sessions are temporal, and AFAIK Mongo doesn't have a built in mechanism for expiring them, plus session data is usually small -- there are other datastores better suited for this type of data.
        antimora 15y ago
        Primarily sessions will live in RAM. And I was going to use MongoDB to persist sessions just in case the daemon dies or session is not found in RAM as a fallback. The reason for using MongoDB is because it's my primary database. But your concern about growing table of sessions is valid, which I will handle by periodically archiving the old sessions so that index sizes remain small.
      - shazow 15y ago
        You don't have to use memcached, the beaker extensions let you use whichever backend storage mechanism you want. Redis Cluster can be replicated and would work fine for this, for example. I don't have a good grasp of what you're trying to achieve and why, but my overengineering alarms are going off.
bigethan 15y ago
is the uWSGI choice a purely based on performance? I like gunicorn for the simplicity, uWSGI has tons of options thought I wonder if they make management more difficult.
- ConceitedCode 15y ago
  I use it for the performance. It takes a little time to learn but it's fairly easy to use.
- antimora 15y ago
  Yes, it was primarily because of the performance. According to these benchmarks, unicorn totally choked: "... At the bottom we have Twisted and Gunicorn ..." http://nichol.as/benchmark-of-python-web-servers http://nichol.as/benchmark-of-python-web-servers
  - cd34 15y ago
    http://www.reddit.com/r/Python/comments/be1q1/benchmark_of_python_web_servers/ http://www.reddit.com/r/Python/comments/be1q1/benchmark_of_p...
  - mardiros 15y ago
    Look at the comment, the twisted code is wrong. By default the twisted reactor is multiplatform, but you can imporve performance for your specific platform. It's in the doc. If you run Twisted under linux and you should, use the epoll reactor.
  - deleted 15y ago
    [deleted]
deleted 15y ago
[deleted]
mtogo 15y ago
Are you worried about Mongo's reliability issues? I like Mongo, but i don't know that i could accept the risk on a project i really care about.
- antimora 15y ago
  Initially I was, because I am coming from the Oracle world. But for the performance gain, I can give up ACID compliance, which is solvable with additional code complexity. For extra reliability in MongoDB, I am thinking to use replica sets (active/passive) and journaling.
  - mtogo 15y ago
    Interesting, thanks.
antihero 15y ago
Hmm, would be interesting to benchmark uWSGI against Gunicorn with Meinheld workers.

shazow 15y ago

Have you considered using Beaker with one of these backend extensions? https://github.com/didip/beaker_extensions https://github.com/didip/beaker_extensions I personally use Redis, but you could shove your sessions into Mongodb since you're already using it. Otherwise my preferred stack is very similar to yours, except I use Mako for templates and PostgreSQL/Redis for backend storage.

antimora 15y ago
Thanks. Actually I did consider Beaker it didn't fulfill my requirement. I wanted to replicate sessions across nodes actively and async, and also I wanted to persist to MongoDB async. So here is how I was considering to build: - sessions updated and validated by a demon process per node. - each validation and update will be one call via ZeroMQ's Req/Rep pattern. With each call I can validate session and reset timestamp. - after each validation I asynchly will replicate session across other nodes via ZeroMQ's Pub/Sub ( I don't care about extra memory) - sessions will also persist in MongoDB (async), just in case each node is restarted, thus preserving session. Btw, I only want to valid/invalidate a session token and keep authorization information. Any other small values I could simply keep in the cookies encrypted, e.g. User ID. Though I could keep session information in cookie as well, but this allows sessions to live forever and it's not good if I want to kick some user out.
- bigethan 15y ago
  What's the use case that raises these requirements? Or, more politely, what problem are you solving with this setup?
  - antimora 15y ago
    Primarily I wanted it to be really fast, be able to invalidate session, and be available on any node via active replication. If I used Beaker, I had to use memcached which is not replicated (by default setup), and each validation would have required 2 round trips to memchached (validation, update stamp). It's easier to write my own with ZeroMQ and I can build with custom logic at demon level.
    - espeed 15y ago
      Awhile back I looked into Mongo for sessions, but it isn't really designed to do this. Sessions are temporal, and AFAIK Mongo doesn't have a built in mechanism for expiring them, plus session data is usually small -- there are other datastores better suited for this type of data.
      - antimora 15y ago
        Primarily sessions will live in RAM. And I was going to use MongoDB to persist sessions just in case the daemon dies or session is not found in RAM as a fallback. The reason for using MongoDB is because it's my primary database. But your concern about growing table of sessions is valid, which I will handle by periodically archiving the old sessions so that index sizes remain small.
    - shazow 15y ago
      You don't have to use memcached, the beaker extensions let you use whichever backend storage mechanism you want. Redis Cluster can be replicated and would work fine for this, for example. I don't have a good grasp of what you're trying to achieve and why, but my overengineering alarms are going off.

bigethan 15y ago

is the uWSGI choice a purely based on performance? I like gunicorn for the simplicity, uWSGI has tons of options thought I wonder if they make management more difficult.

ConceitedCode 15y ago
I use it for the performance. It takes a little time to learn but it's fairly easy to use.
antimora 15y ago
Yes, it was primarily because of the performance. According to these benchmarks, unicorn totally choked: "... At the bottom we have Twisted and Gunicorn ..." http://nichol.as/benchmark-of-python-web-servers http://nichol.as/benchmark-of-python-web-servers
- cd34 15y ago
  http://www.reddit.com/r/Python/comments/be1q1/benchmark_of_python_web_servers/ http://www.reddit.com/r/Python/comments/be1q1/benchmark_of_p...
- mardiros 15y ago
  Look at the comment, the twisted code is wrong. By default the twisted reactor is multiplatform, but you can imporve performance for your specific platform. It's in the doc. If you run Twisted under linux and you should, use the epoll reactor.
- deleted 15y ago
  [deleted]

deleted 15y ago

[deleted]

mtogo 15y ago

Are you worried about Mongo's reliability issues? I like Mongo, but i don't know that i could accept the risk on a project i really care about.

antimora 15y ago
Initially I was, because I am coming from the Oracle world. But for the performance gain, I can give up ACID compliance, which is solvable with additional code complexity. For extra reliability in MongoDB, I am thinking to use replica sets (active/passive) and journaling.
- mtogo 15y ago
  Interesting, thanks.

antihero 15y ago

Hmm, would be interesting to benchmark uWSGI against Gunicorn with Meinheld workers.