5 ms·

Asynchronous Processing in Python Using Redis

42 points by rlander 16y ago

samratjp 16y ago
Looks a lot like GitHub's Resque(Ruby) https://github.com/defunkt/resque https://github.com/defunkt/resque hopefully, this python variant will get all the good stuff that comes with Resque (like a Sinatra app to see your queues - very handy).
- deleted 16y ago
  [deleted]
- rlander 16y ago
  If you're looking for something closer to resque, there's pyres (https://github.com/binarydud/pyres https://github.com/binarydud/pyres), which is pretty cool too.
freyrs3 16y ago
Anyone have some insight on how this would compare performance-wise with Celery + RabbitMQ?
- samratjp 16y ago
  Though I don't have direct experience with this particular one, I can tell you it's similarly implemented to Resque (the ruby async library). In my experience, Resque has been wicked fast thanks to Redis' excellent in-memory database. From my personal use cases, Redis has been pretty fast on both python and ruby. It really depends on what you are trying to do really. If you don't want additional moving parts and have Redis installed, I suppose this will do the job fine.
- praxxis 16y ago
  The development version of Celery also supports Redis as a queue: http://ask.github.com/celery/tutorials/otherqueues.html http://ask.github.com/celery/tutorials/otherqueues.html
- simonw 16y ago
  I'm using Celery with the Redis backend for http://lanyrd.com/ http://lanyrd.com/
  - frankwiles 16y ago
    We're also using that with several clients.
- piotrSikora 16y ago
  It doesn't really compare. HotQueue is just a queue. Celery is full-blown task queue (with periodic tasks, task retries, monitoring, etc) that supports multiple backends (AMPQ, Redis, SQL databases) and even integrates nicely with existing python web frameworks (Django, Flask).
wildmXranat 16y ago
I still like a generic job server like gearman better. You can do sync, async, back the queue with memory, mysql, tokyo cabinet ... and if you have a need of PHP clients talking to Python consumers no, problem as well. Those libraries exist.
unoti 16y ago
I use Python + Redis using Hotqueue (http://pypi.python.org/pypi/hotqueue/0.2.1 http://pypi.python.org/pypi/hotqueue/0.2.1) for all the processing of flurbils.com. All of the scoring for the game comes in from Second Life via http requests that are handed off immediately to a background handler via Redis. It's enabled me to seriously level out load spikes, and enabled me to push off having to upgrade the capacity of my servers as a result. I've used other libraries for doing this in the past. Being able to switch backends using some other library would be nice, I suppose, but there is literally only about 6 lines of code in the software that is specific to hotqueue. This has been working very well in production, and I'm very pleased with it. Using background processing to level out load peaks, plus migrating to Apache worker MPM has enabled me to handle loads on the same hardware that I wouldn't be able to touch using the basic normal approach. Starting off using Redis as a queue is a great way to start with Redis, because everybody can use a queue for something. Then once Redis is running in production, it becomes far easier to start using it for more and more things.

42 points by rlander 16y ago

Looks a lot like GitHub's Resque(Ruby) https://github.com/defunkt/resque https://github.com/defunkt/resque hopefully, this python variant will get all the good stuff that comes with Resque (like a Sinatra app to see your queues - very handy).

deleted 16y ago
[deleted]
rlander 16y ago
If you're looking for something closer to resque, there's pyres (https://github.com/binarydud/pyres https://github.com/binarydud/pyres), which is pretty cool too.

freyrs3 16y ago

Anyone have some insight on how this would compare performance-wise with Celery + RabbitMQ?

samratjp 16y ago
Though I don't have direct experience with this particular one, I can tell you it's similarly implemented to Resque (the ruby async library). In my experience, Resque has been wicked fast thanks to Redis' excellent in-memory database. From my personal use cases, Redis has been pretty fast on both python and ruby. It really depends on what you are trying to do really. If you don't want additional moving parts and have Redis installed, I suppose this will do the job fine.
praxxis 16y ago
The development version of Celery also supports Redis as a queue: http://ask.github.com/celery/tutorials/otherqueues.html http://ask.github.com/celery/tutorials/otherqueues.html
simonw 16y ago
I'm using Celery with the Redis backend for http://lanyrd.com/ http://lanyrd.com/
- frankwiles 16y ago
  We're also using that with several clients.
piotrSikora 16y ago
It doesn't really compare. HotQueue is just a queue. Celery is full-blown task queue (with periodic tasks, task retries, monitoring, etc) that supports multiple backends (AMPQ, Redis, SQL databases) and even integrates nicely with existing python web frameworks (Django, Flask).

wildmXranat 16y ago

I still like a generic job server like gearman better. You can do sync, async, back the queue with memory, mysql, tokyo cabinet ... and if you have a need of PHP clients talking to Python consumers no, problem as well. Those libraries exist.

unoti 16y ago

I use Python + Redis using Hotqueue (http://pypi.python.org/pypi/hotqueue/0.2.1 http://pypi.python.org/pypi/hotqueue/0.2.1) for all the processing of flurbils.com. All of the scoring for the game comes in from Second Life via http requests that are handed off immediately to a background handler via Redis. It's enabled me to seriously level out load spikes, and enabled me to push off having to upgrade the capacity of my servers as a result. I've used other libraries for doing this in the past. Being able to switch backends using some other library would be nice, I suppose, but there is literally only about 6 lines of code in the software that is specific to hotqueue. This has been working very well in production, and I'm very pleased with it. Using background processing to level out load peaks, plus migrating to Apache worker MPM has enabled me to handle loads on the same hardware that I wouldn't be able to touch using the basic normal approach. Starting off using Redis as a queue is a great way to start with Redis, because everybody can use a queue for something. Then once Redis is running in production, it becomes far easier to start using it for more and more things.