7 ms·
> Multiprocess, not Multi-thread or Gevent - which is built on libev and provides constructs like queues, etc to make your multi-processing life much better.
by sandGorgon 9y ago
> Multiprocess, not Multi-thread
or Gevent - which is built on libev and provides constructs like queues, etc to make your multi-processing life much better.
- andy_ppp 9y agoI still find it terrifying monkey patching all the internal functions with Gevent; to be honest it caused a loads of weird bugs with celery that were impossible to debug. I’m pretty sure given my experience I’d choose to avoid it in future.
- sandGorgon 9y agofor standalone programs, there is simply no better multi-processing library than gevent. Forget the libev performance thing, the api is super pleasant... even more so than asyncio. When you use gevent to run explicit green threads, you do NOT monkey patch - you call the library functions. You only monkey patch if you want stuff happening for free. Which also works pretty well in production with gunicorn.. and yes with celery. gevent is officially supported by celery - https://github.com/celery/celery/tree/master/examples/gevent https://github.com/celery/celery/tree/master/examples/gevent http://docs.celeryproject.org/en/latest/_modules/celery/concurrency/gevent.html http://docs.celeryproject.org/en/latest/_modules/celery/conc...
- andy_ppp 9y agoSure, the example doesn’t monkey patch there though. My issues happened around complex multi part Chords and removing Gevent fixed stuff, but then eventually I moved away from celery too and just used Pikka and Rabbitmq directly. This gave me more control and allowed me to see what was happening more easily; celery sometimes seemed to hide exceptions from me. Glad you haven’t had issues.
- srean 9y agoCurious if you have tried or evaluated stackless in production. One doesnt hear a lot on stackless python these dsys. BTW this is for my education only, not advocating one over the other.
- sandGorgon 9y agoNot really. I'm really going with the community here and the recent community is huge. Gevent works in production with celery and even newrelic works with pretty OK (not perfect) When we are talking about production systems, I need the whole ecosystem to work. So I'd rather use vanilla Python with few libraries like gevent.
- mattbillenstein 9y agoI've used gevent monkey patching for years in real codebases with success although I have not used celery. Most problems I've seen with this come from not doing the monkey patching before importing anything else. Also, generally avoiding c-extensions for talking to databases unless they have explicit support for greenlets is advisable. My main webapp currently is Flask on gunicorn with async gevent workers. Mix in Flask-Sockets (using gevent-websocket) and I have websocket support as well and use redis pubsub as a broker to communicate intraprocess. It's a nice and capable system and you can reach into gevent if you want to do scatter-gather sorts of patterns without dealing with threads.
- sandGorgon 9y agoDo you have a small sample of gunicorn + flask + websockets + sqlalchemu config ? I've been exploring a little here and looking for something that works well.
- mattbillenstein 9y agoSorry, I don't have something that is easily sharable.