3 ms·
Author of Celery here (not the article). Celery solves a different problem: it's a distributed system that will help you run your tasks on multiple machines, n
by asksol 10y ago
Author of Celery here (not the article).
Celery solves a different problem: it's a distributed system that will help you run your tasks on multiple machines, not an alternative to asyncio/twisted/tornado etc.
The GIL is often easily worked around by starting N instances of your app, but that doesn't work for all applications (games, video, audio). Celery won't help you in those cases, but you can write C extensions. 99.99% of the time you shouldn't even consider using posix threads in Python, as most libraries (including popular ones) are not thread-safe, resulting in spending precious time fixing tricky bugs.
Celery is also not in contrast to coroutines, actually it's quite common to use Celery as a distributed layer on top of async I/O frameworks (top tip: routing CPU-bound tasks to a prefork worker means they will not block your event loop).
Another thing, the article says `scrape_url.subtask(args=(url,)),` is not very readable, but the idiomatic way to write this is: `scrape_url.s(url)` (yup we have a one letter method name, Django has Q we have .s). See more examples here: http://docs.celeryproject.org/en/latest/userguide/canvas.html http://docs.celeryproject.org/en/latest/userguide/canvas.htm...
- brianwawok 10y agoTIL about the s method.. Thanks for the tip!
- enjo 10y agoNothing useful to add other than thanks for the great library. I've been using Celery for years, it's made my life demonstrably better.
- mdomans 10y agoIn all honesty .subtask() is more readable for me than .s() Also, I don't want Celery to replace asyncio and/or threading. I was trying to point out that the programming model can be back ported to Python as a way of approaching concurrency, which, without having to support "locks everywhere" is simpler to implement with GIL
- asksol 10y agoI think it's more readable in isolation, but when you have workflows it distracts you from what it's actually doing, making it harder to see what the purpose is at a glance. An API can probably not have many shortcuts like this, but you learn what .s does once, and then you barely notice it. But that's my opinion and I have come to learn not everybody value succinctness in code, so you have the choice of both :o) Had I not been constrained by backward compatibility I may have made it so that task(arg) only defines the signature of a task invocation and you'd need to do task(arg).delay() to call it remotely, and task(args)() to call it as a function locally.
- mdomans 10y agoNot merely a choice of mine to make - working in a team actually makes .subtask better