5 ms·

Don't use celery use: https://github.com/binarydud/pyres https://github.com/binarydud/pyres

by coolsunglasses 14y ago

mcintyre1994 14y ago
I've been expecting to use Celery in a project I'm working on, could you explain why?
- coolsunglasses 14y ago
  Celery is a PITA and over-complicated for the problem it solves.
  - rozap 14y ago
    Do you know why pyres relies on itty? Seems like kind of an odd dependency...
    - lost-theory 14y ago
      The pyres web interface (where you can view job status) is built using itty. edit: looks like they're using flask now, I don't see itty being used anywhere. They also split the web interface (resweb) into a separate project, so pyres doesn't depend on a web framework now.
      - rozap 14y ago
        Ah ok, thanks. I guess the documentation is a little outdated then. Thanks for the suggestion, I'll check pyres out when I get to that point in my project.
- hobb0001 14y ago
  Celery is brittle. If you stop a celery service, odds are, it will either not stop at all or leave stray child processes running. It is not transactional, at least not with a MongoDB back-end. If you stop a celery service, any tasks that were in progress may or may not complete, but none of them will be picked up again when you resume. No prioritization of queues or tasks. It is possible to set up separate services that handle different queues, but that's a PITA and there's still no guaranteed order of processing.
the_cat_kittles 14y ago
considering the infrastructure that has built up around celery and particularly its use with django, i dont know why you wouldn't- care to explain?
- dbarlett 14y ago
  Pinterest replaced Celery and RabbitMQ [1] with their fork of pyres [2], presumably because it's 1/100th the codebase but does what they need to. [1] http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html http://highscalability.com/blog/2013/4/15/scaling-pinterest-... [2] https://github.com/pinterest/pyres https://github.com/pinterest/pyres

Don't use celery use: https://github.com/binarydud/pyres https://github.com/binarydud/pyres

mcintyre1994 14y ago

I've been expecting to use Celery in a project I'm working on, could you explain why?

coolsunglasses 14y ago
Celery is a PITA and over-complicated for the problem it solves.
- rozap 14y ago
  Do you know why pyres relies on itty? Seems like kind of an odd dependency...
  - lost-theory 14y ago
    The pyres web interface (where you can view job status) is built using itty. edit: looks like they're using flask now, I don't see itty being used anywhere. They also split the web interface (resweb) into a separate project, so pyres doesn't depend on a web framework now.
    - rozap 14y ago
      Ah ok, thanks. I guess the documentation is a little outdated then. Thanks for the suggestion, I'll check pyres out when I get to that point in my project.
hobb0001 14y ago
Celery is brittle. If you stop a celery service, odds are, it will either not stop at all or leave stray child processes running. It is not transactional, at least not with a MongoDB back-end. If you stop a celery service, any tasks that were in progress may or may not complete, but none of them will be picked up again when you resume. No prioritization of queues or tasks. It is possible to set up separate services that handle different queues, but that's a PITA and there's still no guaranteed order of processing.

the_cat_kittles 14y ago

considering the infrastructure that has built up around celery and particularly its use with django, i dont know why you wouldn't- care to explain?

dbarlett 14y ago
Pinterest replaced Celery and RabbitMQ [1] with their fork of pyres [2], presumably because it's 1/100th the codebase but does what they need to. [1] http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html http://highscalability.com/blog/2013/4/15/scaling-pinterest-... [2] https://github.com/pinterest/pyres https://github.com/pinterest/pyres