6 ms·

Show HN: Dramatiq – An alternative to Celery

53 points by Bogdanp 9y ago

rcarmo 9y ago
I might have missed it, but is this asyncio-ready?
- Bogdanp 9y ago
  It currently isn't. I'm holding out until there is some demand for it as I currently don't need it for my own stuff.
SEJeff 9y ago
Nice touch on having prometheus metric support builtin, but in general, if I want a simpler celery, I always use rq[1]. How does this compare to rq? Having this one the website somewhere obvious and why you wrote it would be a really nice touch. http://python-rq.org/ http://python-rq.org/
- Bogdanp 9y ago
  > Nice touch on having prometheus metric support builtin Thanks! > How does this compare to rq? Having this one the website somewhere obvious and why you wrote it would be a really nice touch. This exists although I suppose I should highlight it better: https://dramatiq.io/motivation.html https://dramatiq.io/motivation.html I would say the biggest differences to RQ are: * Dramatiq supports RabbitMQ in addition to Redis, * Dramatiq uses a mixed multi-process/multi-thread model whereas RQ forks for every task. In my benchmarks, Dramatiq has much, much better throughput than RQ, * Dramatiq has support for task-level priorities, rate limiting and delayed tasks, * Dramatiq comes with an in-memory broker you can use for testing.
adambrenecki 9y ago
> Dramatiq is licensed under the AGPL and it officially supports Python 3.6 and later. Commercial Licensing is also available. Oh. > I’ve done a ton of open source work over the course of my career. Companies have used that work to generate income for themselves. Inevitably, I’ve ended up supporting that software on those companies’ behalf for free and that is not sustainable long term. This is totally 100% true and understandable. Unfortunately, a lot of devs are going to go "hmm, I could battle to convince someone that can approve purchases that it's worth forking out $2k/year for this, or I could just use Celery".
- Bogdanp 9y ago
  Yep, that's understandable. If someone makes a cost-benefit analysis and decides that they prefer the cheaper option then that's perfectly fine with me, I'll still be here a year later when they realize their mistake. ;) That said, I do give out commercial licenses for free for companies that are just starting out. My goal with this is not to get rich, it's just to get people to contribute something back in whatever way they can.
  - pmontra 9y ago
    I approve the share and share back spirit. AGPL for a library means that the application that uses it must be licensed under the AGPL too, right?
    - Bogdanp 9y ago
      It's a little more nuanced[0] than that and I confess I don't fully understand it myself. IANAL, but my understanding is it depends on how it's used. If a user action on your website ends up triggering Dramatiq code then you have to open source that code in addition to providing access to Dramatiq. If all you do is batch processing on your own servers, then you don't have to do anything; you're merely a user of the software. [0]: https://opensource.stackexchange.com/a/5004 https://opensource.stackexchange.com/a/5004
      - btown 9y ago
        Is asynchronous interaction still "interaction" for the purposes of the AGPL? If End User A's manual changes are placed into a database or file which is read by Dramatiq code in a subsequent cron-triggered batch process, is that considered interaction? These questions are not at all cut and dry, but companies like MongoDB who use the AGPL could conceivably argue that this would be considered an interaction. [I am not a lawyer.] As a separate point, it's odd that anyone would use a license that they "don't fully understand," when (presumably) they would hold themselves to a higher standard of understanding the behavior of libraries they uses in their software itself. Is it irrational to consider this, at the very least, an indication of lack of attention to detail, and a tremendous red flag?
vosper 9y ago
Does Dramatiq have first-class support for SQS? That's my biggest gripe with Celery. I mean, it does work with SQS, but it doesn't support it as a result backend, and it doesn't do (eg) batch PUTs automatically, so it's pretty slow. edit: I search the docs for SQS and got no results, so I'm guessing it's not supported.
- Bogdanp 9y ago
  It doesn't, but you're the second person to request it. Feel free to open an issue and I'll take a stab at writing an SQS broker. Having looked at it before, it didn't seem like it would take much work at all.
bjt 9y ago
RabbitMQ is not hard to use directly. I think most developers would be better off doing that, and coming to understand the power of AMQP, rather than adopting cute DSLs like this and Celery that put you into a box and narrow your view of what's possible. I started that way with Celery and Django, maybe 6 years ago. At one point we found a bug with Celery not resolving 'chord' callbacks when all the parallel tasks had completed. It was difficult to debug, going through Celery's layers of code that try to make various backends present the same interface. We weaned ourselves off Celery and started using just the RabbitMQ and Redis libraries directly. It was definitely a shift worth making, allowing us to make performance/reliability tradeoffs that were better suited to our systems, and opened our eyes to possibilities with RabbitMQ and Redis streams that we hadn't been able to see when looking through a Celery lens. If there's a place for DSLs like this, it may be when you have very novice programmers needing to write quick throwaway jobs without wanting to spend a lot of time learning the underlying systems. Maybe analysts on a data team, for example.
- Bogdanp 9y ago
  > I started that way with Celery and Django, maybe 6 years ago. At one point we found a bug with Celery not resolving 'chord' callbacks when all the parallel tasks had completed. It was difficult to debug, going through Celery's layers of code that try to make various backends present the same interface. We weaned ourselves off Celery and started using just the RabbitMQ and Redis libraries directly. It was definitely a shift worth making, allowing us to make performance/reliability tradeoffs that were better suited to our systems, and opened our eyes to possibilities with RabbitMQ and Redis streams that we hadn't been able to see when looking through a Celery lens. Celery's task workflow "features" have bitten me in the past as well and I agree that it is a complicated piece of software (having had to go through its source code many, many times). That doesn't mean all software in this space has to be like that. That is why one of my goals with Dramatiq is for it to have a very simple and easy to understand core. > If there's a place for DSLs like this, it may be when you have very novice programmers needing to write quick throwaway jobs without wanting to spend a lot of time learning the underlying systems. Maybe analysts on a data team, for example. I think you underestimate the amount of value that tools like these bring to the table when it comes to actually shipping a product and getting things done. Of course, you should make an informed decision on the tradeoffs you're making, but calling anyone a "novice" for using an off-the-shelf solution instead of spending valuable time building his/her own is a bit much. :)
coleifer 9y ago
Huey: https://huey.readthedocs.io https://huey.readthedocs.io -- even simpler :) redis or sqlite, mit license, greenlet/thread/process worker model support.
- whalesalad 9y ago
  I love your work (Peewee is a tool I use very often). I would have liked to use Huey in a recent project but it doesn’t have modular backend support for Rabbit.
  - coleifer 9y ago
    Thanks so much. You're right, I think some people have written storage engines for rabbit (as well as mongo) but you might have to search for them. The storage api is simple enough you could write your own in an hour or two.
    - whalesalad 9y ago
      I might need to do this! And thanks again for your work on Peewee... I think people underestimate it’s power due to the funny name.
whalesalad 9y ago
I’ve submitted code to this project and just want to state here for the record that Bogdan is very responsive and accepting of contributions. It was a very pleasing experience compared to other open source projects! I reached out to the Python community recently with the question: “Django is to Flask as Celery is to ______?” Dramatiq was one of the suggested responses and the mission/purpose behind it resonated with me immediately. I’m a big fan of convention over configuration. Most of the time, if I’m writing software in Python, I don’t want to worry about a system that is not Python. IE, I enjoy an ORM that abstracts PostgreSQL. I enjoy this because it abstracts Rabbit. Obviously you need to understand the underlying system and an ORM is no replacement for knowing the ins and outs of Postgres, just as this is no replacement for understanding Rabbit. But... I love that it does everything I want it to do with minimal or zero configuration.
deleted 9y ago
[deleted]
jtharpla 9y ago
I figured I'd add a mention of the task queue we use at close.io: https://github.com/closeio/tasktiger https://github.com/closeio/tasktiger We run several hundred tasktiger workers in production (deployed using Kubernetes) to process the various backend task queues for our service. Project is MIT-licensed if anyone is interested in another celery alternative.

53 points by Bogdanp 9y ago

rcarmo 9y ago

I might have missed it, but is this asyncio-ready?

Bogdanp 9y ago
It currently isn't. I'm holding out until there is some demand for it as I currently don't need it for my own stuff.

SEJeff 9y ago

Nice touch on having prometheus metric support builtin, but in general, if I want a simpler celery, I always use rq[1]. How does this compare to rq? Having this one the website somewhere obvious and why you wrote it would be a really nice touch. http://python-rq.org/ http://python-rq.org/

Bogdanp 9y ago
> Nice touch on having prometheus metric support builtin Thanks! > How does this compare to rq? Having this one the website somewhere obvious and why you wrote it would be a really nice touch. This exists although I suppose I should highlight it better: https://dramatiq.io/motivation.html https://dramatiq.io/motivation.html I would say the biggest differences to RQ are: * Dramatiq supports RabbitMQ in addition to Redis, * Dramatiq uses a mixed multi-process/multi-thread model whereas RQ forks for every task. In my benchmarks, Dramatiq has much, much better throughput than RQ, * Dramatiq has support for task-level priorities, rate limiting and delayed tasks, * Dramatiq comes with an in-memory broker you can use for testing.

adambrenecki 9y ago

> Dramatiq is licensed under the AGPL and it officially supports Python 3.6 and later. Commercial Licensing is also available. Oh. > I’ve done a ton of open source work over the course of my career. Companies have used that work to generate income for themselves. Inevitably, I’ve ended up supporting that software on those companies’ behalf for free and that is not sustainable long term. This is totally 100% true and understandable. Unfortunately, a lot of devs are going to go "hmm, I could battle to convince someone that can approve purchases that it's worth forking out $2k/year for this, or I could just use Celery".

Bogdanp 9y ago
Yep, that's understandable. If someone makes a cost-benefit analysis and decides that they prefer the cheaper option then that's perfectly fine with me, I'll still be here a year later when they realize their mistake. ;) That said, I do give out commercial licenses for free for companies that are just starting out. My goal with this is not to get rich, it's just to get people to contribute something back in whatever way they can.
- pmontra 9y ago
  I approve the share and share back spirit. AGPL for a library means that the application that uses it must be licensed under the AGPL too, right?
  - Bogdanp 9y ago
    It's a little more nuanced[0] than that and I confess I don't fully understand it myself. IANAL, but my understanding is it depends on how it's used. If a user action on your website ends up triggering Dramatiq code then you have to open source that code in addition to providing access to Dramatiq. If all you do is batch processing on your own servers, then you don't have to do anything; you're merely a user of the software. [0]: https://opensource.stackexchange.com/a/5004 https://opensource.stackexchange.com/a/5004
    - btown 9y ago
      Is asynchronous interaction still "interaction" for the purposes of the AGPL? If End User A's manual changes are placed into a database or file which is read by Dramatiq code in a subsequent cron-triggered batch process, is that considered interaction? These questions are not at all cut and dry, but companies like MongoDB who use the AGPL could conceivably argue that this would be considered an interaction. [I am not a lawyer.] As a separate point, it's odd that anyone would use a license that they "don't fully understand," when (presumably) they would hold themselves to a higher standard of understanding the behavior of libraries they uses in their software itself. Is it irrational to consider this, at the very least, an indication of lack of attention to detail, and a tremendous red flag?

vosper 9y ago

Does Dramatiq have first-class support for SQS? That's my biggest gripe with Celery. I mean, it does work with SQS, but it doesn't support it as a result backend, and it doesn't do (eg) batch PUTs automatically, so it's pretty slow. edit: I search the docs for SQS and got no results, so I'm guessing it's not supported.

Bogdanp 9y ago
It doesn't, but you're the second person to request it. Feel free to open an issue and I'll take a stab at writing an SQS broker. Having looked at it before, it didn't seem like it would take much work at all.

bjt 9y ago

RabbitMQ is not hard to use directly. I think most developers would be better off doing that, and coming to understand the power of AMQP, rather than adopting cute DSLs like this and Celery that put you into a box and narrow your view of what's possible. I started that way with Celery and Django, maybe 6 years ago. At one point we found a bug with Celery not resolving 'chord' callbacks when all the parallel tasks had completed. It was difficult to debug, going through Celery's layers of code that try to make various backends present the same interface. We weaned ourselves off Celery and started using just the RabbitMQ and Redis libraries directly. It was definitely a shift worth making, allowing us to make performance/reliability tradeoffs that were better suited to our systems, and opened our eyes to possibilities with RabbitMQ and Redis streams that we hadn't been able to see when looking through a Celery lens. If there's a place for DSLs like this, it may be when you have very novice programmers needing to write quick throwaway jobs without wanting to spend a lot of time learning the underlying systems. Maybe analysts on a data team, for example.

Bogdanp 9y ago
> I started that way with Celery and Django, maybe 6 years ago. At one point we found a bug with Celery not resolving 'chord' callbacks when all the parallel tasks had completed. It was difficult to debug, going through Celery's layers of code that try to make various backends present the same interface. We weaned ourselves off Celery and started using just the RabbitMQ and Redis libraries directly. It was definitely a shift worth making, allowing us to make performance/reliability tradeoffs that were better suited to our systems, and opened our eyes to possibilities with RabbitMQ and Redis streams that we hadn't been able to see when looking through a Celery lens. Celery's task workflow "features" have bitten me in the past as well and I agree that it is a complicated piece of software (having had to go through its source code many, many times). That doesn't mean all software in this space has to be like that. That is why one of my goals with Dramatiq is for it to have a very simple and easy to understand core. > If there's a place for DSLs like this, it may be when you have very novice programmers needing to write quick throwaway jobs without wanting to spend a lot of time learning the underlying systems. Maybe analysts on a data team, for example. I think you underestimate the amount of value that tools like these bring to the table when it comes to actually shipping a product and getting things done. Of course, you should make an informed decision on the tradeoffs you're making, but calling anyone a "novice" for using an off-the-shelf solution instead of spending valuable time building his/her own is a bit much. :)

coleifer 9y ago

Huey: https://huey.readthedocs.io https://huey.readthedocs.io -- even simpler :) redis or sqlite, mit license, greenlet/thread/process worker model support.

whalesalad 9y ago
I love your work (Peewee is a tool I use very often). I would have liked to use Huey in a recent project but it doesn’t have modular backend support for Rabbit.
- coleifer 9y ago
  Thanks so much. You're right, I think some people have written storage engines for rabbit (as well as mongo) but you might have to search for them. The storage api is simple enough you could write your own in an hour or two.
  - whalesalad 9y ago
    I might need to do this! And thanks again for your work on Peewee... I think people underestimate it’s power due to the funny name.

whalesalad 9y ago

I’ve submitted code to this project and just want to state here for the record that Bogdan is very responsive and accepting of contributions. It was a very pleasing experience compared to other open source projects! I reached out to the Python community recently with the question: “Django is to Flask as Celery is to ______?” Dramatiq was one of the suggested responses and the mission/purpose behind it resonated with me immediately. I’m a big fan of convention over configuration. Most of the time, if I’m writing software in Python, I don’t want to worry about a system that is not Python. IE, I enjoy an ORM that abstracts PostgreSQL. I enjoy this because it abstracts Rabbit. Obviously you need to understand the underlying system and an ORM is no replacement for knowing the ins and outs of Postgres, just as this is no replacement for understanding Rabbit. But... I love that it does everything I want it to do with minimal or zero configuration.

deleted 9y ago

[deleted]

jtharpla 9y ago

I figured I'd add a mention of the task queue we use at close.io: https://github.com/closeio/tasktiger https://github.com/closeio/tasktiger We run several hundred tasktiger workers in production (deployed using Kubernetes) to process the various backend task queues for our service. Project is MIT-licensed if anyone is interested in another celery alternative.