4 ms·

RabbitMQ is not hard to use directly. I think most developers would be better off doing that, and coming to understand the power of AMQP, rather than adopting

by bjt 9y ago

RabbitMQ is not hard to use directly. I think most developers would be better off doing that, and coming to understand the power of AMQP, rather than adopting cute DSLs like this and Celery that put you into a box and narrow your view of what's possible. I started that way with Celery and Django, maybe 6 years ago. At one point we found a bug with Celery not resolving 'chord' callbacks when all the parallel tasks had completed. It was difficult to debug, going through Celery's layers of code that try to make various backends present the same interface. We weaned ourselves off Celery and started using just the RabbitMQ and Redis libraries directly. It was definitely a shift worth making, allowing us to make performance/reliability tradeoffs that were better suited to our systems, and opened our eyes to possibilities with RabbitMQ and Redis streams that we hadn't been able to see when looking through a Celery lens. If there's a place for DSLs like this, it may be when you have very novice programmers needing to write quick throwaway jobs without wanting to spend a lot of time learning the underlying systems. Maybe analysts on a data team, for example.

Bogdanp 9y ago
> I started that way with Celery and Django, maybe 6 years ago. At one point we found a bug with Celery not resolving 'chord' callbacks when all the parallel tasks had completed. It was difficult to debug, going through Celery's layers of code that try to make various backends present the same interface. We weaned ourselves off Celery and started using just the RabbitMQ and Redis libraries directly. It was definitely a shift worth making, allowing us to make performance/reliability tradeoffs that were better suited to our systems, and opened our eyes to possibilities with RabbitMQ and Redis streams that we hadn't been able to see when looking through a Celery lens. Celery's task workflow "features" have bitten me in the past as well and I agree that it is a complicated piece of software (having had to go through its source code many, many times). That doesn't mean all software in this space has to be like that. That is why one of my goals with Dramatiq is for it to have a very simple and easy to understand core. > If there's a place for DSLs like this, it may be when you have very novice programmers needing to write quick throwaway jobs without wanting to spend a lot of time learning the underlying systems. Maybe analysts on a data team, for example. I think you underestimate the amount of value that tools like these bring to the table when it comes to actually shipping a product and getting things done. Of course, you should make an informed decision on the tradeoffs you're making, but calling anyone a "novice" for using an off-the-shelf solution instead of spending valuable time building his/her own is a bit much. :)
whalesalad 9y ago
Seriously? RabbitMQ is a huge waste of complexity for like 95% of modern webapp use cases. I use tools like Dramatiq specifically because I want to abstract away the complexity of exchanges, bindings, queues, dead lettering, etc... If you write out all the nouns associated with RMQ you start to realize there is a LOT going on. It’s a very powerful system but convention over configuration doesn’t work here.
- moogie 9y ago
  No offense, but this sounds a lot like "I can't be bothered to learn about the tech but I want to put it on my CV anyway". The "95%" of use cases probably don't need a message broker at all. Far too many "modern webapp" developers are shoving message brokers into websites so they can pretend to be more important and sophisticated than they are. If you're not even willing to learn how to connect to RabbitMQ, or any broker, without a big giant messy crutch, I'd suggest you shouldn't use them at all.
scaryclam 9y ago
I respect the authors of celery and think they've done a good job of making RabbitMQ a bit more accessible over the years. I don't think celery's just for novice users, there are a lot of devs out there who don't really need to invest in learning about their message brokers (short lived projects and non-essential parts of the larger system can easily use celery without needing to worry about it). That said, I really do agree with you on other points. We ditched celery after many years of use in favor of just connecting to RabbitMQ ourselves. Over the last few years, it's saved a massive amount of time and effort and has made debugging and tracing issues so much easier it's rediculous. I don't think many long term celery users really appreciate how much time is getting wasted. I understand how it may look like being able to ignore the underlying technology is a time saver, but that time easily gets blown away the second something doesn't behave as it should (and having gone through many of these problems, they're not exactly uncommon). For anyone curious how much effort it took to write our own consumer/publisher, both are less that 200 lines of code and can be arguably cut down further without losing much (the author liked whitespace etc). Both were written in less than 2 days, including learning time, and are reusable enough to use in other projects.

by bjt 9y ago

Bogdanp 9y ago

> I started that way with Celery and Django, maybe 6 years ago. At one point we found a bug with Celery not resolving 'chord' callbacks when all the parallel tasks had completed. It was difficult to debug, going through Celery's layers of code that try to make various backends present the same interface. We weaned ourselves off Celery and started using just the RabbitMQ and Redis libraries directly. It was definitely a shift worth making, allowing us to make performance/reliability tradeoffs that were better suited to our systems, and opened our eyes to possibilities with RabbitMQ and Redis streams that we hadn't been able to see when looking through a Celery lens. Celery's task workflow "features" have bitten me in the past as well and I agree that it is a complicated piece of software (having had to go through its source code many, many times). That doesn't mean all software in this space has to be like that. That is why one of my goals with Dramatiq is for it to have a very simple and easy to understand core. > If there's a place for DSLs like this, it may be when you have very novice programmers needing to write quick throwaway jobs without wanting to spend a lot of time learning the underlying systems. Maybe analysts on a data team, for example. I think you underestimate the amount of value that tools like these bring to the table when it comes to actually shipping a product and getting things done. Of course, you should make an informed decision on the tradeoffs you're making, but calling anyone a "novice" for using an off-the-shelf solution instead of spending valuable time building his/her own is a bit much. :)

whalesalad 9y ago

Seriously? RabbitMQ is a huge waste of complexity for like 95% of modern webapp use cases. I use tools like Dramatiq specifically because I want to abstract away the complexity of exchanges, bindings, queues, dead lettering, etc... If you write out all the nouns associated with RMQ you start to realize there is a LOT going on. It’s a very powerful system but convention over configuration doesn’t work here.

moogie 9y ago
No offense, but this sounds a lot like "I can't be bothered to learn about the tech but I want to put it on my CV anyway". The "95%" of use cases probably don't need a message broker at all. Far too many "modern webapp" developers are shoving message brokers into websites so they can pretend to be more important and sophisticated than they are. If you're not even willing to learn how to connect to RabbitMQ, or any broker, without a big giant messy crutch, I'd suggest you shouldn't use them at all.

scaryclam 9y ago

I respect the authors of celery and think they've done a good job of making RabbitMQ a bit more accessible over the years. I don't think celery's just for novice users, there are a lot of devs out there who don't really need to invest in learning about their message brokers (short lived projects and non-essential parts of the larger system can easily use celery without needing to worry about it). That said, I really do agree with you on other points. We ditched celery after many years of use in favor of just connecting to RabbitMQ ourselves. Over the last few years, it's saved a massive amount of time and effort and has made debugging and tracing issues so much easier it's rediculous. I don't think many long term celery users really appreciate how much time is getting wasted. I understand how it may look like being able to ignore the underlying technology is a time saver, but that time easily gets blown away the second something doesn't behave as it should (and having gone through many of these problems, they're not exactly uncommon). For anyone curious how much effort it took to write our own consumer/publisher, both are less that 200 lines of code and can be arguably cut down further without losing much (the author liked whitespace etc). Both were written in less than 2 days, including learning time, and are reusable enough to use in other projects.