4 ms·

Really, I wouldn’t teach junior developers that it’s ok to use a database table when a queue is needed. Sure, you can get away with this and there are cases whe

Really, I wouldn’t teach junior developers that it’s ok to use a database table when a queue is needed. Sure, you can get away with this and there are cases when it’s all you need. But I’ve been one of those juniors who forgot to limit the query, who didn’t have enough indices, who tried to order all records by date and had full table scans everywhere, who implemented the worker with a cron job and didn’t synchronize this with a lock. It might work, but it’s not the general case and you might spend more time to debug your table then to write the code to use a real queue. And I’ve also seen people build their own queueing engine for a few hundred tasks per day. Why don’t they just choose one of the very good open source solutions?

blowski 7y ago
Like many design patterns when implemented badly, queuing will usually result in a lot of problems. But if your team is competent, you’re using an off-the-shelf library, and you don’t have crazy demands, then re-using infrastructure can be a good idea as it’s fewer things to manage.
- tonetheman 7y ago
  Fewer things to manage is the key!
rumanator 7y ago
What's the rationale to teach databases as message queues when it requires special querying and updates when there are so many message queues services already available, easy to use, and standard compliant?
- to11mtm 7y ago
  > What's the rationale to teach databases as message queues when it requires special querying and updates when there are so many message queues services already available, easy to use, and standard compliant? If you already need a database for something else, using the DB as a Queue means you don't need to list {mqFlavorOfChoice} as a requirement for new hires. You also don't have to manage that extra infrastructure. Of course, you are putting additional load on the DB. Mind you, I'm speaking of a pub-sub type queue and not a FIFO here. You can do FIFO queues in DB as well of course, it's just not as compelling of a story nowadays. Also way easier to look at and 'poke' a Database queue if you need to. The queries are also not really difficult to write for a general purpose use case.
jf22 7y ago
All the issues you mention are fairly easy to fix.
- rmetzler 7y ago
  After you found them. And you find them after you become aware of problems. And how hard it is to clean up the data at this point in time depends entirely on the kind of system you're working with.
SahAssar 7y ago
I agree that using a traditional DB as a queue has it's pitfalls, but it also has great benefits with regards to consistency, reliability and simplicity. If I was building a new system that required a queue I'd definitely put in the same postgres db as the rest of the data until I had a good reason not to.
andrewstuart 7y ago
"Because someone might add bugs to the code, or do it wrong" isn't a reason not to take a particular approach.
- mzz80 7y ago
  Of course it is. When one approach is far more likely to introduce bugs, complicated interactions, and be more difficult to maintain, it is absolutely a reason to not take a particular approach. It’s one of factors you need to consider in everyday software engineering. If you don’t take this into consideration then you’re detracting from the business to satisfy another need.
- james_s_tayler 7y ago
  Sure it is. If one approach is easier to mess up than the other then take the approach with less footguns.

blowski 7y ago

Like many design patterns when implemented badly, queuing will usually result in a lot of problems. But if your team is competent, you’re using an off-the-shelf library, and you don’t have crazy demands, then re-using infrastructure can be a good idea as it’s fewer things to manage.

tonetheman 7y ago
Fewer things to manage is the key!

rumanator 7y ago

What's the rationale to teach databases as message queues when it requires special querying and updates when there are so many message queues services already available, easy to use, and standard compliant?

to11mtm 7y ago
> What's the rationale to teach databases as message queues when it requires special querying and updates when there are so many message queues services already available, easy to use, and standard compliant? If you already need a database for something else, using the DB as a Queue means you don't need to list {mqFlavorOfChoice} as a requirement for new hires. You also don't have to manage that extra infrastructure. Of course, you are putting additional load on the DB. Mind you, I'm speaking of a pub-sub type queue and not a FIFO here. You can do FIFO queues in DB as well of course, it's just not as compelling of a story nowadays. Also way easier to look at and 'poke' a Database queue if you need to. The queries are also not really difficult to write for a general purpose use case.

jf22 7y ago

All the issues you mention are fairly easy to fix.

rmetzler 7y ago
After you found them. And you find them after you become aware of problems. And how hard it is to clean up the data at this point in time depends entirely on the kind of system you're working with.

SahAssar 7y ago

I agree that using a traditional DB as a queue has it's pitfalls, but it also has great benefits with regards to consistency, reliability and simplicity. If I was building a new system that required a queue I'd definitely put in the same postgres db as the rest of the data until I had a good reason not to.

andrewstuart 7y ago

"Because someone might add bugs to the code, or do it wrong" isn't a reason not to take a particular approach.

mzz80 7y ago
Of course it is. When one approach is far more likely to introduce bugs, complicated interactions, and be more difficult to maintain, it is absolutely a reason to not take a particular approach. It’s one of factors you need to consider in everyday software engineering. If you don’t take this into consideration then you’re detracting from the business to satisfy another need.
james_s_tayler 7y ago
Sure it is. If one approach is easier to mess up than the other then take the approach with less footguns.