6 ms·

> Using an RDBMS as a work queue is an anti-pattern I don't want to appear too ignorant but why? I hear many claims of this but few explanations. The best I've

by Veratyr 10y ago

> Using an RDBMS as a work queue is an anti-pattern I don't want to appear too ignorant but why? I hear many claims of this but few explanations. The best I've heard is something about not being able to handle the load but I don't really buy that unless you don't need persistence/integrity.

rosser 10y ago
Primarily, because it's hard to get right. Of course it can be done well, or correctly (and as The Fine Article points out, adding SKIP LOCKED to SELECT ... FOR UPDATE will go a long way towards that). But most people do them terribly. As just one example, from a previous job: they were using SQL Alchemy in the code that interacted with the work queue, but somehow didn't realize that the DBAPI being implicitly transactional meant they'd need to manage transaction-state themselves. Consequently, they never released the implicitly opened transactions that every poll for new work created. No-one thought it was strange that the work queue system needed to be shut down weekly, so that a VACUUM FULL could reclaim the gibibytes of space that were wasted because the db couldn't clean up tuples (row version) that still appeared live to stale, abandoned transactions. Until I was all, "Uh, guys?" The fix was obvious: disable implicit transactions. But that's just one example, if particularly egregious. I've seen and heard countless more.
- ioquatix 10y ago
  That's a nice summary. But, does that really make it an anti-pattern?
  - rosser 10y ago
    I'd argue that a pattern as prone to subtly (or grossly) incorrect implementation as databases-as-work-queues is qualifies, yes.
    - einhverfr 10y ago
      But all work queue architectures I have worked with have subtle problems which are easy to get wrong. Might we conclude that the use of a work queue is an antipattern?
- Veratyr 10y ago
  That sounds like a problem with using SQL at all more than a problem with using it for a work queue. And really it's not even a problem with SQL, it's a problem with Python's weirdass DBAPI. Wouldn't using a basic wrapper like Celery take care of most of that for you though?
  - Sami_Lehtinen 10y ago
    That's just the point. People seem to think that because we don't even know how to use the current technology correctly, adding dozens of new alternate technologies would somehow make the situation better. - Then they're surprised that adding all those new cool technologies didn't actually fix the issue(s), probably made those just worse. Because nobody knows how to use the new tools correctly either.
  - mkesper 10y ago
    It's Python DBAPI used wrongly. Use the connection in a connection manager (e.g. "with sqlite3.connect(":memory:") as connection:").
- einhverfr 10y ago
  It is definitely hard to get right, for sure. I am actually planning on taking what I have learned on this and releasing it soon. And a lot of cookie-cutter solutions out there don't scale. But a lot of things are less about outlines than details and that is true here. If you do get it right, you get a work queue where jobs disappear properly when completed and if they fail remain in the queue to be retried. The fact is, work queues are hard to get right on any technology. You have a ridiculous number of corner cases and failure conditions you need to account for and the question ends up coming down to what you want the default failure state to be.
  - grzm 10y ago
    PgQ, part of Skype's SkyTools package, is a good example of queues in Postgres. It's used by their Londiste, a replication tool that was developed before built-in replication was mature. https://github.com/markokr/skytools https://github.com/markokr/skytools
    - einhverfr 10y ago
      But you have a fairly major problem with all these solutions and that is when a message leaves a queue. For a message queue you want it to leave the queue when delivered. For a job queue you want it to leave the queue when the work is completed but not to be sent out in the mean time. Now imagine you have processing jobs that take two weeks with 16 cores to complete. That's one thing that makes this a hard problem. If you have open transactions for two weeks "Deleting this...." then autovacuum is going to be totally ineffective. So failure detection and recovery is an important (and probably the hardest) part of the problem. PgQ looks to me like it is a solution to a message queue problem not a work queue problem.
      - grzm 10y ago
        IIRC (it's been a while), PgQ doesn't keep transactions open while events are processed: events are fetched and marked as in process; they're then marked succeeded or failed to finish the event. The bulk copy of initial data which bootstraps replication is a long duration process: I don't believe transactions are held open this entire time. If I understand you correctly, this handles your work queue case.

by Veratyr 10y ago

rosser 10y ago

Primarily, because it's hard to get right. Of course it can be done well, or correctly (and as The Fine Article points out, adding SKIP LOCKED to SELECT ... FOR UPDATE will go a long way towards that). But most people do them terribly. As just one example, from a previous job: they were using SQL Alchemy in the code that interacted with the work queue, but somehow didn't realize that the DBAPI being implicitly transactional meant they'd need to manage transaction-state themselves. Consequently, they never released the implicitly opened transactions that every poll for new work created. No-one thought it was strange that the work queue system needed to be shut down weekly, so that a VACUUM FULL could reclaim the gibibytes of space that were wasted because the db couldn't clean up tuples (row version) that still appeared live to stale, abandoned transactions. Until I was all, "Uh, guys?" The fix was obvious: disable implicit transactions. But that's just one example, if particularly egregious. I've seen and heard countless more.

ioquatix 10y ago
That's a nice summary. But, does that really make it an anti-pattern?
- rosser 10y ago
  I'd argue that a pattern as prone to subtly (or grossly) incorrect implementation as databases-as-work-queues is qualifies, yes.
  - einhverfr 10y ago
    But all work queue architectures I have worked with have subtle problems which are easy to get wrong. Might we conclude that the use of a work queue is an antipattern?
Veratyr 10y ago
That sounds like a problem with using SQL at all more than a problem with using it for a work queue. And really it's not even a problem with SQL, it's a problem with Python's weirdass DBAPI. Wouldn't using a basic wrapper like Celery take care of most of that for you though?
- Sami_Lehtinen 10y ago
  That's just the point. People seem to think that because we don't even know how to use the current technology correctly, adding dozens of new alternate technologies would somehow make the situation better. - Then they're surprised that adding all those new cool technologies didn't actually fix the issue(s), probably made those just worse. Because nobody knows how to use the new tools correctly either.
- mkesper 10y ago
  It's Python DBAPI used wrongly. Use the connection in a connection manager (e.g. "with sqlite3.connect(":memory:") as connection:").
einhverfr 10y ago
It is definitely hard to get right, for sure. I am actually planning on taking what I have learned on this and releasing it soon. And a lot of cookie-cutter solutions out there don't scale. But a lot of things are less about outlines than details and that is true here. If you do get it right, you get a work queue where jobs disappear properly when completed and if they fail remain in the queue to be retried. The fact is, work queues are hard to get right on any technology. You have a ridiculous number of corner cases and failure conditions you need to account for and the question ends up coming down to what you want the default failure state to be.
- grzm 10y ago
  PgQ, part of Skype's SkyTools package, is a good example of queues in Postgres. It's used by their Londiste, a replication tool that was developed before built-in replication was mature. https://github.com/markokr/skytools https://github.com/markokr/skytools
  - einhverfr 10y ago
    But you have a fairly major problem with all these solutions and that is when a message leaves a queue. For a message queue you want it to leave the queue when delivered. For a job queue you want it to leave the queue when the work is completed but not to be sent out in the mean time. Now imagine you have processing jobs that take two weeks with 16 cores to complete. That's one thing that makes this a hard problem. If you have open transactions for two weeks "Deleting this...." then autovacuum is going to be totally ineffective. So failure detection and recovery is an important (and probably the hardest) part of the problem. PgQ looks to me like it is a solution to a message queue problem not a work queue problem.
    - grzm 10y ago
      IIRC (it's been a while), PgQ doesn't keep transactions open while events are processed: events are fetched and marked as in process; they're then marked succeeded or failed to finish the event. The bulk copy of initial data which bootstraps replication is a long duration process: I don't believe transactions are held open this entire time. If I understand you correctly, this handles your work queue case.