4 ms·

> Guaranteed "exactly once" delivery of messages to a consumer within a visibility timeout That's not going to be true. It might be true when things are runnin

> Guaranteed "exactly once" delivery of messages to a consumer within a visibility timeout That's not going to be true. It might be true when things are running well, but when it fails, it'll either be at most once or at least once. You don't build for the steady state, you build against the failure mode. That's an important deciding factor in whether you choose a system: you can accept duplicates gracefully or you can accept some amount of data loss. Without reviewing all of the code, it's not possible to say what this actually is, but since it seems like it's up to the implementor to set up replication, I suspect this is an at-most-once queue (if the client receives a response before the server has replicated the data and the server is destroyed, the data is lost). But depending on the diligence of the developer, it could be that this provides no real guarantees (0-N deliveries).

MuffinFlavored 2y ago
> That's not going to be true. It might be true when things are running well, but when it fails, it'll either be at most once or at least once. Silly question as somebody not very deep in the details on this. It's not easy to make distributed systems idempotent across the board (POST vs PUT, etc.) Distributed rollbacks are also hard once you reach interacting with 3rd party APIs, databases, cache, etc. What is the trick in your "on message received handler" from the queue to achieve "exactly once"? Some kind of "message hash ID" and then you check in Redis if it has already been processed either fully successfully, partially, or with failures? That has drawbacks/problems too, no? Is it an impossible problem?
- samtheprogram 2y ago
  You don’t achieve exactly once at the protocol/queue level, but at the service/consumer level. This is why SQS guarantees at-least-once. It’s generally what you described, but the term I’ve seen is “nonce” which is essentially an ID on the message that’s unique, and you can check against an in-memory data store or similar to see if you’ve already processed the message, and simply return / stop processing the message/job if so.
  - mcqueenjordan 2y ago
    And just to add a small clarification since I had to double take: this isn't exactly-once delivery (which isn't possible), this is exactly-once processing. But even exactly-once processing generally has issues, so it's better to assume at least once processing as the thing to design for and try to make everything within your processing ~idempotent.
- bastawhiz 2y ago
  > What is the trick in your "on message received handler" from the queue to achieve "exactly once"? There's no trick. The problem isn't "how to build a robust enough system" it's "how can you possibly know whether the message was successfully processed or not". If you have one node that stores data for the queue, loss of that node means the data is gone. You don't know you don't have it. If you have multiple nodes running the queue and they stop talking to each other, how do you know whether one of the other nodes already allowed someone to process a particular message (e.g., in the face of a network partition), or didn't let someone else process the message (e.g., that node experienced hardware failure). At some point, you find yourself staring down the Byzantine generals problem. You can make systems pretty robust (and some folks have made extremely robust systems), but there's not really a completely watertight solution. And in most cases, the cost of watertightness simply isn't worth it compared to just making your system idempotent or accepting some reasonably small amount of data loss.
Justsignedup 2y ago
To be honest. I like the at least once constraint. It causes you to think of background jobs in a idempotent way and makes for better designs. So ultimately I never found the removal of it a mitzva. Also if you don't have the constraint and say it works and you need to change systems for any reason, now you gotta rewrite your workers.
- chuckhend 2y ago
  I agree. But it can be useful to have a guarantee, even for a specified period of time, that the message will only be seen once. For example, if the processing of that message is very expensive, such as if that message results in API requests to a very expensive SaaS service. It may be idempotent to process that message more times than necessary, but doing so may be cost prohibitive if you are billed per request. I think this is a case where using the VT to help you only process that message one time could help out quite a bit.
  - pipe_connector 2y ago
    Agreed that this property is useful for efficiency. The danger comes from users believing this property is a guarantee and making correctness decisions based on it. There is no such thing as a distributed lock or exclusive hold.
    - Justsignedup 2y ago
      This is more or less where my brain was heading. The utility of exactly once system is quite high. But the utility of coding knowing that it is at least once, hopefully, and if not we'll do a catch-up, means that I will code knowing that my work must be safe in an environment with these constraints. The best argument I have for a postgres background worker queue is simplicity. You're already managing a db, why not have the db simplify your startup's infrastructure. Honestly, with heroku, sidekiq, and redis it isn't even simplifying. But for startups simplicity is useful so they can scale when they need to and avoid complexity when possible. But that's the only argument I'd make.
cjen 2y ago
It will be true because of the "within a visibility timeout" right? Of course that makes the claim way less interesting. I took a peek at the code and it looks like their visibility timeout is pretty much a lock on a message. So it's not exactly once for any meaningful definition, but it does prevent the same message from being consumed multiple times within the visibility timeout.
- bastawhiz 2y ago
  > it does prevent the same message from being consumed multiple times within the visibility timeout. ... When there is no failure of the underlying system. The expectation of any queue is that messages are only delivered once. But that's not what's interesting: what matters is what happens when there's a system failure: either the message gets delivered more than once, the message gets delivered zero times, or a little of column A and a little of column B (which is the worst of both worlds and is a bug). If you have one queue node, it can fail and lose your data. If you have multiple queue nodes, you can have a network partition. In all cases, it's possible to not know whether the message was processed or not at some point
  - thethimble 2y ago
    The Two Generals Problem is a great thought experiment that describes how distributed consensus is impossible when communication between nodes has the possibility of failing. https://en.wikipedia.org/wiki/Two_Generals%27_Problem https://en.wikipedia.org/wiki/Two_Generals%27_Problem
    - immibis 2y ago
      Distributed consensus within a quorum is possible. With only two nodes there is no quorum. With three nodes and at most one offline, consensus is possible.
bilekas 2y ago
I'm curious about the same thing.. I use a deadletter queue in sqs and even sns and here I can see an archiver but it's not clear if I need to rollout my own deadletter behaviour here.. not sure I would be too confident in it either.. A nice novel project here but I'm a bit skeptical of its application for me at least.
- chuckhend 2y ago
  pgmq.archive() gives us an API to retain messages on your queue, its an alternative to pgmq.delete(). For me as a long-time Redis user, message retention was always important and was always extra work to implement. DLQ isn't a built-in feature to PGMQ yet. We run PGMQ our SaaS at Tembo.io, and the way we implement the DLQ is by checking the message's read_ct value. When it exceeds some value, we send the message to another queue rather than processing it. Successfully processed messages end up getting pgmq.archive()'d. Soon, we will be integrating https://github.com/tembo-io/pg_tier https://github.com/tembo-io/pg_tier into pgmq so that the archive table is put directly into cloud storage/S3.
chuckhend 2y ago
If the message never reaches the queue (network error, database is down, app is down, etc), then yes that is a 0 delivery scenario. Once the message reaches the queue though, it is guaranteed that only a single consumer can read the message for the duration of the visibility timeout. FOR UPDATE guarantees only a single consumer can read the record, and the visibility timeout means we don't have to hold a lock. After that visibility timeout expires, it is an at-least-once scenario. Any suggestion for how we could change the verbiage on the readme to make that more clear?
- Fire-Dragon-DoL 2y ago
  You are talking about distributed systems, nobody expects to read "exactly once" delivery. If I read that on the docs, I consider that a huge red flag. And the fact is that what you describe is a performance optimization, I still have to write my code so that it is idempotent, so that optimization does not affect me in any other way, because exactly once is not a thing. All of this to say, I'm not even sure it's worth mentioning?
  - peter_l_downs 2y ago
    You should let the Apache Flink team know, they mention exactly-once processing on their home page (under "correctness guarantees") and in their list of features. [0] https://flink.apache.org/ https://flink.apache.org/ [1] https://flink.apache.org/what-is-flink/flink-applications/#building-blocks-for-streaming-applications https://flink.apache.org/what-is-flink/flink-applications/#b...
    - sethammons 2y ago
      A common mistake: Exactly once delivery is not the same as exactly once processing. Exactly once delivery is an impossibility; see the two generals problem. Exactly once processing is usually at-least-once delivery coupled with deduplication or an otherwise idempotent action
    - Fire-Dragon-DoL 2y ago
      They are not the first or the last. You CAN have "at most once" processing, since there is always the option of processing zero messages due to systems being down, but it's a joint effort, nobody is able to offer it "for free" (as in, no effort to implement it)
cryptonector 2y ago
Quite. The problem with a visibility timeout is that there can be delays in handling a message, and if the original winner fails to do it in time then some other process/thread will, yes, but if processing has external side-effects that can't be undone then the at-most-once guarantee will definitely be violated. And if you run out of storage space and messages have to get dropped then you'll definitely violate the at-least-once guarantee.

MuffinFlavored 2y ago

> That's not going to be true. It might be true when things are running well, but when it fails, it'll either be at most once or at least once. Silly question as somebody not very deep in the details on this. It's not easy to make distributed systems idempotent across the board (POST vs PUT, etc.) Distributed rollbacks are also hard once you reach interacting with 3rd party APIs, databases, cache, etc. What is the trick in your "on message received handler" from the queue to achieve "exactly once"? Some kind of "message hash ID" and then you check in Redis if it has already been processed either fully successfully, partially, or with failures? That has drawbacks/problems too, no? Is it an impossible problem?

samtheprogram 2y ago
You don’t achieve exactly once at the protocol/queue level, but at the service/consumer level. This is why SQS guarantees at-least-once. It’s generally what you described, but the term I’ve seen is “nonce” which is essentially an ID on the message that’s unique, and you can check against an in-memory data store or similar to see if you’ve already processed the message, and simply return / stop processing the message/job if so.
- mcqueenjordan 2y ago
  And just to add a small clarification since I had to double take: this isn't exactly-once delivery (which isn't possible), this is exactly-once processing. But even exactly-once processing generally has issues, so it's better to assume at least once processing as the thing to design for and try to make everything within your processing ~idempotent.
bastawhiz 2y ago
> What is the trick in your "on message received handler" from the queue to achieve "exactly once"? There's no trick. The problem isn't "how to build a robust enough system" it's "how can you possibly know whether the message was successfully processed or not". If you have one node that stores data for the queue, loss of that node means the data is gone. You don't know you don't have it. If you have multiple nodes running the queue and they stop talking to each other, how do you know whether one of the other nodes already allowed someone to process a particular message (e.g., in the face of a network partition), or didn't let someone else process the message (e.g., that node experienced hardware failure). At some point, you find yourself staring down the Byzantine generals problem. You can make systems pretty robust (and some folks have made extremely robust systems), but there's not really a completely watertight solution. And in most cases, the cost of watertightness simply isn't worth it compared to just making your system idempotent or accepting some reasonably small amount of data loss.

Justsignedup 2y ago

To be honest. I like the at least once constraint. It causes you to think of background jobs in a idempotent way and makes for better designs. So ultimately I never found the removal of it a mitzva. Also if you don't have the constraint and say it works and you need to change systems for any reason, now you gotta rewrite your workers.

chuckhend 2y ago
I agree. But it can be useful to have a guarantee, even for a specified period of time, that the message will only be seen once. For example, if the processing of that message is very expensive, such as if that message results in API requests to a very expensive SaaS service. It may be idempotent to process that message more times than necessary, but doing so may be cost prohibitive if you are billed per request. I think this is a case where using the VT to help you only process that message one time could help out quite a bit.
- pipe_connector 2y ago
  Agreed that this property is useful for efficiency. The danger comes from users believing this property is a guarantee and making correctness decisions based on it. There is no such thing as a distributed lock or exclusive hold.
  - Justsignedup 2y ago
    This is more or less where my brain was heading. The utility of exactly once system is quite high. But the utility of coding knowing that it is at least once, hopefully, and if not we'll do a catch-up, means that I will code knowing that my work must be safe in an environment with these constraints. The best argument I have for a postgres background worker queue is simplicity. You're already managing a db, why not have the db simplify your startup's infrastructure. Honestly, with heroku, sidekiq, and redis it isn't even simplifying. But for startups simplicity is useful so they can scale when they need to and avoid complexity when possible. But that's the only argument I'd make.

cjen 2y ago

It will be true because of the "within a visibility timeout" right? Of course that makes the claim way less interesting. I took a peek at the code and it looks like their visibility timeout is pretty much a lock on a message. So it's not exactly once for any meaningful definition, but it does prevent the same message from being consumed multiple times within the visibility timeout.

bastawhiz 2y ago
> it does prevent the same message from being consumed multiple times within the visibility timeout. ... When there is no failure of the underlying system. The expectation of any queue is that messages are only delivered once. But that's not what's interesting: what matters is what happens when there's a system failure: either the message gets delivered more than once, the message gets delivered zero times, or a little of column A and a little of column B (which is the worst of both worlds and is a bug). If you have one queue node, it can fail and lose your data. If you have multiple queue nodes, you can have a network partition. In all cases, it's possible to not know whether the message was processed or not at some point
- thethimble 2y ago
  The Two Generals Problem is a great thought experiment that describes how distributed consensus is impossible when communication between nodes has the possibility of failing. https://en.wikipedia.org/wiki/Two_Generals%27_Problem https://en.wikipedia.org/wiki/Two_Generals%27_Problem
  - immibis 2y ago
    Distributed consensus within a quorum is possible. With only two nodes there is no quorum. With three nodes and at most one offline, consensus is possible.

bilekas 2y ago

I'm curious about the same thing.. I use a deadletter queue in sqs and even sns and here I can see an archiver but it's not clear if I need to rollout my own deadletter behaviour here.. not sure I would be too confident in it either.. A nice novel project here but I'm a bit skeptical of its application for me at least.

chuckhend 2y ago
pgmq.archive() gives us an API to retain messages on your queue, its an alternative to pgmq.delete(). For me as a long-time Redis user, message retention was always important and was always extra work to implement. DLQ isn't a built-in feature to PGMQ yet. We run PGMQ our SaaS at Tembo.io, and the way we implement the DLQ is by checking the message's read_ct value. When it exceeds some value, we send the message to another queue rather than processing it. Successfully processed messages end up getting pgmq.archive()'d. Soon, we will be integrating https://github.com/tembo-io/pg_tier https://github.com/tembo-io/pg_tier into pgmq so that the archive table is put directly into cloud storage/S3.

chuckhend 2y ago

If the message never reaches the queue (network error, database is down, app is down, etc), then yes that is a 0 delivery scenario. Once the message reaches the queue though, it is guaranteed that only a single consumer can read the message for the duration of the visibility timeout. FOR UPDATE guarantees only a single consumer can read the record, and the visibility timeout means we don't have to hold a lock. After that visibility timeout expires, it is an at-least-once scenario. Any suggestion for how we could change the verbiage on the readme to make that more clear?

Fire-Dragon-DoL 2y ago
You are talking about distributed systems, nobody expects to read "exactly once" delivery. If I read that on the docs, I consider that a huge red flag. And the fact is that what you describe is a performance optimization, I still have to write my code so that it is idempotent, so that optimization does not affect me in any other way, because exactly once is not a thing. All of this to say, I'm not even sure it's worth mentioning?
- peter_l_downs 2y ago
  You should let the Apache Flink team know, they mention exactly-once processing on their home page (under "correctness guarantees") and in their list of features. [0] https://flink.apache.org/ https://flink.apache.org/ [1] https://flink.apache.org/what-is-flink/flink-applications/#building-blocks-for-streaming-applications https://flink.apache.org/what-is-flink/flink-applications/#b...
  - sethammons 2y ago
    A common mistake: Exactly once delivery is not the same as exactly once processing. Exactly once delivery is an impossibility; see the two generals problem. Exactly once processing is usually at-least-once delivery coupled with deduplication or an otherwise idempotent action
  - Fire-Dragon-DoL 2y ago
    They are not the first or the last. You CAN have "at most once" processing, since there is always the option of processing zero messages due to systems being down, but it's a joint effort, nobody is able to offer it "for free" (as in, no effort to implement it)

cryptonector 2y ago

Quite. The problem with a visibility timeout is that there can be delays in handling a message, and if the original winner fails to do it in time then some other process/thread will, yes, but if processing has external side-effects that can't be undone then the at-most-once guarantee will definitely be violated. And if you run out of storage space and messages have to get dropped then you'll definitely violate the at-least-once guarantee.