5 ms·

I mean the following (let me simplify it). You pull stuff from Kafka in batches of 4 and immediately send each message to a parallel worker, first parsing them

I mean the following (let me simplify it). You pull stuff from Kafka in batches of 4 and immediately send each message to a parallel worker, first parsing them in-thread and sequentially and these are the results of the parsing: 1. ok 2. error 3. ok 4. ok I cannot not-ACK message#2 because that means message#1 is not ACK-ed as well. Does NATS solve this? F.ex. can I get a reference to each message in my parallel workers for them to also say "I am not ACK-ing this because I failed processing it, let the next batch include it again"?

osigurdson 2y ago
Yes. This will work fine. Each message is ACK-ed.
- pdimitar 2y ago
  Thanks. I'm asking because in Kafka if you ACK a message at offset 15 then all messages from 1 to 14 are ACK-ed as well. You can't just say "ACK all from 1 to 15 except 9". But if NATS supports that use case then great, I'll migrate to it for that reason alone.
rcombatwombat 2y ago
Yes, one of the many differences (advantages) of NATS JetStream over Kafka: with NATS you can ack explicitly each message individually, and even better if you set your stream to be in 'work-queue' mode it will also automatically (and atomically) delete the ack'd message from the stream (i.e. like a 'proper' queue) another difference with Kafka where you can't delete individual messages in the middle of a stream (only trim the tail end). You can also 'negative ack' messages, specify a back-off period before the message is re-delivered (because NATS automatically re-delivers un-acked (or nacked) messages) when you can't temporarily process it, or 'term' a message (don't try to re-deliver it, e.g. because the payload is bad), or even 'ask for more time before needing to ack the message (if you are temporarily too slow at processing the message).
- pdimitar 2y ago
  Ohhh, this is an awesomely informative and concrete message! Extremely useful, thank you! I like everything about this: the ability to NACK individual messages, the specifying of a backoff period, _and_ to just discard a message f.ex. if you really cannot do anything about it. Super nice. I am grateful.

osigurdson 2y ago

Yes. This will work fine. Each message is ACK-ed.

pdimitar 2y ago
Thanks. I'm asking because in Kafka if you ACK a message at offset 15 then all messages from 1 to 14 are ACK-ed as well. You can't just say "ACK all from 1 to 15 except 9". But if NATS supports that use case then great, I'll migrate to it for that reason alone.

rcombatwombat 2y ago

Yes, one of the many differences (advantages) of NATS JetStream over Kafka: with NATS you can ack explicitly each message individually, and even better if you set your stream to be in 'work-queue' mode it will also automatically (and atomically) delete the ack'd message from the stream (i.e. like a 'proper' queue) another difference with Kafka where you can't delete individual messages in the middle of a stream (only trim the tail end). You can also 'negative ack' messages, specify a back-off period before the message is re-delivered (because NATS automatically re-delivers un-acked (or nacked) messages) when you can't temporarily process it, or 'term' a message (don't try to re-deliver it, e.g. because the payload is bad), or even 'ask for more time before needing to ack the message (if you are temporarily too slow at processing the message).

pdimitar 2y ago
Ohhh, this is an awesomely informative and concrete message! Extremely useful, thank you! I like everything about this: the ability to NACK individual messages, the specifying of a backoff period, _and_ to just discard a message f.ex. if you really cannot do anything about it. Super nice. I am grateful.