3 ms·

I'm looking for something like kafka with an at least once guarantee. I believe this can be achieved with the kafka java client (not sure on that) but librdkaf

by nope_42 10y ago

I'm looking for something like kafka with an at least once guarantee. I believe this can be achieved with the kafka java client (not sure on that) but librdkafka (C++ client) doesn't seem to support this guarantee. Performance is secondary to messages not getting dropped in my use cases. What kind of guarantees does tank make?

agibsonccc 10y ago
FWIW exactly once is being worked on now: https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+E...
markpapadakis 10y ago
The Tank Client will immediately publish to Tank (it doesn't buffer requests). You get at least once semantics /w Tank(exactly once pretty much means at least once but with dupes-detection).
- nope_42 10y ago
  So if I have a subscriber that simply publishes a transformed message onto another topic I can have a guarantee that if the publish fails it wont move on to the next message in the subscription?
  - markpapadakis 10y ago
    The consumers applications (which interface with Tank brokers via a Tank client library) are responsible for maintaining the sequence number they are consuming from. Suppose one such application is consuming every new event that's publishing ("tailing the log"). As soon as another application successfully publishes 1 or more new messages, the consumer will get them immediately. If the application that attempted to publish failed to do so, or didn't get an ACK for success, then you are guaranteed that no new message(s) were published(i.e no partial message content). I am not sure if that answers your question, if not, can you please elaborate?
    - nope_42 10y ago
      I believe so. I suppose I'm asking for an abstraction that makes maintaining the sequence number simple and fails safely in the presence of errors. I'd basically like to be able to map messages from one topic to another with a guarantee that none of those messages will be lost; even when some error occurs (either a programming error, system downtime, or network partitions). I'd prefer the application to stop producing messages than lose any of them. It sounds like that is possible with Tank so I may end up giving it a try.
  - deleted 10y ago
    [deleted]

by nope_42 10y ago

agibsonccc 10y ago

FWIW exactly once is being worked on now: https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+E...

markpapadakis 10y ago

The Tank Client will immediately publish to Tank (it doesn't buffer requests). You get at least once semantics /w Tank(exactly once pretty much means at least once but with dupes-detection).

nope_42 10y ago
So if I have a subscriber that simply publishes a transformed message onto another topic I can have a guarantee that if the publish fails it wont move on to the next message in the subscription?
- markpapadakis 10y ago
  The consumers applications (which interface with Tank brokers via a Tank client library) are responsible for maintaining the sequence number they are consuming from. Suppose one such application is consuming every new event that's publishing ("tailing the log"). As soon as another application successfully publishes 1 or more new messages, the consumer will get them immediately. If the application that attempted to publish failed to do so, or didn't get an ACK for success, then you are guaranteed that no new message(s) were published(i.e no partial message content). I am not sure if that answers your question, if not, can you please elaborate?
  - nope_42 10y ago
    I believe so. I suppose I'm asking for an abstraction that makes maintaining the sequence number simple and fails safely in the presence of errors. I'd basically like to be able to map messages from one topic to another with a guarantee that none of those messages will be lost; even when some error occurs (either a programming error, system downtime, or network partitions). I'd prefer the application to stop producing messages than lose any of them. It sounds like that is possible with Tank so I may end up giving it a try.
- deleted 10y ago
  [deleted]