3 ms·

Say I need to add messaging with persistence (at least once delivery) for my stack (multiple services, often but not always on the same machine), and I want the

Say I need to add messaging with persistence (at least once delivery) for my stack (multiple services, often but not always on the same machine), and I want the stack to not have external dependencies-so it's easy to install. Does liftbridge fill this niche? Would NATS streaming be a better solution or maybe Akka with Persistence module?

paddybyers 7y ago
I can't speak for NATS streaming or Akka but it does look like liftbridge can fit that niche very well, depending on what features and semantics you need. Your services that publish can be assured that a message has been persisted to a quorum of replicas (strictly, the in-sync replicas) before receiving an acknowledgement, so you are able to assure that all onward processing to your downstream services occurs at least once for all acknowledged messages. If you require fanout to your stream consumers then this is supported simply by having multiple consumers subscribe to each stream. If you require a competing consumers model then this isn't directly supported, but you can have load-balance groups which you could use to achieve something similar, but with a static configuration - that is, your consumer population can't resize dynamically, but you can distribute messages to a fixed set of streams, each of which has a consumer. liftbridge doesn't yet support durable consumer-offset checkpointing, although this is advertised as being on the roadmap. In the meantime, if you wanted the consumer offset to survive consumer restarts, then the consumer itself would need to take care of persisting its current offset.
- tylertreat 7y ago
  This is correct. Also, I'm planning to introduce a mode that runs NATS "embedded" within the Liftbridge server, such that NATS basically becomes an implementation detail of Liftbridge. Re: consumer-offset checkpointing, this will be implemented in the form of consumer groups, similar to how Kafka works where a consumer group checkpoints its position in the log by writing it to the log itself.
atombender 7y ago
I'm not familiar with Akka, but both NATS Streaming and Liftbridge sound like they'd fit your purpose. NATS on its own won't cut it, since it has no persistence and is entirely real-time (new clients receive messages published after they connect, not old messages). I've not used Liftbridge yet, but plan to. While NATS Streaming is solid, its distributed design seems a bit ramshackle, relying on a combination of Raft and a shared state (a distributed file system or an SQL database) for failover. Liftbridge seems like a better design to me.

paddybyers 7y ago

I can't speak for NATS streaming or Akka but it does look like liftbridge can fit that niche very well, depending on what features and semantics you need. Your services that publish can be assured that a message has been persisted to a quorum of replicas (strictly, the in-sync replicas) before receiving an acknowledgement, so you are able to assure that all onward processing to your downstream services occurs at least once for all acknowledged messages. If you require fanout to your stream consumers then this is supported simply by having multiple consumers subscribe to each stream. If you require a competing consumers model then this isn't directly supported, but you can have load-balance groups which you could use to achieve something similar, but with a static configuration - that is, your consumer population can't resize dynamically, but you can distribute messages to a fixed set of streams, each of which has a consumer. liftbridge doesn't yet support durable consumer-offset checkpointing, although this is advertised as being on the roadmap. In the meantime, if you wanted the consumer offset to survive consumer restarts, then the consumer itself would need to take care of persisting its current offset.

tylertreat 7y ago
This is correct. Also, I'm planning to introduce a mode that runs NATS "embedded" within the Liftbridge server, such that NATS basically becomes an implementation detail of Liftbridge. Re: consumer-offset checkpointing, this will be implemented in the form of consumer groups, similar to how Kafka works where a consumer group checkpoints its position in the log by writing it to the log itself.

atombender 7y ago

I'm not familiar with Akka, but both NATS Streaming and Liftbridge sound like they'd fit your purpose. NATS on its own won't cut it, since it has no persistence and is entirely real-time (new clients receive messages published after they connect, not old messages). I've not used Liftbridge yet, but plan to. While NATS Streaming is solid, its distributed design seems a bit ramshackle, relying on a combination of Raft and a shared state (a distributed file system or an SQL database) for failover. Liftbridge seems like a better design to me.