4 ms·

Thanks. That confirms pretty much the picture I had in my head of how Redis works these days. It's frustrating that there's no obvious middle ground this and K

Thanks. That confirms pretty much the picture I had in my head of how Redis works these days. It's frustrating that there's no obvious middle ground this and Kafka and Pulsar, both of which are memory-hungry JVM apps with multiple external dependencies. Both require ZooKeeper; Pulsar also requires BookKeeper. None of these components are operationally simple. I'm a fan of NATS itself, but NATS Streaming's clustering design leaves a lot to be desired. In particular, it punts on failover/HA, asking you to instead run an SQL database or shared file system that provides this. (An obvious low-maintenance option here would be CockroachDB, but NATS doesn't support it.)

manigandham 7y ago
There are other options. RabbitMQ [1] is the usual although clustering has always been fragile. If you dont care about open-source then there are plenty of other options like AMPS [2] or Solace [3]. The latter has a free edition. 1. https://www.rabbitmq.com/ https://www.rabbitmq.com/ 2. http://www.crankuptheamps.com/ http://www.crankuptheamps.com/ 3. https://solace.com/ https://solace.com/
- atombender 7y ago
  I wouldn't put RabbitMQ in this category — it is a classical message broker, not a log. Once you've consumed a message, it's gone, unless you have set it up so ACKed messages are funneled into another queue, but that stuff is finicky and doesn't patch over the fact that underneath it's designed for mutable, ephemeral queues. In particular, you can't peek back into the queue to find older items. You have zero visibility into the contents of the queuem, and you certainly can't treat it as a dependable database. And, as you say, fragile. I've run RMQ in production for years and I would be very happy if I could throw it out. It's the least well-behaved component in any stack I've used it in. Even Elasticsearch (shudder) is better at not losing data. Not just the clustering, either. Even for a persistent queue, RMQ will start to chug RAM for any message that is delivered but not yet ACKed, for example, making it dangerous for apps that want to batch large groups of messages for efficiency. (It seems to me that it was not designed for that at all, but for one-by-one consumption, which is of course much slower.) I'm looking for a mature distributed log that is clustered and lightweight. Kafka except, say, written in Go.
  - manigandham 7y ago
    Try out Solace then. It's fast and supports HA for the free edition with throughput limits.

manigandham 7y ago

There are other options. RabbitMQ [1] is the usual although clustering has always been fragile. If you dont care about open-source then there are plenty of other options like AMPS [2] or Solace [3]. The latter has a free edition. 1. https://www.rabbitmq.com/ https://www.rabbitmq.com/ 2. http://www.crankuptheamps.com/ http://www.crankuptheamps.com/ 3. https://solace.com/ https://solace.com/

atombender 7y ago
I wouldn't put RabbitMQ in this category — it is a classical message broker, not a log. Once you've consumed a message, it's gone, unless you have set it up so ACKed messages are funneled into another queue, but that stuff is finicky and doesn't patch over the fact that underneath it's designed for mutable, ephemeral queues. In particular, you can't peek back into the queue to find older items. You have zero visibility into the contents of the queuem, and you certainly can't treat it as a dependable database. And, as you say, fragile. I've run RMQ in production for years and I would be very happy if I could throw it out. It's the least well-behaved component in any stack I've used it in. Even Elasticsearch (shudder) is better at not losing data. Not just the clustering, either. Even for a persistent queue, RMQ will start to chug RAM for any message that is delivered but not yet ACKed, for example, making it dangerous for apps that want to batch large groups of messages for efficiency. (It seems to me that it was not designed for that at all, but for one-by-one consumption, which is of course much slower.) I'm looking for a mature distributed log that is clustered and lightweight. Kafka except, say, written in Go.
- manigandham 7y ago
  Try out Solace then. It's fast and supports HA for the free edition with throughput limits.