3 ms·

That's an inherent issue with distributed solutions and is impossible to solve. The only way to deal with it is using various techniques like acknowledgements,

That's an inherent issue with distributed solutions and is impossible to solve. The only way to deal with it is using various techniques like acknowledgements, retries, local storage, idempotency, etc. MSMQ handles that stuff behind the scenes but the problem itself will always exist if there's a network boundary. These other systems are designed to be remote with a network interface. You can use the client drivers to handle acknowledgements/retries/local-buffering in your own app or use something like Logstash [1], FluentD [2], or Vector [3] for message forwarding if you want a local agent to send to. You might have to wire up several connectors since none of them forward directly to Pulsar today. Also RabbitMQ is absolute crap. There are better options for every scenario so I advise using something else like Redis, NATS, Kafka, or Pulsar. 1. https://www.elastic.co/products/logstash https://www.elastic.co/products/logstash 2. https://www.fluentd.org/ https://www.fluentd.org/ 3. https://vector.dev/ https://vector.dev/

GordonS 7y ago
> Also RabbitMQ is absolute crap. There are better options for every scenario so I advise using something else like Redis, NATS, Kafka, or Pulsar Yikes, that's a bit harsh! I've been using RabbitMQ on multiple projects for several years, and I think it's a great pub/sub system. It also has a large userbase and community around it, as well as lots of plugins available. I've never heard of using Redis for pub/sub - were you suggesting rolling your own on top of Redis? I'm not familiar with NATS (but it's been mentioned several times here, so I will definitely learn more!), but Kafka is a streaming log/event system, not a pub/sub system like RabbitMQ (different use cases). I will say though that if something is wrong in your RabbitMQ config, the stack traces that Erlang produces when it crashes are a nightmare to decipher!
- manigandham 7y ago
  Just because it's widely used doesn't mean it's good (see PHP). It's slow with single-threaded topics that usually max out around 25k msgs/sec, fragile with dropped connections, stalled queues, corrupted data, terrible clustering that breaks often and doesn't support sharding, has a silly HIPE mode where you can choose to compile the Erlang code for more performance which turns startup time into minutes, etc. It's poor architecture and implementation. One of the worst products built with Erlang. It's hard to work on (for new contributors) while not really using any of the natural advantages of the language. Redis supports pub/sub channels. It's very fast and if you already use it then it saves running another system. Streaming log vs pub/sub is a fuzzy distinction and basically has no difference at this point. Publishers send to a topic and consumers listen. You can do the ephemeral pub/sub in Kafka by reducing retention to seconds, using a single partition per pub/sub topic, and having consumers listen from latest without a consumer group. Or use Pulsar which does both much more naturally. Also I didn't mention but there are great commercial messaging products like AMPS [1] or Solace [2] if you need much more advanced features and support. 1. https://www.crankuptheamps.com/ https://www.crankuptheamps.com/ 2. https://solace.com/ https://solace.com/
  - GordonS 7y ago
    I didn't k ow about Redis pub/sub, I'll check it out! With a quick glance, I'm not sure it supports message durability (persisting messages to storage)? IME, any Erlang system is difficult for noobs to contribute to; Erlang's syntax alone is... frightening :) I really haven't hit any of the issues you mentioned with rabbit. Several clusters in production for years have been rock solid, connections are stable (and when they do drop, the client libraries, of which there are a multitude, can handle reconnection), throughput is excellent, and I've never once seen data corruption (and combined we must have processed many billions of messages). The whole HIPE debacle I can at least agree on though! (I seem to recall it's deprecated now, but I might have dreamed that)
    - manigandham 7y ago
      Redis pub/sub is ephemeral. Send to a topic and any active listeners will receive. If you need persistence then Redis v5 has a new data type called Streams which is similar to Kafka/Pulsar. Push messages to a stream and read with multiple consumers (using consumer groups) that can ack each msg to track read state.

GordonS 7y ago

> Also RabbitMQ is absolute crap. There are better options for every scenario so I advise using something else like Redis, NATS, Kafka, or Pulsar Yikes, that's a bit harsh! I've been using RabbitMQ on multiple projects for several years, and I think it's a great pub/sub system. It also has a large userbase and community around it, as well as lots of plugins available. I've never heard of using Redis for pub/sub - were you suggesting rolling your own on top of Redis? I'm not familiar with NATS (but it's been mentioned several times here, so I will definitely learn more!), but Kafka is a streaming log/event system, not a pub/sub system like RabbitMQ (different use cases). I will say though that if something is wrong in your RabbitMQ config, the stack traces that Erlang produces when it crashes are a nightmare to decipher!

manigandham 7y ago
Just because it's widely used doesn't mean it's good (see PHP). It's slow with single-threaded topics that usually max out around 25k msgs/sec, fragile with dropped connections, stalled queues, corrupted data, terrible clustering that breaks often and doesn't support sharding, has a silly HIPE mode where you can choose to compile the Erlang code for more performance which turns startup time into minutes, etc. It's poor architecture and implementation. One of the worst products built with Erlang. It's hard to work on (for new contributors) while not really using any of the natural advantages of the language. Redis supports pub/sub channels. It's very fast and if you already use it then it saves running another system. Streaming log vs pub/sub is a fuzzy distinction and basically has no difference at this point. Publishers send to a topic and consumers listen. You can do the ephemeral pub/sub in Kafka by reducing retention to seconds, using a single partition per pub/sub topic, and having consumers listen from latest without a consumer group. Or use Pulsar which does both much more naturally. Also I didn't mention but there are great commercial messaging products like AMPS [1] or Solace [2] if you need much more advanced features and support. 1. https://www.crankuptheamps.com/ https://www.crankuptheamps.com/ 2. https://solace.com/ https://solace.com/
- GordonS 7y ago
  I didn't k ow about Redis pub/sub, I'll check it out! With a quick glance, I'm not sure it supports message durability (persisting messages to storage)? IME, any Erlang system is difficult for noobs to contribute to; Erlang's syntax alone is... frightening :) I really haven't hit any of the issues you mentioned with rabbit. Several clusters in production for years have been rock solid, connections are stable (and when they do drop, the client libraries, of which there are a multitude, can handle reconnection), throughput is excellent, and I've never once seen data corruption (and combined we must have processed many billions of messages). The whole HIPE debacle I can at least agree on though! (I seem to recall it's deprecated now, but I might have dreamed that)
  - manigandham 7y ago
    Redis pub/sub is ephemeral. Send to a topic and any active listeners will receive. If you need persistence then Redis v5 has a new data type called Streams which is similar to Kafka/Pulsar. Push messages to a stream and read with multiple consumers (using consumer groups) that can ack each msg to track read state.