4 ms·

This might be entirely off topic, but I'm having issues using RabbitMQ whereby durability suffers because messages are sent to remote hosts thus exposing them t

This might be entirely off topic, but I'm having issues using RabbitMQ whereby durability suffers because messages are sent to remote hosts thus exposing them to both the network and remote host availability. On a previous platform I used an MSMQ based system which didn't have this problem since it uses a local store and forward service. So all sends are to localhost and are not affected by the network or the receiver availability. The MSMQ system was my first and only experience with messaging up to now, so I was surprised that any system would not work that way. How is this dealt with in other systems? Is it just a feature that exists or not and you just decide if it's important? And maybe just to shoe horn it to be on topic, does Pulsar use a local service?

bauerd 7y ago
You're free to have queue and workers run on the same machine, just bind to loopback. As soon as you deal with more than one machine, which is required in HA scenarios, you deal with a networked (distributed) system. I might not have understood your question correctly though ... Edit: Maybe you're looking for acks/confirms? https://www.rabbitmq.com/confirms.html https://www.rabbitmq.com/confirms.html
- mbostleman 7y ago
  I have many machines, each of which have one or many applications that send messages. And I have one machine with an instance of Rabbit to which all messages are sent. If the network is down or the Rabbit machine is down, the messages are gone along with their data. Clustering the Rabbit machine helps one particular failure scenario, but it's not a solution to the problem.
  - shantly 7y ago
    I haven't worked with RabbitMQ in years, but IIRC I solved this with federation. My sends were all to localhost or a VM on the same machine. That solves the problem of lost messages if you have e.g. a network problem between servers. I don't know what to do about RabbitMQ being down, all you can really do about that is move the exact same problem somewhere else by making things much more complicated, such that it's probably not worth it. If the place you're storing outbound messages is broken, you (obviously) can't store them, whether that's PostgreSQL, Redis, a remote RabbitMQ server, a local RabbitMQ, whatever.
manigandham 7y ago
That's an inherent issue with distributed solutions and is impossible to solve. The only way to deal with it is using various techniques like acknowledgements, retries, local storage, idempotency, etc. MSMQ handles that stuff behind the scenes but the problem itself will always exist if there's a network boundary. These other systems are designed to be remote with a network interface. You can use the client drivers to handle acknowledgements/retries/local-buffering in your own app or use something like Logstash [1], FluentD [2], or Vector [3] for message forwarding if you want a local agent to send to. You might have to wire up several connectors since none of them forward directly to Pulsar today. Also RabbitMQ is absolute crap. There are better options for every scenario so I advise using something else like Redis, NATS, Kafka, or Pulsar. 1. https://www.elastic.co/products/logstash https://www.elastic.co/products/logstash 2. https://www.fluentd.org/ https://www.fluentd.org/ 3. https://vector.dev/ https://vector.dev/
- GordonS 7y ago
  > Also RabbitMQ is absolute crap. There are better options for every scenario so I advise using something else like Redis, NATS, Kafka, or Pulsar Yikes, that's a bit harsh! I've been using RabbitMQ on multiple projects for several years, and I think it's a great pub/sub system. It also has a large userbase and community around it, as well as lots of plugins available. I've never heard of using Redis for pub/sub - were you suggesting rolling your own on top of Redis? I'm not familiar with NATS (but it's been mentioned several times here, so I will definitely learn more!), but Kafka is a streaming log/event system, not a pub/sub system like RabbitMQ (different use cases). I will say though that if something is wrong in your RabbitMQ config, the stack traces that Erlang produces when it crashes are a nightmare to decipher!
  - manigandham 7y ago
    Just because it's widely used doesn't mean it's good (see PHP). It's slow with single-threaded topics that usually max out around 25k msgs/sec, fragile with dropped connections, stalled queues, corrupted data, terrible clustering that breaks often and doesn't support sharding, has a silly HIPE mode where you can choose to compile the Erlang code for more performance which turns startup time into minutes, etc. It's poor architecture and implementation. One of the worst products built with Erlang. It's hard to work on (for new contributors) while not really using any of the natural advantages of the language. Redis supports pub/sub channels. It's very fast and if you already use it then it saves running another system. Streaming log vs pub/sub is a fuzzy distinction and basically has no difference at this point. Publishers send to a topic and consumers listen. You can do the ephemeral pub/sub in Kafka by reducing retention to seconds, using a single partition per pub/sub topic, and having consumers listen from latest without a consumer group. Or use Pulsar which does both much more naturally. Also I didn't mention but there are great commercial messaging products like AMPS [1] or Solace [2] if you need much more advanced features and support. 1. https://www.crankuptheamps.com/ https://www.crankuptheamps.com/ 2. https://solace.com/ https://solace.com/
NicoJuicy 7y ago
Look up : outbox pattern
- GordonS 7y ago
  Was going to suggest this too. I'm build a distributed system with RabbitMQ just now, where producers may be offline due to transient networking issues. I write messages to a local SQLite database, with another thread responsible for sending them to RabbitMQ and deleting them on successful delivery.
  - NicoJuicy 7y ago
    Well, Postgres has an event source system. NoSql for this seems to be a better use-case. I also depend on nats.io instead of RabbitMQ
    - GordonS 7y ago
      Not sure what you're getting at with NoSql here - how does NoSql vs relational affect the choice of message broker/queue? My (very limited, and purely gained through this thread!) understanding of nats suggests it doesn't handle persistence, instead leaving it up to producers and consumers - but I need to not lose any messages in the event of an outage. Waiting for consumers to signal completion of processing isn't practical, as it will sometimes take several minutes or even hours before messages are processed; I think the real solution here is persistence at the broker?
      - NicoJuicy 7y ago
        I'm talking about the backend for storing the messages/events in the outbox pattern. That's how to persist in case of a network failure. The outbox-pattern pattern removes the send message from it's datastore when send/acknowledged. Persistency is different from the outbox-pattern, it handles what is does if the message broker received the message. The outbox pattern handles failure before the message reached the message broker. NoSql and event stores normally have methods to adjust data to the latest version of the model, eg. Postgress ( which can be used as NoSql using bson) supports js v8 transformations. This is useful for edge-cases in the outbox pattern. In short, I was more or less talking about the tech used. Rabbit vs Nats and sqllite vs NoSql ( in my case, using Postgress as NoSql). Ps. Nats can be persistent as others mentioned in this thread, look up: nats streaming

bauerd 7y ago

You're free to have queue and workers run on the same machine, just bind to loopback. As soon as you deal with more than one machine, which is required in HA scenarios, you deal with a networked (distributed) system. I might not have understood your question correctly though ... Edit: Maybe you're looking for acks/confirms? https://www.rabbitmq.com/confirms.html https://www.rabbitmq.com/confirms.html

mbostleman 7y ago
I have many machines, each of which have one or many applications that send messages. And I have one machine with an instance of Rabbit to which all messages are sent. If the network is down or the Rabbit machine is down, the messages are gone along with their data. Clustering the Rabbit machine helps one particular failure scenario, but it's not a solution to the problem.
- shantly 7y ago
  I haven't worked with RabbitMQ in years, but IIRC I solved this with federation. My sends were all to localhost or a VM on the same machine. That solves the problem of lost messages if you have e.g. a network problem between servers. I don't know what to do about RabbitMQ being down, all you can really do about that is move the exact same problem somewhere else by making things much more complicated, such that it's probably not worth it. If the place you're storing outbound messages is broken, you (obviously) can't store them, whether that's PostgreSQL, Redis, a remote RabbitMQ server, a local RabbitMQ, whatever.

manigandham 7y ago

That's an inherent issue with distributed solutions and is impossible to solve. The only way to deal with it is using various techniques like acknowledgements, retries, local storage, idempotency, etc. MSMQ handles that stuff behind the scenes but the problem itself will always exist if there's a network boundary. These other systems are designed to be remote with a network interface. You can use the client drivers to handle acknowledgements/retries/local-buffering in your own app or use something like Logstash [1], FluentD [2], or Vector [3] for message forwarding if you want a local agent to send to. You might have to wire up several connectors since none of them forward directly to Pulsar today. Also RabbitMQ is absolute crap. There are better options for every scenario so I advise using something else like Redis, NATS, Kafka, or Pulsar. 1. https://www.elastic.co/products/logstash https://www.elastic.co/products/logstash 2. https://www.fluentd.org/ https://www.fluentd.org/ 3. https://vector.dev/ https://vector.dev/

GordonS 7y ago
> Also RabbitMQ is absolute crap. There are better options for every scenario so I advise using something else like Redis, NATS, Kafka, or Pulsar Yikes, that's a bit harsh! I've been using RabbitMQ on multiple projects for several years, and I think it's a great pub/sub system. It also has a large userbase and community around it, as well as lots of plugins available. I've never heard of using Redis for pub/sub - were you suggesting rolling your own on top of Redis? I'm not familiar with NATS (but it's been mentioned several times here, so I will definitely learn more!), but Kafka is a streaming log/event system, not a pub/sub system like RabbitMQ (different use cases). I will say though that if something is wrong in your RabbitMQ config, the stack traces that Erlang produces when it crashes are a nightmare to decipher!
- manigandham 7y ago
  Just because it's widely used doesn't mean it's good (see PHP). It's slow with single-threaded topics that usually max out around 25k msgs/sec, fragile with dropped connections, stalled queues, corrupted data, terrible clustering that breaks often and doesn't support sharding, has a silly HIPE mode where you can choose to compile the Erlang code for more performance which turns startup time into minutes, etc. It's poor architecture and implementation. One of the worst products built with Erlang. It's hard to work on (for new contributors) while not really using any of the natural advantages of the language. Redis supports pub/sub channels. It's very fast and if you already use it then it saves running another system. Streaming log vs pub/sub is a fuzzy distinction and basically has no difference at this point. Publishers send to a topic and consumers listen. You can do the ephemeral pub/sub in Kafka by reducing retention to seconds, using a single partition per pub/sub topic, and having consumers listen from latest without a consumer group. Or use Pulsar which does both much more naturally. Also I didn't mention but there are great commercial messaging products like AMPS [1] or Solace [2] if you need much more advanced features and support. 1. https://www.crankuptheamps.com/ https://www.crankuptheamps.com/ 2. https://solace.com/ https://solace.com/

NicoJuicy 7y ago

Look up : outbox pattern

GordonS 7y ago
Was going to suggest this too. I'm build a distributed system with RabbitMQ just now, where producers may be offline due to transient networking issues. I write messages to a local SQLite database, with another thread responsible for sending them to RabbitMQ and deleting them on successful delivery.
- NicoJuicy 7y ago
  Well, Postgres has an event source system. NoSql for this seems to be a better use-case. I also depend on nats.io instead of RabbitMQ
  - GordonS 7y ago
    Not sure what you're getting at with NoSql here - how does NoSql vs relational affect the choice of message broker/queue? My (very limited, and purely gained through this thread!) understanding of nats suggests it doesn't handle persistence, instead leaving it up to producers and consumers - but I need to not lose any messages in the event of an outage. Waiting for consumers to signal completion of processing isn't practical, as it will sometimes take several minutes or even hours before messages are processed; I think the real solution here is persistence at the broker?
    - NicoJuicy 7y ago
      I'm talking about the backend for storing the messages/events in the outbox pattern. That's how to persist in case of a network failure. The outbox-pattern pattern removes the send message from it's datastore when send/acknowledged. Persistency is different from the outbox-pattern, it handles what is does if the message broker received the message. The outbox pattern handles failure before the message reached the message broker. NoSql and event stores normally have methods to adjust data to the latest version of the model, eg. Postgress ( which can be used as NoSql using bson) supports js v8 transformations. This is useful for edge-cases in the outbox pattern. In short, I was more or less talking about the tech used. Rabbit vs Nats and sqllite vs NoSql ( in my case, using Postgress as NoSql). Ps. Nats can be persistent as others mentioned in this thread, look up: nats streaming