3 ms·

While I generally agree with the idea that a direct HTTP request for a single point to point message would be more reliable than routing through a message queue

by tinco 3y ago

While I generally agree with the idea that a direct HTTP request for a single point to point message would be more reliable than routing through a message queue, I'm a little bit surprised that there would be so many messages lost by NATS it had significant impact on their services. Wouldn't a lost message just mean NATS would retry delivery until succesful? Anyone know why they would experience noticeable unreliability?

dilyevsky 3y ago
> Wouldn't a lost message just mean NATS would retry delivery until succesful? Anyone know why they would experience noticeable unreliability? I believe if you're using core nats (not JetStream) there's no option for re-delivering like at all.
- tinco 3y ago
  Well if that's what they did, then then using NATS over HTTP in that scenario is just switching a single point of failure for two single points of failure with no feedback on the second point, did they just pick it for the convenience of the NATS interface and only later realize their mistake?
  - emmanueloga_ 3y ago
    I’m guessing they realized late that core nats is a message broker without features like durability and at least once delivery. Jetstream was released less than a year ago (in nats 2.2) so perhaps by the time they made the switch it was not an option.
    - caleblloyd 3y ago
      NATS 2.2 with initial JetStream support was actually released 3 years ago.
klabb3 3y ago
I am also very curious to know more. I’m sure the NATS maintainers are as well. Their architecture is extremely intuitive and appealing to me, so I wonder where things went south. Nats has a lot of tunable parameters with Jetstream. For instance, an in-memory stream with a time-based duplicate detection window, both push and pull semantics, and configurable re-delivery and ack policies. The one thing where I can see an impedance mismatch is ephemeral single-message connections which I don’t think it’s built for. In either case, more details would be invaluable. (Fly folks, please consider sharing!)
tptacek 3y ago
We're not dunking on NATS. We were probably holding it wrong. But it turns out, we didn't need it; a message layer wasn't really making things any more expressive, just harder to test and monitor.
- re5i5tor 3y ago
  "holding it wrong" OMG ;-)

by tinco 3y ago

dilyevsky 3y ago

> Wouldn't a lost message just mean NATS would retry delivery until succesful? Anyone know why they would experience noticeable unreliability? I believe if you're using core nats (not JetStream) there's no option for re-delivering like at all.

tinco 3y ago
Well if that's what they did, then then using NATS over HTTP in that scenario is just switching a single point of failure for two single points of failure with no feedback on the second point, did they just pick it for the convenience of the NATS interface and only later realize their mistake?
- emmanueloga_ 3y ago
  I’m guessing they realized late that core nats is a message broker without features like durability and at least once delivery. Jetstream was released less than a year ago (in nats 2.2) so perhaps by the time they made the switch it was not an option.
  - caleblloyd 3y ago
    NATS 2.2 with initial JetStream support was actually released 3 years ago.

klabb3 3y ago

I am also very curious to know more. I’m sure the NATS maintainers are as well. Their architecture is extremely intuitive and appealing to me, so I wonder where things went south. Nats has a lot of tunable parameters with Jetstream. For instance, an in-memory stream with a time-based duplicate detection window, both push and pull semantics, and configurable re-delivery and ack policies. The one thing where I can see an impedance mismatch is ephemeral single-message connections which I don’t think it’s built for. In either case, more details would be invaluable. (Fly folks, please consider sharing!)

tptacek 3y ago

We're not dunking on NATS. We were probably holding it wrong. But it turns out, we didn't need it; a message layer wasn't really making things any more expressive, just harder to test and monitor.

re5i5tor 3y ago
"holding it wrong" OMG ;-)