4 ms·

We turned off Pub/Sub and nobody noticed

44 points by patrickhamann 2mo ago

8e8nyc 2mo ago
[dead]
rorymalcolm 2mo ago
Disclaimer: I work at the incident company This is one of those projects where the end goal is to build the scariest button possible. You cannot complete the project until you’ve pressed that button. You do all the work to build the button, then you have to press it. Never not scary, never not eventually fun, but type 2 fun for sure.
shintoist 2mo ago
NATS is such a cool piece of technology that deserves more hype than it gets
- jdon 2mo ago
  Given their recentish argument with the CNCF[0] not sure how much I would trust it. [0] https://www.cncf.io/blog/2025/05/01/protecting-nats-and-the-integrity-of-open-source-cncfs-commitment-to-the-community/ https://www.cncf.io/blog/2025/05/01/protecting-nats-and-the-...
  - Aeolun 1mo ago
    But that's the company that build the thing, which has apparently been told off by the CNCF? Seems like for once the thing functioned as it was intended to function?
- BerislavLopac 1mo ago
  I still can't get over the demise of Memphis [0]. :'( [0] https://github.com/superstreamlabs/memphis https://github.com/superstreamlabs/memphis
- jamescun 1mo ago
  Second for NATS. I've deployed Core NATS as few times now at scale, including within financial trading. It's been a dream to operate. I have limited experience with JetStream, but coming from Kafka and Rabbit MQ, it seemed nearly as featureful but also easier to maintain.
- DavyJone 1mo ago
  I really like NATS and I agree, but im generally hesitant to use it in critical systems due to https://jepsen.io/analyses/nats-2.12.1 https://jepsen.io/analyses/nats-2.12.1 What is your take on this?
trio8453 1mo ago
[dead]
DavyJone 1mo ago
I like the article, but I think there is a clear misconception of what a SPOF is, and I've noticed this happening across many engineers and engineering teams lately. A system, like Pub/Sub, being a critical dependency or hard dependency does not make it a SPOF. Otherwise any system is a SPOF, creating absurd architectures that end up with more complexity and downtime than the original one. For example, I assume incident.io are yet to see if their added complexity of multiple event streams does not cause worse availability issues than what they had, by introducing complexity like problems with data consistency. If the the underlying "system" components are not redundant and highly available, then you have a SPOF. eg. A single machine is what provides Pub/Sub. In this case, what you have is a hard-dependancy with an avaialbility lower than what you want to achieve, but there are always shared hard-dependancies that they will have. There is no way around this. What we should aim for is that any system that is a critical dependancy or a hard dependancy with such a big blast radious, to be redundant and robust in itself. Not just the system, but the practices around it.
deleted 1mo ago
[deleted]

44 points by patrickhamann 2mo ago

[dead]

Disclaimer: I work at the incident company This is one of those projects where the end goal is to build the scariest button possible. You cannot complete the project until you’ve pressed that button. You do all the work to build the button, then you have to press it. Never not scary, never not eventually fun, but type 2 fun for sure.

shintoist 2mo ago

NATS is such a cool piece of technology that deserves more hype than it gets

jdon 2mo ago
Given their recentish argument with the CNCF[0] not sure how much I would trust it. [0] https://www.cncf.io/blog/2025/05/01/protecting-nats-and-the-integrity-of-open-source-cncfs-commitment-to-the-community/ https://www.cncf.io/blog/2025/05/01/protecting-nats-and-the-...
- Aeolun 1mo ago
  But that's the company that build the thing, which has apparently been told off by the CNCF? Seems like for once the thing functioned as it was intended to function?
BerislavLopac 1mo ago
I still can't get over the demise of Memphis [0]. :'( [0] https://github.com/superstreamlabs/memphis https://github.com/superstreamlabs/memphis
jamescun 1mo ago
Second for NATS. I've deployed Core NATS as few times now at scale, including within financial trading. It's been a dream to operate. I have limited experience with JetStream, but coming from Kafka and Rabbit MQ, it seemed nearly as featureful but also easier to maintain.
DavyJone 1mo ago
I really like NATS and I agree, but im generally hesitant to use it in critical systems due to https://jepsen.io/analyses/nats-2.12.1 https://jepsen.io/analyses/nats-2.12.1 What is your take on this?

trio8453 1mo ago

[dead]

DavyJone 1mo ago

I like the article, but I think there is a clear misconception of what a SPOF is, and I've noticed this happening across many engineers and engineering teams lately. A system, like Pub/Sub, being a critical dependency or hard dependency does not make it a SPOF. Otherwise any system is a SPOF, creating absurd architectures that end up with more complexity and downtime than the original one. For example, I assume incident.io are yet to see if their added complexity of multiple event streams does not cause worse availability issues than what they had, by introducing complexity like problems with data consistency. If the the underlying "system" components are not redundant and highly available, then you have a SPOF. eg. A single machine is what provides Pub/Sub. In this case, what you have is a hard-dependancy with an avaialbility lower than what you want to achieve, but there are always shared hard-dependancies that they will have. There is no way around this. What we should aim for is that any system that is a critical dependancy or a hard dependancy with such a big blast radious, to be redundant and robust in itself. Not just the system, but the practices around it.

deleted 1mo ago

[deleted]