4 ms·

This is an excellent reminder of the danger of being locked into closed systems. I wonder how many companies (like mine) have literally ground to a halt becaus

by lylo 6y ago

This is an excellent reminder of the danger of being locked into closed systems. I wonder how many companies (like mine) have literally ground to a halt because of this? Do other companies have a risk-documented backup plan B for times like this? Presumably the default is for everyone to resort to email? More worryingly is the number of ChatOps processes and alerting/observability systems that are in place around Slack. Not being able to chat with co-workers for an hour or two is fine, but not being able to safely manage CI/CD/deployments is a big risk.

deleted 6y ago
[deleted]
Scandiravian 6y ago
I just finished an 5 hour debugging session on what turned out to be several cascading bugs in one of the older systems at my company Can't deploy the fix, because - developers trigger deployments through slack and I don't have access to the underlying deployment system - infrastructure guys who have access aren't responding to my emails
- vlod 6y ago
  Sounds like a great opportunity to work on not having Slack as the only trigger mechanism. :) Or a least document how to call the slack bot manually. (assuming it's just a http endpoint)
  - Scandiravian 6y ago
    I agree 100%! Though I think it might be dangerous to prepare for the "last disaster". It'll be some other system breaking next time, so I think we instead should identify what systems that do not have some kind of redundancy and determine blast radius of those crashing I'm good at not panicking about things I can't change, but I worry about some of my colleagues who find it difficult to not have control in these situations I can't do anything to help them at the moment, so for now I'm heading to my couch with my analogue book :)
hn_throwaway_99 6y ago
> This is an excellent reminder of the danger of being locked into closed systems. Do you honestly think a self managed solution or open source solution would be more reliable for most companies?
- NationalPark 6y ago
  When application engineers say stuff like this, they're also implying that there's a giant infra/ops team who will be wiling and able to do all the work for them. Nobody actually wants to be responsible for this stuff.
- lylo 6y ago
  Not at all, I think closed private systems are far better (better products, support, service) but when an entire company runs its operations on a single system like Slack, there is a big risk when it goes away and you need contingency. I’d still rather be on Slack and suffer a day of lost productivity than force people to use only email or IRC.

by lylo 6y ago

deleted 6y ago

[deleted]

Scandiravian 6y ago

I just finished an 5 hour debugging session on what turned out to be several cascading bugs in one of the older systems at my company Can't deploy the fix, because - developers trigger deployments through slack and I don't have access to the underlying deployment system - infrastructure guys who have access aren't responding to my emails

vlod 6y ago
Sounds like a great opportunity to work on not having Slack as the only trigger mechanism. :) Or a least document how to call the slack bot manually. (assuming it's just a http endpoint)
- Scandiravian 6y ago
  I agree 100%! Though I think it might be dangerous to prepare for the "last disaster". It'll be some other system breaking next time, so I think we instead should identify what systems that do not have some kind of redundancy and determine blast radius of those crashing I'm good at not panicking about things I can't change, but I worry about some of my colleagues who find it difficult to not have control in these situations I can't do anything to help them at the moment, so for now I'm heading to my couch with my analogue book :)

hn_throwaway_99 6y ago

> This is an excellent reminder of the danger of being locked into closed systems. Do you honestly think a self managed solution or open source solution would be more reliable for most companies?

NationalPark 6y ago
When application engineers say stuff like this, they're also implying that there's a giant infra/ops team who will be wiling and able to do all the work for them. Nobody actually wants to be responsible for this stuff.
lylo 6y ago
Not at all, I think closed private systems are far better (better products, support, service) but when an entire company runs its operations on a single system like Slack, there is a big risk when it goes away and you need contingency. I’d still rather be on Slack and suffer a day of lost productivity than force people to use only email or IRC.