3 ms·

'But then the author engages in an absurd rant about how the entire software engineering industry is a “bit of a clusterfuck”,' I mean, it is a bit of a cluste

'But then the author engages in an absurd rant about how the entire software engineering industry is a “bit of a clusterfuck”,' I mean, it is a bit of a cluster fuck. Most of the issues I've seen are due to speed or cost. We don't have time for tests.we don't have time to record the business requirements in a collective place (just look through multiple JIRA stories and piece it together). We don't have money for dedicated QA roles. So of course problems happen. Luckily my team only works on a lower criticality site. None of this is really a dev's fault. It's leadership and the culture they incentivise.

acdha 2y ago
I blame the courts largely buying the argument that companies can disclaim all responsibility with terms of service. Most of these problems would become far less common if the situation was more balanced and managers had to consider the possibility of serious financial consequences for rushing something out without adequate testing.
- giantg2 2y ago
  I mostly agree, but that still won't fix this issue. Nothing has 100% uptime. Let's say Crowdstrike has an SLA of 99.99%. Even with this outage, they probably meet that depeding on the time frame. So which mangers bear the blame here? The ones who likely met their SLAs, or the ones who knew their systems relied on software that wasn't 100% reliable and didn't have a backup plan?
  - acdha 2y ago
    Pure SLAs matter less than the negligence: this showed critical lack of testing, following robust engineering practices for progressive deployments or fail safe design, etc. If a bank loses $1M they don’t get to argue that they shouldn’t compensate clients because they processed 99.9999% of their transactions correctly.
    - giantg2 2y ago
      In that example, the banks wouldn't compensate people if their payment system was down and couldn't process orders. That really what happened here - loss of sales on the clients' end. If it were securities, then they would have as-of processing to fix it. But the same argument about lack of testing can be made for the companies using the system. Do they have tests for what happens when pieces of their systems or infrastructure go down? Did they ever do a disaster recovery drill? Honestly, there's blame all around because almost nobody is doing it right. Even the ones who are have failures.
      - acdha 2y ago
        > Do they have tests for what happens when pieces of their systems or infrastructure go down? Did they ever do a disaster recovery drill? Yes, but this forced them to run the worst DR routine short of Microsoft going rogue. The scale of the testing problem is orders of magnitude larger on one side: people trusted them to be minimally competent and they just weren’t.
        giantg2 2y ago
        Not going to lie, you sound like you've never made any mistakes ever. The problematic culture is on both sides. Anyone who works in infrastructure, SCCM packaging, or data centers knows you roll out updates to a isolated set of machines to test it first. Trust no one. If they really relied on them that much, they should have penalties in their contracts. If not, that would be an example of not being minimally competent - any enterprise sourcing team would look into this.
        acdha 2y ago
        Nobody does much in ops without making mistakes. I’ve done all of the roles you mentioned and learned from mistakes and oversights, which is why I know that you should start by looking at what you assume is available in each scenario. For example, I’ve twice seen prolonged outages in a data center due to failures in the power distribution equipment when doing a failover test – the physical plant guys had checked the UPS/battery systems but didn’t think about what would happen if that component failed and then learned that spare parts were out of stock in Southern California and the manufacturer had to put someone on the plane from Colorado, which meant we had to roll the failover to another data center. All of us had technically known that the redundant hardware didn’t mean n=2 parts could fail simultaneously but had incorrectly assumed the odds of a correlated failure were much lower or that the vendor a couple miles away would be able to fix a failed unit. I mention that last because that’s what happened to a lot of people here. They had DR plans assuming that they had their management infrastructure or could quickly bring it back online, but then they had things like CrowdStrike taking out the servers holding BitLocker recovery keys and other critical infrastructure. One of the under-appreciated outcomes from the general push to secure things has been that a lot of systems are now less robust because they depend on a few security critical components with no easy path to recovery if those fail. Full disk encryption is great from the perspective of data loss but it also means key management is mission critical in a way senior management probably wasn’t fully appreciative of when setting funding plans.

acdha 2y ago

I blame the courts largely buying the argument that companies can disclaim all responsibility with terms of service. Most of these problems would become far less common if the situation was more balanced and managers had to consider the possibility of serious financial consequences for rushing something out without adequate testing.

giantg2 2y ago
I mostly agree, but that still won't fix this issue. Nothing has 100% uptime. Let's say Crowdstrike has an SLA of 99.99%. Even with this outage, they probably meet that depeding on the time frame. So which mangers bear the blame here? The ones who likely met their SLAs, or the ones who knew their systems relied on software that wasn't 100% reliable and didn't have a backup plan?
- acdha 2y ago
  Pure SLAs matter less than the negligence: this showed critical lack of testing, following robust engineering practices for progressive deployments or fail safe design, etc. If a bank loses $1M they don’t get to argue that they shouldn’t compensate clients because they processed 99.9999% of their transactions correctly.
  - giantg2 2y ago
    In that example, the banks wouldn't compensate people if their payment system was down and couldn't process orders. That really what happened here - loss of sales on the clients' end. If it were securities, then they would have as-of processing to fix it. But the same argument about lack of testing can be made for the companies using the system. Do they have tests for what happens when pieces of their systems or infrastructure go down? Did they ever do a disaster recovery drill? Honestly, there's blame all around because almost nobody is doing it right. Even the ones who are have failures.
    - acdha 2y ago
      > Do they have tests for what happens when pieces of their systems or infrastructure go down? Did they ever do a disaster recovery drill? Yes, but this forced them to run the worst DR routine short of Microsoft going rogue. The scale of the testing problem is orders of magnitude larger on one side: people trusted them to be minimally competent and they just weren’t.
      - giantg2 2y ago
        Not going to lie, you sound like you've never made any mistakes ever. The problematic culture is on both sides. Anyone who works in infrastructure, SCCM packaging, or data centers knows you roll out updates to a isolated set of machines to test it first. Trust no one. If they really relied on them that much, they should have penalties in their contracts. If not, that would be an example of not being minimally competent - any enterprise sourcing team would look into this.
        acdha 2y ago
        Nobody does much in ops without making mistakes. I’ve done all of the roles you mentioned and learned from mistakes and oversights, which is why I know that you should start by looking at what you assume is available in each scenario. For example, I’ve twice seen prolonged outages in a data center due to failures in the power distribution equipment when doing a failover test – the physical plant guys had checked the UPS/battery systems but didn’t think about what would happen if that component failed and then learned that spare parts were out of stock in Southern California and the manufacturer had to put someone on the plane from Colorado, which meant we had to roll the failover to another data center. All of us had technically known that the redundant hardware didn’t mean n=2 parts could fail simultaneously but had incorrectly assumed the odds of a correlated failure were much lower or that the vendor a couple miles away would be able to fix a failed unit. I mention that last because that’s what happened to a lot of people here. They had DR plans assuming that they had their management infrastructure or could quickly bring it back online, but then they had things like CrowdStrike taking out the servers holding BitLocker recovery keys and other critical infrastructure. One of the under-appreciated outcomes from the general push to secure things has been that a lot of systems are now less robust because they depend on a few security critical components with no easy path to recovery if those fail. Full disk encryption is great from the perspective of data loss but it also means key management is mission critical in a way senior management probably wasn’t fully appreciative of when setting funding plans.