4 ms·

Author makes some good points about designing human computer interfaces, but has a very opinionated view of how AI can be used in systems engineering tooling wh

Author makes some good points about designing human computer interfaces, but has a very opinionated view of how AI can be used in systems engineering tooling which seems like it misses a lot of places where AI can be useful even without humans in the loop?

swiftcoder 1y ago
The scenarios in the article are all about mission-critical disaster recovery - we don't even trust the majority of our human colleagues with those scenarios! AI won't make inroads there without humans in the loop, until AI is 100% trustworthy.
- datadrivenangel 1y ago
  And the author assumes that these humans are going to be very rigorous, which is good for SRE teams, but even then not consistently.
  - agentultra 1y ago
    We don't need humans to be perfect to have reliable responses to critical situations. Systems are more important than individuals at that level. We understand people make mistakes and design systems and processes to compensate. The problem with unattended AI in these situations is precisely the lack of context, awareness, intuition, intention, and communication skills. If you want automation in your disaster recovery system you want something that fails reliably and immediately. Non-determinism is not part of a good plan. Maybe it will recover from the issue or maybe it will delete the production database and beg for forgiveness later isn't what you want to lean on. Humans have deleted databases before and will again, I'm sure. And we have backups in place if that happens. And if you don't then you should fix that. But we should also fix the part of the system that allows a human to accidentally delete a database. But an AI could do that too! No. It's not a person. It's an algorithm with lots of data that can do neat things but until we can make sure it does one particular thing deterministically there's no point in using it for critical systems. It's dangerous. You don't want a human operator coming into a fire and the AI system having already made the fire worse for you... and then having to respond to that mess on top of everything else.
    - lupire 1y ago
      What happens when you walk into a fire and you don't know what to do? Or can't do it quickly enough?
      - agentultra 1y ago
        Who is sending in untrained people to manage fires? Maybe that organization deserves what's coming to them. An extreme example: nuclear reactors. You don't want untrained people walking into a fire with the expectation that they can manage the situation. Less extreme example: financial systems. You don't want untrained people walking into a fire losing your customers' funds and expect them to manage the situation.
      - swiftcoder 1y ago
        We don't throw new-hires in the deep end of the on call rotation on their first day. We make sure they learn the systems, we provide them with runbooks, assign an experienced mentor for their first on call rotation, and have a clear escalation path if they are in over their heads or need additional resources.
- topaz0 1y ago
  Or it will, and disaster will ensue.
- tptacek 1y ago
  Right, so: having an agent go drop index segments from a search cluster to resolve a volume utilization problem is a bad idea, rather than just suggesting "these old index segments are using up 70% of the storage on this volume and your emergency search cluster outage would be resolved if you dropped them, here's how you'd do that". But there are plenty of active investigative steps you'd want to take in generating hypotheses for an outage. Weakly's piece strongly suggests AI tools not take these actions, but rather suggest them to operators. This is a waste of time, and time is the currency of incident resolution.

swiftcoder 1y ago

The scenarios in the article are all about mission-critical disaster recovery - we don't even trust the majority of our human colleagues with those scenarios! AI won't make inroads there without humans in the loop, until AI is 100% trustworthy.

datadrivenangel 1y ago
And the author assumes that these humans are going to be very rigorous, which is good for SRE teams, but even then not consistently.
- agentultra 1y ago
  We don't need humans to be perfect to have reliable responses to critical situations. Systems are more important than individuals at that level. We understand people make mistakes and design systems and processes to compensate. The problem with unattended AI in these situations is precisely the lack of context, awareness, intuition, intention, and communication skills. If you want automation in your disaster recovery system you want something that fails reliably and immediately. Non-determinism is not part of a good plan. Maybe it will recover from the issue or maybe it will delete the production database and beg for forgiveness later isn't what you want to lean on. Humans have deleted databases before and will again, I'm sure. And we have backups in place if that happens. And if you don't then you should fix that. But we should also fix the part of the system that allows a human to accidentally delete a database. But an AI could do that too! No. It's not a person. It's an algorithm with lots of data that can do neat things but until we can make sure it does one particular thing deterministically there's no point in using it for critical systems. It's dangerous. You don't want a human operator coming into a fire and the AI system having already made the fire worse for you... and then having to respond to that mess on top of everything else.
  - lupire 1y ago
    What happens when you walk into a fire and you don't know what to do? Or can't do it quickly enough?
    - agentultra 1y ago
      Who is sending in untrained people to manage fires? Maybe that organization deserves what's coming to them. An extreme example: nuclear reactors. You don't want untrained people walking into a fire with the expectation that they can manage the situation. Less extreme example: financial systems. You don't want untrained people walking into a fire losing your customers' funds and expect them to manage the situation.
    - swiftcoder 1y ago
      We don't throw new-hires in the deep end of the on call rotation on their first day. We make sure they learn the systems, we provide them with runbooks, assign an experienced mentor for their first on call rotation, and have a clear escalation path if they are in over their heads or need additional resources.
topaz0 1y ago
Or it will, and disaster will ensue.
tptacek 1y ago
Right, so: having an agent go drop index segments from a search cluster to resolve a volume utilization problem is a bad idea, rather than just suggesting "these old index segments are using up 70% of the storage on this volume and your emergency search cluster outage would be resolved if you dropped them, here's how you'd do that". But there are plenty of active investigative steps you'd want to take in generating hypotheses for an outage. Weakly's piece strongly suggests AI tools not take these actions, but rather suggest them to operators. This is a waste of time, and time is the currency of incident resolution.