3 ms·

I don't think we believe that the automation makes the human operators safer, but that it is overall safer than the human operator alone.

cyanydeez 4mo ago
ijterestingly, AI tools are improving code theoughput but also elevating catastrophic error rates. even if the average rate goes up tor net benefit, are organizations prepared for increased carastrophic failures?
- alfredoh07 4mo ago
  [flagged]
- singpolyma3 4mo ago
  For sure I not think you can compare agentic coding to automation in this context
  - cyanydeez 4mo ago
    Sure you can; right now, planes fly 90% on automation. They fall back to pilots to correct issues, engine outs or touchdowns. Right now, that fallback condition is mostly reasonable: pilots are trained extensively and have thousands of hours of flight experience to manage it. But as that automation approaches 100%, that fall back condition crosses a line where it's more likely the pilot is both not well trained because of it and their instinct in an emergency is entirely untrustable. So too is what's happening with LLMs: they're writing code that the programmer is increasinging unaware of and the programmer is increasingly not capable of understanding because they dont have the experience of writing and navigating complex codes. So in the event of a permissions prompt, the fallback condition, you have the same race condition between increasing automation removing the knowledge generation of the operator. So there's obviously a rubicon where automation _has_ to be 100% because no operator can be a fallback.
kqr 4mo ago
Well, it's differently safer. It's impossible to make categorical statements of more or less safe with automation because it depends massively on the design.
bumby 4mo ago
I think part of the point of the article is that it also makes edge-cases more dangerous and catastrophic than if there was no autopilot at all. From the article: >The argument for automation is that it frees up cognitive bandwidth. Fewer routine decisions means more headroom to think carefully about the ones that matter. So if the expectation is that the human pilot is expected to pay attention to mitigate the dangerous edge cases "that matter", there is a contradiction: the tool that promises to free up the bandwidth for that attention creates a complacency that prevents that attention from being applied. In other words, it makes the normal situations safer but the abnormal situations more dangerous.
- adampunk 4mo ago
  Landing is an abnormal situation for an aircraft which we make SUBSTANTIALLY less dangerous through intense automation. Do you want to rip out automated landing systems?
  - bumby 4mo ago
    If you expect every aircraft to land, it seems to meet the very definition of "normal" operation. An abnormal landing would be something like trying to land with a broken elevator surface.
    - adampunk 4mo ago
      And you contend that autopilot makes that situation more dangerous? Do you have any support for this?
      - bumby 4mo ago
        As the article already states, there is a well known phenomenon in aviation called automation-induced complacency. So, yes, if you automate landing to the extent that human pilots no longer pay attention to abnormal signals that indicate something is wrong, or no longer feel the need to train or stay vigilant, it can make things more dangerous. There is plenty of research on this, but here's the first that came up in a cursory search: https://ntrs.nasa.gov/api/citations/20020021642/downloads/20020021642.pdf https://ntrs.nasa.gov/api/citations/20020021642/downloads/20... A more recent example is the Boeing 737-Max where there was a focus on automating trim control. In that case, the automation made the system more complex, to the detriment of a pilot understanding and reacting to an abnormal operation. We should also be careful that we don't create a false dichotomy between "all automated or no automation", or an expectation that more automation is always better. The goal should be the right balance that increases reliability/safety.
palmotea 4mo ago
> I don't think we believe that the automation makes the human operators safer, but that it is overall safer than the human operator alone. If the automation is for the easy/routine stuff, then no. The automation doesn't work in exactly the most safety-critical situations, and then the human operator is thrust into fixing the situation without the full context.
- bluGill 4mo ago
  I don't know about flying, but in machining it is the easy/routine stuff that will take your hand off. New machinists rarely get the serious injuries that more experienced ones do (they get more minor injuries) - the experienced machinist gets complacent about how dangerous the machine is and stops thinking to keep their hands away until they lose it. Automation has saves a lot of limbs (and lives) because you don't need to get close to the moving parts anymore.

cyanydeez 4mo ago

ijterestingly, AI tools are improving code theoughput but also elevating catastrophic error rates. even if the average rate goes up tor net benefit, are organizations prepared for increased carastrophic failures?

alfredoh07 4mo ago
[flagged]
singpolyma3 4mo ago
For sure I not think you can compare agentic coding to automation in this context
- cyanydeez 4mo ago
  Sure you can; right now, planes fly 90% on automation. They fall back to pilots to correct issues, engine outs or touchdowns. Right now, that fallback condition is mostly reasonable: pilots are trained extensively and have thousands of hours of flight experience to manage it. But as that automation approaches 100%, that fall back condition crosses a line where it's more likely the pilot is both not well trained because of it and their instinct in an emergency is entirely untrustable. So too is what's happening with LLMs: they're writing code that the programmer is increasinging unaware of and the programmer is increasingly not capable of understanding because they dont have the experience of writing and navigating complex codes. So in the event of a permissions prompt, the fallback condition, you have the same race condition between increasing automation removing the knowledge generation of the operator. So there's obviously a rubicon where automation _has_ to be 100% because no operator can be a fallback.

kqr 4mo ago

Well, it's differently safer. It's impossible to make categorical statements of more or less safe with automation because it depends massively on the design.

bumby 4mo ago

I think part of the point of the article is that it also makes edge-cases more dangerous and catastrophic than if there was no autopilot at all. From the article: >The argument for automation is that it frees up cognitive bandwidth. Fewer routine decisions means more headroom to think carefully about the ones that matter. So if the expectation is that the human pilot is expected to pay attention to mitigate the dangerous edge cases "that matter", there is a contradiction: the tool that promises to free up the bandwidth for that attention creates a complacency that prevents that attention from being applied. In other words, it makes the normal situations safer but the abnormal situations more dangerous.

adampunk 4mo ago
Landing is an abnormal situation for an aircraft which we make SUBSTANTIALLY less dangerous through intense automation. Do you want to rip out automated landing systems?
- bumby 4mo ago
  If you expect every aircraft to land, it seems to meet the very definition of "normal" operation. An abnormal landing would be something like trying to land with a broken elevator surface.
  - adampunk 4mo ago
    And you contend that autopilot makes that situation more dangerous? Do you have any support for this?
    - bumby 4mo ago
      As the article already states, there is a well known phenomenon in aviation called automation-induced complacency. So, yes, if you automate landing to the extent that human pilots no longer pay attention to abnormal signals that indicate something is wrong, or no longer feel the need to train or stay vigilant, it can make things more dangerous. There is plenty of research on this, but here's the first that came up in a cursory search: https://ntrs.nasa.gov/api/citations/20020021642/downloads/20020021642.pdf https://ntrs.nasa.gov/api/citations/20020021642/downloads/20... A more recent example is the Boeing 737-Max where there was a focus on automating trim control. In that case, the automation made the system more complex, to the detriment of a pilot understanding and reacting to an abnormal operation. We should also be careful that we don't create a false dichotomy between "all automated or no automation", or an expectation that more automation is always better. The goal should be the right balance that increases reliability/safety.

palmotea 4mo ago

> I don't think we believe that the automation makes the human operators safer, but that it is overall safer than the human operator alone. If the automation is for the easy/routine stuff, then no. The automation doesn't work in exactly the most safety-critical situations, and then the human operator is thrust into fixing the situation without the full context.

bluGill 4mo ago
I don't know about flying, but in machining it is the easy/routine stuff that will take your hand off. New machinists rarely get the serious injuries that more experienced ones do (they get more minor injuries) - the experienced machinist gets complacent about how dangerous the machine is and stops thinking to keep their hands away until they lose it. Automation has saves a lot of limbs (and lives) because you don't need to get close to the moving parts anymore.