5 ms·
The better the autopilot the worse the pilot
- kotaKat 4mo agoAmerican Airlines captain Warran VanderBurgh once called this phenomenon "the children of the magenta"[1] in his talk on automation dependency in the 90s. I wonder what we'd call the children today in hindsight and what line they're chasing now... [1] https://www.youtube.com/watch?v=5ESJH1NLMLs https://www.youtube.com/watch?v=5ESJH1NLMLs
- singpolyma3 4mo agoI don't think we believe that the automation makes the human operators safer, but that it is overall safer than the human operator alone.
- cyanydeez 4mo agoijterestingly, AI tools are improving code theoughput but also elevating catastrophic error rates. even if the average rate goes up tor net benefit, are organizations prepared for increased carastrophic failures?
- alfredoh07 4mo ago[flagged]
- singpolyma3 4mo agoFor sure I not think you can compare agentic coding to automation in this context
- cyanydeez 4mo agoSure you can; right now, planes fly 90% on automation. They fall back to pilots to correct issues, engine outs or touchdowns. Right now, that fallback condition is mostly reasonable: pilots are trained extensively and have thousands of hours of flight experience to manage it. But as that automation approaches 100%, that fall back condition crosses a line where it's more likely the pilot is both not well trained because of it and their instinct in an emergency is entirely untrustable. So too is what's happening with LLMs: they're writing code that the programmer is increasinging unaware of and the programmer is increasingly not capable of understanding because they dont have the experience of writing and navigating complex codes. So in the event of a permissions prompt, the fallback condition, you have the same race condition between increasing automation removing the knowledge generation of the operator. So there's obviously a rubicon where automation _has_ to be 100% because no operator can be a fallback.
- kqr 4mo agoWell, it's differently safer. It's impossible to make categorical statements of more or less safe with automation because it depends massively on the design.
- bumby 4mo agoI think part of the point of the article is that it also makes edge-cases more dangerous and catastrophic than if there was no autopilot at all. From the article: >The argument for automation is that it frees up cognitive bandwidth. Fewer routine decisions means more headroom to think carefully about the ones that matter. So if the expectation is that the human pilot is expected to pay attention to mitigate the dangerous edge cases "that matter", there is a contradiction: the tool that promises to free up the bandwidth for that attention creates a complacency that prevents that attention from being applied. In other words, it makes the normal situations safer but the abnormal situations more dangerous.
- adampunk 4mo agoLanding is an abnormal situation for an aircraft which we make SUBSTANTIALLY less dangerous through intense automation. Do you want to rip out automated landing systems?
- bumby 4mo agoIf you expect every aircraft to land, it seems to meet the very definition of "normal" operation. An abnormal landing would be something like trying to land with a broken elevator surface.
- adampunk 4mo agoAnd you contend that autopilot makes that situation more dangerous? Do you have any support for this?
- bumby 4mo agoAs the article already states, there is a well known phenomenon in aviation called automation-induced complacency. So, yes, if you automate landing to the extent that human pilots no longer pay attention to abnormal signals that indicate something is wrong, or no longer feel the need to train or stay vigilant, it can make things more dangerous. There is plenty of research on this, but here's the first that came up in a cursory search: https://ntrs.nasa.gov/api/citations/20020021642/downloads/20020021642.pdf https://ntrs.nasa.gov/api/citations/20020021642/downloads/20... A more recent example is the Boeing 737-Max where there was a focus on automating trim control. In that case, the automation made the system more complex, to the detriment of a pilot understanding and reacting to an abnormal operation. We should also be careful that we don't create a false dichotomy between "all automated or no automation", or an expectation that more automation is always better. The goal should be the right balance that increases reliability/safety.
- palmotea 4mo ago> I don't think we believe that the automation makes the human operators safer, but that it is overall safer than the human operator alone. If the automation is for the easy/routine stuff, then no. The automation doesn't work in exactly the most safety-critical situations, and then the human operator is thrust into fixing the situation without the full context.
- bluGill 4mo agoI don't know about flying, but in machining it is the easy/routine stuff that will take your hand off. New machinists rarely get the serious injuries that more experienced ones do (they get more minor injuries) - the experienced machinist gets complacent about how dangerous the machine is and stops thinking to keep their hands away until they lose it. Automation has saves a lot of limbs (and lives) because you don't need to get close to the moving parts anymore.
- deleted 4mo ago[deleted]
- supriyo-biswas 4mo agoAlso, ironies of automation[1], etc. [1] https://static1.squarespace.com/static/644321e78cd2dd37613af33e/t/6694873f71612132a84371c7/1721009983702/Ironies+of+Automation_Bainbridge_1983.pdf https://static1.squarespace.com/static/644321e78cd2dd37613af...
- analogpixel 4mo agoThey should get an auotpilot to fix their website to render on desktops and not just phones.
- julienreszka 4mo agoJust `commmand` + `+`
- jonstewart 4mo agoAm I missing something? Is TFA only 2-3 paragraphs of a generic metaphor, with no actual data/research from aviation (or other fields) to back up the core thesis?
- schmookeeg 4mo agoWell, don't discount the equal number of affiliate links to tryware.
- NiloCK 4mo ago> Automation doesn't make operators more careful. It makes them forget how to be. The more reliable the system, the less ready the human. The entire premise of a system is that it removes the need for careful attention. system: signal lights tell me whether or not I can pass through an intersection, so that I do not have to attend to potentially high speed traffic from a variety of directions. system: the side my knife blade sits on my arched guide fingers, so that I do not have to attend to the edge of the blade or the location of my fingers. etc etc.
- andai 4mo ago> system: signal lights tell me whether or not I can pass through an intersection, so that I do not have to attend to potentially high speed traffic from a variety of directions. You know I noticed this... I lived in a country where people obey traffic laws, and in a country where they very much don't. I witnessed many more traffic accidents in the country where people are used to relying on the traffic lights to tell them if it's safe or not. Whereas in the other country, everyone correctly assumes that the other drivers are completely insane, and so they stay vigilant.
- pjc50 4mo agoThere's some documented studies of removing all the street clutter and lines from residential area intersections forcing drivers to be more careful, especially around pedestrians, reducing overall accidents. But this does reduce throughput slightly.
- raincole 4mo agoExcept if you look at this map: https://en.wikipedia.org/wiki/List_of_countries_by_traffic-related_death_rate https://en.wikipedia.org/wiki/List_of_countries_by_traffic-r... People die on road more in countries that conventionally don't follow traffic laws.
- infecto 4mo agoNot sure why this would downvoted. Go to a less developed nation where traffic laws are not important and it’s one of those sense of false security ideas.
- swordlucky666 4mo ago[flagged]
- adampunk 4mo agoThe crew and the plane are a single system. It is meaningless to imagine pilot “skill” without the plane. Further, we killed so many pilots in the 20th century through impossible workloads which we have now automated, it’s almost cruel to be wistful for it. I know this is an analogy to AI, but I wish we would dispense with this idea that there’s some appropriate level of machinery which was reached just a hair before right now. There is no appropriate level of machinery, no point at which the nature of the system itself will unambiguously say “that’s enough.”
- brindleth 4mo agoYet another AI written slop article hitting the top of HN. Evidence: Look at the most recent article on this blog: https://julienreszka.com/blog/difficult-conversations-don-t-need-to-be-messy/ https://julienreszka.com/blog/difficult-conversations-don-t-... "Memory is reconstructive. When someone recalls events differently, it feels like gaslighting. It usually isn't. Document first, then negotiate." Bleugh
- raincole 4mo agoAnd that's the whole freaking point. If the humans have to obtain and maintain the exact same skills as before than what's the point of automation? Actually this could be a side channel to measure the efficiency of automation. If the human operators are just as good in every aspect as before, we know all the money put into automation is wasted.
- jonesai 4mo agoIn this case the only reason I'm a pilot is the existence of autopilot haha
- black6 4mo agoThis is the concern I have with all the driving aids: adaptive cruise control, automatic braking, lane departure assist, et cetera. It all leads to drivers paying less attention to the road and more attention to their @#$%&*! phones because "Muh car drives itself."
- bluGill 4mo agoHuman drivers have been day dreaming for decades. It is not hard to move your car down the road without paying attention - until you run right through a red light or worse into a car.
- 21asdffdsa12 4mo agoThe more the pilot flies, the worse the pilot- long distance vs short distance, can create the absurd problem that you forget how to land and start, the most important pilot tasks, while flying long haul.
- 21asdffdsa12 4mo agohttps://theaircurrent.com/aviation-safety/emirates-777-ek231-close-call/ https://theaircurrent.com/aviation-safety/emirates-777-ek231...
- tippa123 4mo agoFrom my manufacturing experience on both ends of the spectrum of highly automated versus manual, the best analogy that comes to mind is driving a manual/stick versus an automatic car. If you know how to drive a manual, you can adapt to an automatic very quickly. The reverse, of course, is not true. There are, of course, many benefits to automation such standardisation, measurability and the list goes on. Plus cuurrently we have this sweet spot where the workforce contains several generations who have experienced both very manual and highly automated processes. This dual experience is invaluable for investigation and continuous improvement. It makes me wonder what will happen when the workforce consists entirely of operators and engineers who simply press start most of the time.
- cucumber3732842 4mo ago>The reverse, of course, is not true. 5min to learn. A week of normal driving to get not bad. I wouldn't say it's hard.
- rconti 4mo agoYes, but the magnitude of the difference is clear.
- jplusequalt 4mo agoHow long did it take you to understand what good code/engineering designs looked like. How long to debug tricky bugs? It certainly takes more than a week!
- mjmsmith 4mo agoI drive a manual Mazda 3 because I like the greater feeling of connection to the machine, but also because I think it makes me pay more attention. When I rent an automatic, it feels like a (literal) blunt instrument in comparison. I understand the appeal of (semi-) self driving, but the whole idea still makes me nervous. I can't really imagine sitting in a car doing nothing for long stretches of time but still being alert and ready to take control at any random moment.
- NalNezumi 4mo agoThere's been many studies around the concept "how do you get expertise? In what field can you actually develop expertise?" such as Naturalistic Decision Making [1] [2]. The wisdom from there can be applied to the question of automation. If your task is for example, to pilot a preset route with stable condition and very low surprise, you will fall for the "getting too comfortable" trap and we tend to start to get lazy(or efficient) and offload the mental effort and skills atrophy. A common workaround to this is to have regular training(deliberate practice) that introduce the "tricky" situation to keep the skill up. Problem ofc arises when people don't keep this up. This can be seen in the diagnostic performance differences between junior and senior doctors, not always in the favor to the senior [3]. If you add a layer of automation but the insight gathered by working on that layer is great (and falls off) then deliberate practice start to become a requirement [1] https://commoncog.com/putting-mental-models-to-practice/ https://commoncog.com/putting-mental-models-to-practice/ [2] https://youtu.be/5eW6Eagr9XA?si=Y9exacaW-F4PDOKF https://youtu.be/5eW6Eagr9XA?si=Y9exacaW-F4PDOKF [3] https://pubmed.ncbi.nlm.nih.gov/26375267/ https://pubmed.ncbi.nlm.nih.gov/26375267/
- timcobb 4mo agoIt's interesting living through all this civilizational flailing. Remarkable. I read that when people invented writing it was very controversial because the facility of memory was the prized intellectual capability of the time. I never thought I'd see this kind of thing in person
- deleted 4mo ago[deleted]
- ninalanyon 4mo agoI'm not convinced that this is necessarily always a bad thing on balance. It seems to me that automation in safety critical situations might make things worse when they finally go wrong but also that the number of times that they go wrong at a lower level is so much reduced that the result is net positive. Also if you know that automation induced complacency is a thing then it must surely become a target for training, surveillance, and adaptation not mere hand wringing.
- scottbez1 4mo agoI think the big question here is how effective can you make training and monitoring across a widespread population in practice? In aviation, commercial pilots have very strict and extensive training and monitoring and as a result are generally able to utilize automation effectively while keeping up their manual skills. There are very rarely CFIT incidents in major commercial airlines. The opposite is true in general aviation (small private Cessnas, etc), where it’s extremely common for pilots to buy more plane than they can handle and then rely on automation to bridge their skill gap. CFIT is much more common in general aviation, along with incorrect actions in response to real system failures that should have been recoverable. Automation complacency regularly kills in general aviation. A key thing to notice is that automation isn’t outright prohibited in either commercial or general aviation, but there are distinct regulatory frameworks based on potential impact. We accept looser rules for general aviation because the failures are societally less severe and because the population is much larger so effective training and enforcement would be significantly harder. In commercial airliners where failures are catastrophic, we have much stricter policies and require training and testing regularly to avoid automation complacency. Will we start to see this practice in software? Probably, but only if/when the societal cost of NOT doing it becomes more clear. We regulated aviation because crashing planes are obviously bad. We license structural engineers because collapsing bridges are obviously bad. Will automation-induced software failures hit a similar tipping point?
- mschuster91 4mo agoIn Munich, the U-Bahn subway normally drives semi-automatically (i.e. the driver only opens and closes the doors, then hands over control to the computer), but every driver has to regularly drive without automation assistance ("Fahren nach ortsfesten Signalen") to practice.
- deleted 4mo ago[deleted]
- zzyzxd 4mo agoAutomation should always be the serialization of understanding of the system, which can execute on its own under normal conditions. When it breaks, your reaction should not be removing the automation and asking humans to get back to manual operation, that is the wrong approach. Instead, you should ask humans to leverage their understanding to step in, reason about the automation system, and fix it. This is how I have been doing SRE for the past decade. And the inability to practice this under-the-hood reasoning is the main reason why I don't trust any modern AI based automation systems. They are helpers for some manual work, but they can't be automation.
- eternityforest 4mo agoThere are two other solutions, make the automation so reliable that the human skill is not needed, or make the consequences of failure low enough that you can just accept failure. As far as I know, Places that actually care about safety apply all of them at once. They add layers of failsafes that don't rely on humans, they make the automation better and better, and for the most critical stuff, they continue training for things that will almost certainly not happen. And for the rare(depending on personality) cases where we care about the pilot's ability more than the result, just don't automate at all.