4 ms·
Just one of the risks of automation, and a good reminder why human monitoring is necessary. Having said that, we deployed a system that was mostly automated, w
by malbs 13y ago
Just one of the risks of automation, and a good reminder why human monitoring is necessary.
Having said that, we deployed a system that was mostly automated, with the human operator to oversee investments and if any out-of-the-ordinary transactions (based on experience) were taking place, to shut it down. She happily sat there approving the recommendations even though the recommendations were absolutely outside of anything we'd ever generated in the past, and bled accounts dry in one evening, so sometimes even with a human observing you're still boned.
- fiatmoney 13y agoYou should read the linked PDF - they had systems that were 100% dependent on human monitoring, that no one was checking, or where no one recognized anything unusual. If anything, their failures were due to massive lack of automation in deployment, testing, and monitoring.
- malbs 13y agoYeah ours was monitoring just rubber stamped it. Afterwards everyone remarked, everyone could tell they were bad just looking at what was in front of them, our theory was she was too busy watching the breaking bad final episode or something.
- 4hthth4 13y agoIt makes me think you shouldn't allow only a single person to monitor something like that...
- drill_sarge 13y agoYes, the more people monitoring the better. But in this situation someone has to be in charge and make a decision. And nobody wants to be that guy.
- comrade_ogilvy 13y agoI would argue that many more people monitoring can encourage the fear of making the wrong call. "Hey, someone smarter and more experienced than me should makes sense of this." "Hey, the smart new guy is supposed to be watching this. I will look more closely later." What you want is two or three people really in charge, where the individuals are empowered to say: "I am totally confused. If someone cannot explain what is going on to me so that I understand, I am starting shutdown procedures, immediately. Do YOU know exactly what is going on?"
- jrochkind1 13y agoYou know how if the first 100 times a dialog box comes up, the correct response is to click 'ok', then people start just clicking 'ok' on every dialog box, and then the 201st one comes up "Destroy everything? [cancel] [ok], and they click 'ok' too, and don't think anything of it?
- Dylan16807 13y agoSure, you click reflexively, but you should notice the text was bad either after clicking or within a couple more clicks. Letting the first few errors through is reasonable, but letting a wall of them through without noticing anything wrong, when reading them is your job, is inexcusable.
- jrochkind1 13y agoI don't know if it's excusable or not, but it may be incompatible with typical human cognition to expect someone to be able to do that. Maybe you have to figure out a way to test people for unusually high aptitude at looking at mind-numbingly dull repetitive things over and over again, but then still being able to notice the aberrant ones. And then only put people in that job with unusually high aptitude there. Or have people only do pretty short shifts at that task. I'm pretty confident that this person wasn't unusually negligent, if you have most anyone doing that job hour after hour day after day they will lose the ability to flag the aberant stuff.
- comrade_ogilvy 13y agoYours is the correct viewpoint: it is incompatible with human cognition. If an alert system is not perceived as highly reliable in directing positive action, then the humans involved will inevitably disable the alert system, either by pulling out a screwdriver or rewriting their mental rubrics to ignore the messages as noise. Knight Capital is just the finance version of Three Mile Island and Deepwater Horizon -- the means to mitigate or prevent disaster were on hand, but the people in charge just dithered by the kill switch because they were confused. Well, if the people in charge are confused, that is a reason to start the emergency procedures.
- malbs 13y agoOh the other amazing thing we used to have - a system that generated so many alerts that everyone just created outlook rules to automatically delete all the alerts. I mean really? I wondered if we should modify the system to only generate messages when there are actual exceptional cases? People looked at me like I was an idiot
- rmc 13y agoTotally agree. You should not have too much noise.
- moath 13y agoI'm not seeing anything here that makes me think they had any kind of automation at all! In my experience, if they had automation things would have consistently failed on all servers the deployment was executed on. That said, if you're going to fly the jet liner in full manual mode, you better make sure your co-pilot is watching the instruments.
- SideburnsOfDoom 13y agoThe parent post may be pointing out that the point of this software is to automate trading on the stock market. It's risky if your software testing, rollout, monitoring and rollback process is not sufficiently automated. This second kind of automation is the kind that you or I are most likely more familiar with. And was lacking.
- SideburnsOfDoom 13y ago> Just one of the risks of automation Automated deployments would have helped them. They made lots of mistakes, but IMHO not automating the deploy was likely the #1 mistake here.