13 ms·
This is literally an episode of Better Off Ted.[1] In it, the titular Ted is inadvertenly deleted from the company system when trying to correct a misspelling
by bonyt 8y ago
This is literally an episode of Better Off Ted.[1] In it, the titular Ted is inadvertenly deleted from the company system when trying to correct a misspelling of his last name. Eventually, he is forced to interview for his own job as the system had already put out an ad for his replacement. I think the most striking part of it, and of the true story from the post, is the human factor - the idea that the humans involved looked to the system as an authority and followed its orders blindly.
I wonder what other examples there are of people blindly following technology - people driving into lakes because their GPS told them to, etc. Plus, as our society gets more and more dependent on these systems, we may lose out on the flexibility that human mediators and problem solvers once gave us. The human tendency to defer to authority may never be as terrifying as when that authority is held by an uncaring machine with a couple bugs.
What was once satire has become too real.
[1]: http://betteroffted.wikia.com/wiki/Goodbye,_Mr._Chips http://betteroffted.wikia.com/wiki/Goodbye,_Mr._Chips
- doc_gunthrop 8y agoReminds me of Frank Herbert's ominous (fictional) prophecy: “Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.”
- rixed 8y agoSounds like what Brecht once wrote about the luddites, something along the lines of: "you appear to be dominated by machines only because there are men hiding behind those machines" (can't retrieve the actual quote now, was on a treatise on theater)
- greedo 8y agoTime for Mentats! The Butlerian Jihad seemed so unrealistic when I first read Dune, but now it seems prophetic.
- vvanders 8y agoThat show was incredibly on the mark, if Black Mirror was a comedy it would have been that show. The episode about the black engineer who isn't detected by the motion sensors is basically straight out of HP's webcam fiasco[1](although the show takes it to the logical and hilarious extreme). [1] http://newsblogs.chicagotribune.com/race/2009/12/hp-webcam-colorblindbut-not-in-a-good-way.html http://newsblogs.chicagotribune.com/race/2009/12/hp-webcam-c...
- bobbyi_settv 8y agoHere's that episode on Hulu for anyone who happens to have a subscription and want to check it out: https://www.hulu.com/watch/81765 https://www.hulu.com/watch/81765 In my opinion, BOT is one of the best comedy shows of the past decade and that episode is widely considered to be the best one.
- letsgetphysITal 8y agoThis has been a real problem with facial recognition. Faces used to train engines are typically Caucasian, so the ability to distinguish other ethnicities is quite sub-par. https://www.theatlantic.com/technology/archive/2016/04/the-underlying-bias-of-facial-recognition-systems/476991/ https://www.theatlantic.com/technology/archive/2016/04/the-u...
- pbhjpbhj 8y agoIsn't there a problem with shadow contrast which makes features more difficult to pick out on darker skinned faces? I vaguely recall an East Asian developer giving that reason for why he developed on Western data-sets.
- Rjevski 8y agoMost likely it’s an issue with contrast and overall darker skin colour. Can be solved by illuminating the face with IR though, which should work across all ethnicities.
- 8y ago
- sigil 8y agoSee also the 1985 film “Brazil,” where a fly stuck in a printer (a bug!) sends a SWAT team after one Mr. Buttle instead of one Mr. Tuttle.
- btschaegg 8y agoA classic! :) Mistake? *chuckles* We don't make mistakes! *crashing sound as an improvised plug for the new hole in the ground falls straight through* Bloody typical! They've gone back to metric without telling us!
- duxup 8y agoIn the book Catch 22 the character "Major Major Major" is made a ... Major, I belive due to an IBM computer error.
- ionwake 8y agoGood Catch
- duxup 8y agoThe best there is.
- robterrell 8y agoAnd he can’t get promoted to captain!
- erl 8y agoWe are already there - "Death by GPS" is a concept today: https://arstechnica.com/cars/2016/05/death-by-gps/?amp=1 https://arstechnica.com/cars/2016/05/death-by-gps/?amp=1
- bsenftner 8y agoA lot of biometric technologies are treated like authorities by "our authorities" to a ridiculous fault.
- teekert 8y agoSo true. Recently I was at the pharmacy, the doctor wrote a wrong prescription and my wife was in the car having a hefty migraine. They wouldn't give me the drug even though they could see she has been using it for years and the wrong prescription could be solved afterwards. I think such people will be the first to be completely replaced by robots, I for sure wouldn't notice the difference, in fact I expect a robot to be inhuman so it would be less frustrating.
- ReverseCold 8y agoOkay this example makes sense. No prescription == no drugs (especially for painkillers and other things that people abuse). If you think this is a bad thing, then it's an organizational problem and has nothing to do with computers.
- teekert 8y agoIf you are a human and you can see in the computer that the prescription is regularly updated and you can even call the doctor if you doubted it. You can even check the prescription afterwards and if my story doesn't check out, do something about it. The migraine was pretty bad, imo it was like not treating a broken leg because of a missing insurance card or something.
- jexah 8y agoYou're really blaming the wrong people. Blame the doctor for screwing it up, your wife for not checking the prescription, or the government for creating/enforcing the relevant laws that would have put not only the chemist but their entire franchise underground for "doing the right thing". Laws are laws, broskie, don't expect everybody to break them for you. If the migraine was really bad (bad enough to cause tangible damage) maybe you should sue the doctor for damages, or if it wasn't that bad, report him/her and go somewhere else next time.
- ajeet_dhaliwal 8y agoSo in effect he's right to suggest the pharmacist should just be replaced by a robot because you would have them follow the rules no matter what with all humanity stripped out. If there was more than one pharmacist available it wouldn't have killed him/her to go and take a look at the wife in the car.
- zaarn 8y agoThe problem isn't that people refer to a machine as authority but that the process has no abort button. When you design business processes like these, firing, hiring or managing people always keep in mind something can go wrong. Always include an abort button to either stop or roll back the process.
- bena 8y agoThere's a problem with the "human override" option as well. Make it too easy to override and then the system just doesn't matter anymore. It's basically the issue we're trying to solve, where to draw that line?
- zaarn 8y agoThe system is meant to automated a certain process, some business process among different people. It does not loose meaning because it can be aborted if stuff goes FUBAR.
- zby 8y agoI have noticed that people following machine orders is also used by corporations to force customers to give up on their rights. I have a minor dispute with one of our communication providers and it is a real nightmare - according to the law I have the right, but the system does not include it and people don't know what to do in this case. Some other people higher up just wait until I give up.
- Rjevski 8y agoDon’t pay the invoice until they either fix it or let it go (they can’t go to court as you’re in the right from a legal point of view). That’s what I always do with shitty utility providers - it’s funny how the “computer says no” excuse suddenly disappears when they start to be out of pocket.
- UweSchmidt 8y agoCareful, automatic processes can give you trouble, cancel your service or hurt your credit rating. I recommend complying with the process, mentally account for the extra costs when you sign up, stick to defaults and avoid complexity whenever possible. On the other hand nowadays companies do respond to complaints much better than a few decades ago if you file complaints through their websites. Taking it to social media may also result in quick and drastic measures.
- Rjevski 8y ago> cancel your service By the time I arrive to this stage, I’ve already got a replacement lined up and all it takes is to flick a switch. > hurt your credit rating From experience, the problem eventually gets resolved and they cancel any black marks on your credit report. But personally I never cared about mine anyway.
- vinceguidry 8y agoThe other thing about credit reports is that 1) companies not in the business of providing credit have an uphill battle to claim that you're indebted to them, and 2) you can challenge such things, and the law requires them to show evidence. Due process has favored debtors for some time now, but the information isn't published, leaving shady "credit cleanup" companies to charge hundreds to do what you can do in many cases simply by logging into a website.
- c12 8y ago"We were only following orders" suddenly feels even more scary when they come from a computer and not a human.
- lmm 8y agoIt shouldn't. Blindly following orders from humans has much the same results.
- c12 8y agoWhen the orders come from a human then there is someone who is ultimatly responsible, someone to put through the courts, someone to throw in jail. When the orders come from a computer. Will any of that happen, will there be someone responsible?
- wowtip 8y ago"It looks like the devs didn't do their job properly..."?
- yipbub 8y agoyeah, but punitive justice is the least consequential thing in those situations. The major reason you'd punish someone there is to stop someone from willfully repeating the scenario. There exists a path for learning and prevention in automated systems too. The danger I see is the stifling of potential human subjective decisions. That broad decisions will be made and carried out without a sanity/humanity check by humans. I think measures should be put in place for a guarantee of human intervention in the case of situational anomaly or system error as well as an emphasis on redress. Since automation scales and a human workforce doesn't, a guarantee of human intervention might not be practically possible. Anyone have any ideas on how this kind of thing might be mitigated?
- haskellandchill 8y agoI think it won't be mitigated, given my experience working as a data scientist. It will be ignored because it is profitable to ignore it. If we slowed down and figured out how to make machines mimic human empathy before we make them make decisions it could be mitigated, but we won't do that. Ask your boss about it.
- hugi 8y agoRecently we had a lot of tourists drive through the unfinished tunnel Vaðlaheiðargöng (Iceland) because Apple Maps already marked it as part of Route 1. So we had people drive a workers' road past multiple signs banning entrance, and then drive on gravel through a tunnel filled with heavy machinery and workers. Quite funny.
- bewo001 8y agoWell, Iceland has a lot of gravel roads so they might assume that this is normal..
- JdeBP 8y agoI recommend reading the RISKS Digest. It has been a forum for this stuff for many years. * http://catless.ncl.ac.uk/risks/ http://catless.ncl.ac.uk/risks/
- haskellandchill 8y agoPhenomenal, I'm late to the party. Glad there is party.
- throwaway43532 8y ago> The human tendency to defer to authority may never be as terrifying as when that authority is held by an uncaring machine with a couple bugs. To this observation I highly recommend E.M. Forster's The Machine Stops, first published in 1909(!). [0] http://www.ele.uri.edu/faculty/vetter/Other-stuff/The-Machine-Stops.pdf http://www.ele.uri.edu/faculty/vetter/Other-stuff/The-Machin...
- da_chicken 8y ago> people driving into lakes because their GPS told them to That one has actually happened: http://bgr.com/2016/05/17/car-gps-mapping-directions-lake/ http://bgr.com/2016/05/17/car-gps-mapping-directions-lake/ It's a little unfair, though. There was heavy rain, fog, and a second article [0] says it was also in the middle of the night. The photo there also does make it look easy to mistake as a road. [0]: http://torontosun.com/2016/05/13/woman-follows-gps-ends-up-in-ontario-lake/wcm/fddda6d6-6b6e-41c7-88e8-aecc501faaa5 http://torontosun.com/2016/05/13/woman-follows-gps-ends-up-i...
- at-fates-hands 8y agoDidn't this happen in an Office episode? I remember Dwight and Michael arguing in Michael's car over whether the GPS was right as it kept telling them to go right which was directly into a lake.
- checkdigit15 8y agoYep: "Dunder Mifflin Infinity" - Season 4 Episode 3/4
- kbenson 8y agoOh, yeah. From that picture, it took me a second to realize I wasn't looking at a wet road-top and instead water in a lake, which is very close to the same height as the surrounding road. I'm glad it wasn't me driving there at night and in the rain, I could see myself making the same very bad mistake.
- at-fates-hands 8y agoAlso very similar to the classic "forgotten employee" story which made the rounds in the early aughts: https://sites.google.com/site/forgottenemployee/ https://sites.google.com/site/forgottenemployee/
- pure-awesome 8y agoOh man! I just read that, and absolutely loved it. A pretty brilliant piece of writing. Reminds me of the Google Ultron IT guy story, from 2014 (I think? How do 4Chan/American dates work?) https://imgur.com/a/iJD8f https://imgur.com/a/iJD8f