9 ms·

On Robots Killing People

77 points by zdw 3y ago

gailees 3y ago
If anything this piece convinced me that robots aren’t actually that dangerous…
- baz00 3y ago
  I don't worry about robots. I worry about arrogant billionaires.
  - moffkalast 3y ago
    The most likely way people will get killed by robots is by taking their job, forcing them into poverty and eventually letting them die on the street.
    - pixl97 3y ago
      War kills millions, starvation kills hundreds of millions.
    - tivert 3y ago
      > The most likely way people will get killed by robots is by taking their job, forcing them into poverty and eventually letting them die on the street. Serves 'em right for not being able to provide value to capital. /s
SmoothBrain12 3y ago
Good
gcanyon 3y ago
> the robot simply determined that “the most efficient way to eliminate the threat was to push the worker into an adjacent machine.” This seems inflammatory and highly misleading. I could be wrong, but I highly doubt it: the robot didn't "determine" anything. It had assembly instructions to follow, and it followed them. The worker got in the way and was struck. Schneier falls into this same error, saying Tesla Autopilot sometimes "ignor[es] red lights, stop signs, and crosswalks." Tesla isn't perfect, but I'm pretty confident there's no code in Autopilot that says, "yeah, I see that crosswalk, but Elon's in a hurry, so let's goooo!" -- okay, maybe in Elon mode. But not in general. So no, Autopilot doesn't "ignore" those items: it misses them. Assigning intention to robots at this point is pretty much equivalent to assigning intention to a clock because the hands go around. We have a long way to go before we can/should assign intentionality to "robots" this way. Of course, at the rate things are going, "a long way" could happen in the next few years. But it's not happening now. I'll happily consider counter-examples.
- ekanes 3y ago
  Absolutely, but you're missing the most inflammatory part of all! The robot had to "eliminate the threat" -- when this was about a human who got in the way.
- dfxm12 3y ago
  What's the meaningful distinction between the robot ignoring or missing relevant information? We're not in a court of law trying to suss out intent or lay blame on a specific entity. The essay is about robots killing people, and supposing this will continue without government regulation. The people will be killed either way...
  - NikolaNovak 3y ago
    We can't address and fix a problem if we fundamentally misunderstand it.
    - dfxm12 3y ago
      The irony is that we're in this problem in the first place because people are releasing problematic tech before it is understood in the first place. Humanity at large shouldn't be part of the experimentation process, and the essay is arguing that we need tighter regulations around that. The essay suggests "Any regulation of industrial robots stems from existing industrial regulation, which has been evolving for many decades" & that regulations need to be clear about other robots as well. Interestingly enough, if we applied existing regulations to Tesla's Autopilot as we would to a driver who racked up as many fatalities and crashes (736 crashes and 17 deaths in the U.S. since 2019 [0]), they'd be behind bars long ago, let alone never able to drive again. 0 - https://www.caranddriver.com/news/a44185487/report-tesla-autopilot-crashes-since-2019/ https://www.caranddriver.com/news/a44185487/report-tesla-aut...
- nonrandomstring 3y ago
  Intentionality is a central concept in AI philosophy - from debates between Dennet, Churchland and Searle going back to the 1980s. Aat this stage I agree that we are dealing with "blind machines", with no more "intention" than a steam engine. However, the central argument in Schneier's piece seems to be that robots inherit the intentionality of their creators. If I create a machine that kills someone, my "intentionality" is fair game to be dissected in court, and ultimately I may be found guilty as if I had killed with my own hands. That is the kind of high bar I think society expects, and is certainly one that should scare the crap out of robotics developers (aside: I have been one) to ensure the highest standards of engineering discipline and industrial ethics.
ekanes 3y ago
Interesting that the article focuses on the unintentional killing of people by robots. That is unfortunate and worth writing about, but society is good at proportional reaction to things like that. The more disasters there are, the more we'll take it seriously, and the more lawsuits & regulation that will follow. See: self-driving cars. The more challenging topic is military/government use of robots in war or citizen control scenarios. Areas like that seem very relevant if you look forward a few decades given the pace of progress of Boston Robotics and AI. See: Terminator, the movie. Time travel aside... :P
- ReptileMan 3y ago
  >The more challenging topic is military/government use of robots in war or citizen control scenarios. Absolutely not challenging. They will come and will be used. Nothing you can do about it unless you DRM everything. And then of course the government will still have them. Topic closed. >Interesting that the article focuses on the unintentional killing of people by robots I doubt that there was ever killing. The first two examples are people trying to do maintenance on plugged machines. If you try to fix moving steamroller by standing in front of the wheel - don't be surprised at the consequences. The south africa one - it depends - if the turret just misfired in this general direction - is no different than a mine or shell exploding. If it scanned the soldiers, recognized them as targets and then shoot - then it is killing.
falcor84 3y ago
I absolutely agree with the comment from "Winter" at the bottom of the post: > The examples are more about Machines killing people than Robots killing people... The watershed will be when robots are build to kill people and then actually do kill people the intended way, but not necessarily the intended targets.
- ajuc 3y ago
  Mines are robots that kill not necessarily the intended people the intended way.
  - pixl97 3y ago
    A mine is a machine, not a robot. At least the more simple mechanical mines. The distinction for robots in general is some kind of automated intelligence where it is actively selective of its targets. Some modern missiles and drones would be far better examples.
    - ajuc 3y ago
      Modern mines have several kinds of sensors (pressure, accelerometer, magnetic, timer) and only activate when there's a pre-programmed combination of events (to prevent easy de-mining). They can often also be detonated or disabled remotely.
- krisoft 3y ago
  Watershed event and "when"? There are multiple documented instances of the exact thing happening: https://en.wikipedia.org/wiki/Civilian_casualties_from_U.S._drone_strikes https://en.wikipedia.org/wiki/Civilian_casualties_from_U.S._...
  - muststopmyths 3y ago
    I think the difference is that these incidents are operator decisions. I presume the GP is talking about the case when the robot just goes "Nah, I'm going to ignore your target and kill these other people over here".
Physkal 3y ago
41 deaths seems to be on the low end to me. Also sounds like he's blaming the tool, instead of the operator of the tool. I really expected that number to be higher when you account for: Have all the operators had proper training to even work on this machinery? We'lre all the safety features built into the machine from the OEM being utilized? Alot of the additional safety features are disabled to make it easier to run. Also no training is needed to have someone press the run button. We're there auxillary safeties installed? (Light curtains, safety scanners with properly set warning and red zones). A lot of times machine operators have no control over the work environment or equipment integration and are told to just get the job done with what they have.
photochemsyn 3y ago
As far as I know, the closest thing currently being developed that fits the sci-fi model of 'killer robot' is an autonomous airborne drone programmed to fly to a particular set of GPS coordinates and then kill every human it can recognize with its built-in AI-powered target-recognition system within 1000 meters of those GPS coordinates. This fits into the 'fire and forget' category of munitions - and rumors are that something like this has already been used in Libya. Note this kind of system isn't really subject to jamming as it's not being controlled by a human operator. See: https://thebulletin.org/2021/05/was-a-flying-killer-robot-used-in-libya-quite-possibly/ https://thebulletin.org/2021/05/was-a-flying-killer-robot-us... As far as the effort to regulate LLMs, it seems more about the desire to keep them from generating 'bad ideas' or teaching people how to do bad things (which they can already learn how to do in any library or on the internet already, incidentally). Unaligned LLMs are not likely to go around killing people, although they could be very offensive.
AndrewKemendo 3y ago
Let’s not mince words Military organizations globally are now working at breakneck speed to create *literal* “terminators.” I know cause I make them. Replicator is skynet v2…we tried it and weren’t ready with MDC2 in the teens ABMS etc are the follow on attempts. Every “unmanned” “cooperative” etc… system are the terminators As Bertrand Russell said : “until there is no more war, all technologies will be used for war”
- bell-cot 3y ago
  True. But, beyond mashing people's emotional buttons, how exactly do realistic terminator robots differ, morally, from many of the robot-free (but lethal) military technologies which were widely used a century-ish ago? - mass bombing of heavily populated areas, artillery barrages, minefields, etc.? Germany alone produced almost 2 million of their S-mines for WWII.
  - AndrewKemendo 3y ago
    It’s not, we already have the problem with mines
- c0pium 3y ago
  ABMS is sensor fusion, relax. You’re not that important. Don’t worry, the pentagon is totally incapable of breakneck speed.
  - AndrewKemendo 3y ago
    ABMS is dead so not even really relevant anymore
- WeylandYutani 3y ago
  If we can send robots to clean up shit hole countries so that decent people don't have to develop PTSD I'm all for it.
  - morkalork 3y ago
deleted 3y ago
[deleted]
JoeAltmaier 3y ago
With autonomous public robots, it becomes fair to question their programming. For instance I was derided on this venue for unreasonably introducing the 'trolley car' philosophical puzzle unnecessarily into the mix. But already programmers have intentionally written code to react when facing this dilemma. We just don't know what code they put in there. Example: a truck stops in front of the autodriving car, too quickly for it to stop in time. It knows this; it understands its vehicle handling to many decimal places. So what should it do? Continue futilely braking to a likely-fatal accident when it impacts the truck? Or swerve to avoid? Swerving may involves other people now: people standing by the side of the road, or emergency vehicles and personnel at work. Even the people in cars behind, auto-driving or not they are part of this. We hit the truck; they hit us. Or we swerve; they hit the truck. You see? Kill my passenger, or potentially more by saving my passenger. It's not simple, and not obvious (thus the longevity of the philosophical puzzle). But now robots are making the decision. We can make laws to force the programmers' hands. Then it's no longer the intentionality of the programmer we have to examine. It's the rules and regulations we put in place, the intentionality of the regulator, of the standards committee. Ultimately of voters. My humble opinion is, autodriving cars should never leave the roadway. See, the people on the road have voluntarily put themselves 'in the game'. Kids playing by the side of the road are not part of this; they should under no circumstances ever be involved if possible. Anyway, the trolley car problem. It sucks. It's here.
- ReptileMan 3y ago
  >Example: a truck stops in front of the autodriving car, too quickly for it to stop in time. That is physically extremely rare - trucks have lower deceleration and higher breaking distance than high performance passenger cars. Also why TF is not the autodriving car keeping distance allowing it to break in time. The focus should be on not getting into shitty situations - and because AI attention is unlimited, we can do all kinds of sensors and also mesh the cards, couldn't be bored and is vigilant 24/7 it should be theoretically possible.
  - dirtyv 3y ago
    The specific example is beside the point. There are any number of events that can happen outside of a driver's control, human or computer, that could cause this or a similar predicament, e.g. the truck could have just changed lanes abruptly without signaling
LinuxBender 3y ago
In my opinion robots are the up-and-coming problem that will exacerbate existing problems that need to be addressed first. I would like to see his opinion on the existing problem of drive-by-wire vessels or at the risk of sounding like hyperbole what I call Internet-Connected-Death-Machines or ICDM's. Not just automobiles but also aircraft, trains and sea vessels. Internet connected vehicles plus control of steering, acceleration and braking via the vessels computer is already an existing problem. Some researchers got into trouble some time ago for taking over remote control of a vehicle on a live highway. There are also videos of people mysteriously accelerating and driving into buildings whilst the driver is trying and failing to stop the vehicle. I'm not linking the ring videos of Anne Heche but those can not be unseen. Before we can secure robots and prevent them from taking people out I think we first have to secure their predecessors of which they are based on. If a trusted partner or random skiddie can execute malicious undocumented codes or add their own malicious code then the game was over before it started in my cranky opinion. Cars and their remote networks get released and patched just like existing internet services and have a long lag time in between patches giving malevolent parties plenty of time to erase drivers. Also missing from these automobiles is a truly-tamper-and-fire-proof "little black box" that records every instruction executed by the car for the last 72 hours, including the undocumented or hidden instructions.
eggy 3y ago
I grew up riding mechanical, carburetor, dumb motorcycles, and every mishap was due to my youth, bad environment, judgement, what have you. However, I now ride a motorcycle with a 6-axis accelerometer, inclinometer, ABS braking, fuel-mapped ignition, and all controlled by the onboard ECU. I can say it has been a disruptive experience. I can disable things when I want to do stupid stuff - burnouts, wheelies, etc. - or leave them on so they can save my tired ass in the rain or a tight turn I've entered too fast. I have felt the cornering ABS kick in once. The throttle by wire is ultra responsive and programmable depending on the mode you are in, and not like the mechanical throttle of my old bikes. The fuel mapping is intelligent and changes under different temperatures and demands. I'd say it has saved rather than killed me. But, I must say, I talk to my moto, Ginger, and thank her for taking care of me on every trip. It makes me feel better ;)
- ubermonkey 3y ago
  As a fellow motorcyclist, it falls to me to ask "what bike?"
  - eggy 3y ago
    2021 KTM Duke 890R. Decat with dual-titanium Arrow race exhausts, Power Commander, Rottweiler race air intake. Replaced Michelin PC2s with Michelin P5's..Love it! And you?
    - ubermonkey 3y ago
      Those are VERY cool and VERY powerful! As for me, I waited until I was sure I wouldn't kill myself to buy a motorcycle, which for me meant "last year, at age 52." Sure, I still gotta worry about everyone else, but that's a risk I've been taking on a bicycle for years already. Mine is a Royal-Enfield Classic 350, which is perfect for 85% of what I'd want to do -- errands in town, etc -- but is obviously limited to surface streets and slower back roads. It tops out at 70, notionally, but it's not super happy above 55(mph). We'd dropped to one car during the pandemic, because my wife has traditionally taken the bus to work. I'd have the car during the day for the odd daytime errand. Then she got a better job that required driving to work, and we really really didn't want to buy another car - and I had always wanted a motorcycle... She wasn't super excited, but the price tag helped a lot ($4800US). It's lovely, and I get comments on it constantly, but after a year I'm definitely starting to think about Next Bike. Top contender is a Bonneville T120.
WaitWaitWha 3y ago
Schneier is asking for an dedicated, FAA-like Robot/AI governmental regulatory agency. I do not believe we need one. Farm implements have been killing us for far longer, and in far larger numbers. (approx. 20 deaths per 100K, out of 2.1mm in agro in 2020. [1]) Yet, we do not have a "tractor overturn death prevention agency (TODPA)" There is already ROPS 1928.51. In the USA, re-evaluating the OSHA requirements would reduce the number of at-work robot accidents. [1] https://www.cdc.gov/niosh/topics/aginjury/default.html https://www.cdc.gov/niosh/topics/aginjury/default.html caveat emptor lots of lumped in deaths
- bell-cot 3y ago
  THIS. But add in all the accidents with non-"robot" industrial machinery, and mine accidents, and people killed by livestock, and etc. The problem is millennia old. Yes, the regulations do need to be savvy about the more-complex ways that robots and AI's can go wrong. But that is also true of industrial accidents at a large chemical works. If Tank 372 is full, and Valve 8601b has a faulty sensor, and ... and ... and BOOM.
- WaitWaitWha 3y ago
  Oh, all the down votes because the bad Agency name. How about "Safety Alliance for Farm Equipment Turnover Risk Elimination and Awareness Division (SAFE-TREAD)"? :D
cushychicken 3y ago
I think Bruce has the right perspective, but I also can’t help but wonder about whether calls to slow the pace of AI research are loud warnings about the barn door being open long after the horse has left. For example: the Chinese have created a massive surveillance network of CCTVs and drones feeding data into the AI-augmented social credit score system. These are the AI technologies of theirs that we know about. I wouldn’t be surprised if they have lethal drones controlled by AI as a military project. We’ve seen time and time again that the Chinese Communist Party are perfectly happy subverting anything and everything in the service of upholding their own power. It’s a no-brainer to assume that AI ethics are just one more line item in the laundry list of the things they’d subvert. I wonder about the world’s capabilities to counteract such a ruthlessly amoral threat without a deep understanding of the technologies they use, and how to counteract them.
rob74 3y ago
> The 1905 Grover Shoe Factory disaster led to regulations governing the safe operation of steam boilers That seems awfully late?! Germany already had the "Society for the Supervision and Insurance of Steam Boilers" since 1866, and according to https://en.wikipedia.org/wiki/Technischer_%C3%9Cberwachungsverein https://en.wikipedia.org/wiki/Technischer_%C3%9Cberwachungsv..., "from 1871, membership in such an association exempted them from inspection by a state inspector" - so at that point, there was already a system of state inspection. And all this emulated a system that was already established in the UK, so the concept is even older. But maybe it's the American attitude to regulation that delayed similar regulations for so long in the US?
eduction 3y ago
There is simply not a culture of quality in software, and trying to graft one on to software controlling robots does not get to the root of the problem. To take just one example, software makers are uniquely able to shield themselves from liability via EULA. I can sell you software for millions of dollars, it can say “reliable, helps route 911 calls” on the tin, but it I make you click an EULA as everyone does you can’t sue me even for the most basic negligent defect in the software. Indeed, we tend not to even use words like “defect” when discussing software. We use words with more forgiving associations like “bug.” We do not have any widely used metric of software quality. Even hypothesizing such a measure feels a little absurd in todays climate. It’s basically still the Wild West in software. Even as the systems controlled by software become more critical and the those critical systems become more widely deployed.
- zer0tonin 3y ago
  I think some people attempted to get us out of this Wild West during the 20th century, with people like Margaret Hamilton trying to turn software engineering into "real" engineering. I don't really know why we seem to have reverted back to this stage nowadays, I suspect the boom/burst cycles of tech, especially 2001, might not have helped.
dgroshev 3y ago
It's interesting how public discourse has those blind spots, with people talking about autonomous machines killing people as if it's some danger looming on the horizon. They are already here and have been around for decades, we just call them mines (some of which attempt to filter out unintended targets), missiles (anti-ship missiles were collaboratively and selectively targeting ships since the 80s), and torpedoes. Anti-radiation loitering munitions are effectively autonomous killer drones, and Harpy first flew in 1989. Wider recognition of the fact would do good to the quality of the discourse. What exactly are we scared of, considering that we had decades of experience with these systems? How much truth is there to those fears? How much of it is shallow media driven emotion vs genuine risk?
loup-vaillant 3y ago
> From 1992 to 2017, workplace robots were responsible for 41 recorded deaths in the United States—and that’s likely an underestimate, especially when you consider knock-on effects from automation, such as job loss. That sentence right there makes me angry. Not at Schneier specifically, but at this blatant lie that somehow automation is responsible for even a fraction of job loss, unemployment, and the deaths that ensued. Wake the fuck up, people: it’s not robots, it’s capitalism. I was born in 1982. Many people my age still remember being promised that machines would lift the burden of work from us, that we’d all enjoy 3-day work weeks and so on. A huge amount of automation did happen, but the rest, not so much. There are many reasons for this, but a big one is that fewer and fewer people concentrated a bigger and bigger share of the profits. Now when we automate something, it’s not, as we were promised, to reduce the physical or mental burden of workers. It’s to reduce the costs associated with wages. The only way Schneier’s sentence makes sense is if we can’t change our economic system, but for some reason we can stop or limit automation. But even that isn’t going to work: capital owners are going to continue automating stuff as long as doing so increases their profits. (Edit: I guess that kind of criticism is not exactly welcome in a startup forum.)
swayvil 3y ago
Can you suss out the roots of your own intentions and motivations? It's a welling-up from mysterious depths and only the closer tip is "rational", if at all. And no, the machine does not access these depths (which is kind of the whole point of machines) and has no "intentions".
dav_Oz 3y ago
My expectation: It's about the history, present and future capabilities of "robots"/"automation processes" specifically designed to neutralize enemy targets; one of the optimization being, of course: to kill as many humans as possible. Alas, it's about the unintended dangers of ever sophisticated and fully automated tool use. Is this satire? Yeah, something heavy on wheels can roll you over, crush you and kill you. I'm not trying to dismiss any safety regulation but isn't the big elephant in the room one of the main driver of all technology: military use? Maybe most people gave up on that and try to focus on things in the civilian sphere that they can potentially impact. A "healthy" compartmentalization of sorts. But for me the contrast is so stark that the examples mentioned in the article are unintentionally comical and forced, even if the underlying questions valid and transferable to military use.
tivert 3y ago
> Large corporations have a tendency to develop computer technologies to self-servingly shift the burdens of their own shortcomings onto society at large, or to claim that safety regulations protecting society impose an unjust cost on corporations themselves, or that security baselines stifle innovation. You can say that again. Also the first part ("technologies to self-servingly shift the burdens ... onto society at large") is frequently the very innovation they're talking about in the last part.
robbywashere_ 3y ago
I’m starting to see the whole “robots killing people” is really just going to be the flagrant deflection of responsibility and blame by the people who deploy them. Just think, smart humanoid “intelligence” will make a wonderful scapegoat.
mcoliver 3y ago
Wired for War came out in 2009 and if you are new to this topic I would recommend giving it a read. It focuses on offensive military uses rather than civilian "accidents" discussed in this article. I do think there are crossover ethical dilemmas at play with both. https://en.wikipedia.org/wiki/Wired_for_War https://en.wikipedia.org/wiki/Wired_for_War

77 points by zdw 3y ago

gailees 3y ago

If anything this piece convinced me that robots aren’t actually that dangerous…

baz00 3y ago
I don't worry about robots. I worry about arrogant billionaires.
- moffkalast 3y ago
  The most likely way people will get killed by robots is by taking their job, forcing them into poverty and eventually letting them die on the street.
  - pixl97 3y ago
    War kills millions, starvation kills hundreds of millions.
  - tivert 3y ago
    > The most likely way people will get killed by robots is by taking their job, forcing them into poverty and eventually letting them die on the street. Serves 'em right for not being able to provide value to capital. /s

SmoothBrain12 3y ago

Good

gcanyon 3y ago

> the robot simply determined that “the most efficient way to eliminate the threat was to push the worker into an adjacent machine.” This seems inflammatory and highly misleading. I could be wrong, but I highly doubt it: the robot didn't "determine" anything. It had assembly instructions to follow, and it followed them. The worker got in the way and was struck. Schneier falls into this same error, saying Tesla Autopilot sometimes "ignor[es] red lights, stop signs, and crosswalks." Tesla isn't perfect, but I'm pretty confident there's no code in Autopilot that says, "yeah, I see that crosswalk, but Elon's in a hurry, so let's goooo!" -- okay, maybe in Elon mode. But not in general. So no, Autopilot doesn't "ignore" those items: it misses them. Assigning intention to robots at this point is pretty much equivalent to assigning intention to a clock because the hands go around. We have a long way to go before we can/should assign intentionality to "robots" this way. Of course, at the rate things are going, "a long way" could happen in the next few years. But it's not happening now. I'll happily consider counter-examples.

ekanes 3y ago
Absolutely, but you're missing the most inflammatory part of all! The robot had to "eliminate the threat" -- when this was about a human who got in the way.
dfxm12 3y ago
What's the meaningful distinction between the robot ignoring or missing relevant information? We're not in a court of law trying to suss out intent or lay blame on a specific entity. The essay is about robots killing people, and supposing this will continue without government regulation. The people will be killed either way...
- NikolaNovak 3y ago
  We can't address and fix a problem if we fundamentally misunderstand it.
  - dfxm12 3y ago
    The irony is that we're in this problem in the first place because people are releasing problematic tech before it is understood in the first place. Humanity at large shouldn't be part of the experimentation process, and the essay is arguing that we need tighter regulations around that. The essay suggests "Any regulation of industrial robots stems from existing industrial regulation, which has been evolving for many decades" & that regulations need to be clear about other robots as well. Interestingly enough, if we applied existing regulations to Tesla's Autopilot as we would to a driver who racked up as many fatalities and crashes (736 crashes and 17 deaths in the U.S. since 2019 [0]), they'd be behind bars long ago, let alone never able to drive again. 0 - https://www.caranddriver.com/news/a44185487/report-tesla-autopilot-crashes-since-2019/ https://www.caranddriver.com/news/a44185487/report-tesla-aut...
nonrandomstring 3y ago
Intentionality is a central concept in AI philosophy - from debates between Dennet, Churchland and Searle going back to the 1980s. Aat this stage I agree that we are dealing with "blind machines", with no more "intention" than a steam engine. However, the central argument in Schneier's piece seems to be that robots inherit the intentionality of their creators. If I create a machine that kills someone, my "intentionality" is fair game to be dissected in court, and ultimately I may be found guilty as if I had killed with my own hands. That is the kind of high bar I think society expects, and is certainly one that should scare the crap out of robotics developers (aside: I have been one) to ensure the highest standards of engineering discipline and industrial ethics.

ekanes 3y ago

Interesting that the article focuses on the unintentional killing of people by robots. That is unfortunate and worth writing about, but society is good at proportional reaction to things like that. The more disasters there are, the more we'll take it seriously, and the more lawsuits & regulation that will follow. See: self-driving cars. The more challenging topic is military/government use of robots in war or citizen control scenarios. Areas like that seem very relevant if you look forward a few decades given the pace of progress of Boston Robotics and AI. See: Terminator, the movie. Time travel aside... :P

ReptileMan 3y ago
>The more challenging topic is military/government use of robots in war or citizen control scenarios. Absolutely not challenging. They will come and will be used. Nothing you can do about it unless you DRM everything. And then of course the government will still have them. Topic closed. >Interesting that the article focuses on the unintentional killing of people by robots I doubt that there was ever killing. The first two examples are people trying to do maintenance on plugged machines. If you try to fix moving steamroller by standing in front of the wheel - don't be surprised at the consequences. The south africa one - it depends - if the turret just misfired in this general direction - is no different than a mine or shell exploding. If it scanned the soldiers, recognized them as targets and then shoot - then it is killing.

falcor84 3y ago

I absolutely agree with the comment from "Winter" at the bottom of the post: > The examples are more about Machines killing people than Robots killing people... The watershed will be when robots are build to kill people and then actually do kill people the intended way, but not necessarily the intended targets.

ajuc 3y ago
Mines are robots that kill not necessarily the intended people the intended way.
- pixl97 3y ago
  A mine is a machine, not a robot. At least the more simple mechanical mines. The distinction for robots in general is some kind of automated intelligence where it is actively selective of its targets. Some modern missiles and drones would be far better examples.
  - ajuc 3y ago
    Modern mines have several kinds of sensors (pressure, accelerometer, magnetic, timer) and only activate when there's a pre-programmed combination of events (to prevent easy de-mining). They can often also be detonated or disabled remotely.
krisoft 3y ago
Watershed event and "when"? There are multiple documented instances of the exact thing happening: https://en.wikipedia.org/wiki/Civilian_casualties_from_U.S._drone_strikes https://en.wikipedia.org/wiki/Civilian_casualties_from_U.S._...
- muststopmyths 3y ago
  I think the difference is that these incidents are operator decisions. I presume the GP is talking about the case when the robot just goes "Nah, I'm going to ignore your target and kill these other people over here".

Physkal 3y ago

41 deaths seems to be on the low end to me. Also sounds like he's blaming the tool, instead of the operator of the tool. I really expected that number to be higher when you account for: Have all the operators had proper training to even work on this machinery? We'lre all the safety features built into the machine from the OEM being utilized? Alot of the additional safety features are disabled to make it easier to run. Also no training is needed to have someone press the run button. We're there auxillary safeties installed? (Light curtains, safety scanners with properly set warning and red zones). A lot of times machine operators have no control over the work environment or equipment integration and are told to just get the job done with what they have.

photochemsyn 3y ago

As far as I know, the closest thing currently being developed that fits the sci-fi model of 'killer robot' is an autonomous airborne drone programmed to fly to a particular set of GPS coordinates and then kill every human it can recognize with its built-in AI-powered target-recognition system within 1000 meters of those GPS coordinates. This fits into the 'fire and forget' category of munitions - and rumors are that something like this has already been used in Libya. Note this kind of system isn't really subject to jamming as it's not being controlled by a human operator. See: https://thebulletin.org/2021/05/was-a-flying-killer-robot-used-in-libya-quite-possibly/ https://thebulletin.org/2021/05/was-a-flying-killer-robot-us... As far as the effort to regulate LLMs, it seems more about the desire to keep them from generating 'bad ideas' or teaching people how to do bad things (which they can already learn how to do in any library or on the internet already, incidentally). Unaligned LLMs are not likely to go around killing people, although they could be very offensive.

AndrewKemendo 3y ago

Let’s not mince words Military organizations globally are now working at breakneck speed to create *literal* “terminators.” I know cause I make them. Replicator is skynet v2…we tried it and weren’t ready with MDC2 in the teens ABMS etc are the follow on attempts. Every “unmanned” “cooperative” etc… system are the terminators As Bertrand Russell said : “until there is no more war, all technologies will be used for war”

bell-cot 3y ago
True. But, beyond mashing people's emotional buttons, how exactly do realistic terminator robots differ, morally, from many of the robot-free (but lethal) military technologies which were widely used a century-ish ago? - mass bombing of heavily populated areas, artillery barrages, minefields, etc.? Germany alone produced almost 2 million of their S-mines for WWII.
- AndrewKemendo 3y ago
  It’s not, we already have the problem with mines
c0pium 3y ago
ABMS is sensor fusion, relax. You’re not that important. Don’t worry, the pentagon is totally incapable of breakneck speed.
- AndrewKemendo 3y ago
  ABMS is dead so not even really relevant anymore
WeylandYutani 3y ago
If we can send robots to clean up shit hole countries so that decent people don't have to develop PTSD I'm all for it.
- morkalork 3y ago

deleted 3y ago

[deleted]

JoeAltmaier 3y ago

With autonomous public robots, it becomes fair to question their programming. For instance I was derided on this venue for unreasonably introducing the 'trolley car' philosophical puzzle unnecessarily into the mix. But already programmers have intentionally written code to react when facing this dilemma. We just don't know what code they put in there. Example: a truck stops in front of the autodriving car, too quickly for it to stop in time. It knows this; it understands its vehicle handling to many decimal places. So what should it do? Continue futilely braking to a likely-fatal accident when it impacts the truck? Or swerve to avoid? Swerving may involves other people now: people standing by the side of the road, or emergency vehicles and personnel at work. Even the people in cars behind, auto-driving or not they are part of this. We hit the truck; they hit us. Or we swerve; they hit the truck. You see? Kill my passenger, or potentially more by saving my passenger. It's not simple, and not obvious (thus the longevity of the philosophical puzzle). But now robots are making the decision. We can make laws to force the programmers' hands. Then it's no longer the intentionality of the programmer we have to examine. It's the rules and regulations we put in place, the intentionality of the regulator, of the standards committee. Ultimately of voters. My humble opinion is, autodriving cars should never leave the roadway. See, the people on the road have voluntarily put themselves 'in the game'. Kids playing by the side of the road are not part of this; they should under no circumstances ever be involved if possible. Anyway, the trolley car problem. It sucks. It's here.

ReptileMan 3y ago
>Example: a truck stops in front of the autodriving car, too quickly for it to stop in time. That is physically extremely rare - trucks have lower deceleration and higher breaking distance than high performance passenger cars. Also why TF is not the autodriving car keeping distance allowing it to break in time. The focus should be on not getting into shitty situations - and because AI attention is unlimited, we can do all kinds of sensors and also mesh the cards, couldn't be bored and is vigilant 24/7 it should be theoretically possible.
- dirtyv 3y ago
  The specific example is beside the point. There are any number of events that can happen outside of a driver's control, human or computer, that could cause this or a similar predicament, e.g. the truck could have just changed lanes abruptly without signaling

LinuxBender 3y ago

In my opinion robots are the up-and-coming problem that will exacerbate existing problems that need to be addressed first. I would like to see his opinion on the existing problem of drive-by-wire vessels or at the risk of sounding like hyperbole what I call Internet-Connected-Death-Machines or ICDM's. Not just automobiles but also aircraft, trains and sea vessels. Internet connected vehicles plus control of steering, acceleration and braking via the vessels computer is already an existing problem. Some researchers got into trouble some time ago for taking over remote control of a vehicle on a live highway. There are also videos of people mysteriously accelerating and driving into buildings whilst the driver is trying and failing to stop the vehicle. I'm not linking the ring videos of Anne Heche but those can not be unseen. Before we can secure robots and prevent them from taking people out I think we first have to secure their predecessors of which they are based on. If a trusted partner or random skiddie can execute malicious undocumented codes or add their own malicious code then the game was over before it started in my cranky opinion. Cars and their remote networks get released and patched just like existing internet services and have a long lag time in between patches giving malevolent parties plenty of time to erase drivers. Also missing from these automobiles is a truly-tamper-and-fire-proof "little black box" that records every instruction executed by the car for the last 72 hours, including the undocumented or hidden instructions.

eggy 3y ago

I grew up riding mechanical, carburetor, dumb motorcycles, and every mishap was due to my youth, bad environment, judgement, what have you. However, I now ride a motorcycle with a 6-axis accelerometer, inclinometer, ABS braking, fuel-mapped ignition, and all controlled by the onboard ECU. I can say it has been a disruptive experience. I can disable things when I want to do stupid stuff - burnouts, wheelies, etc. - or leave them on so they can save my tired ass in the rain or a tight turn I've entered too fast. I have felt the cornering ABS kick in once. The throttle by wire is ultra responsive and programmable depending on the mode you are in, and not like the mechanical throttle of my old bikes. The fuel mapping is intelligent and changes under different temperatures and demands. I'd say it has saved rather than killed me. But, I must say, I talk to my moto, Ginger, and thank her for taking care of me on every trip. It makes me feel better ;)

ubermonkey 3y ago
As a fellow motorcyclist, it falls to me to ask "what bike?"
- eggy 3y ago
  2021 KTM Duke 890R. Decat with dual-titanium Arrow race exhausts, Power Commander, Rottweiler race air intake. Replaced Michelin PC2s with Michelin P5's..Love it! And you?
  - ubermonkey 3y ago
    Those are VERY cool and VERY powerful! As for me, I waited until I was sure I wouldn't kill myself to buy a motorcycle, which for me meant "last year, at age 52." Sure, I still gotta worry about everyone else, but that's a risk I've been taking on a bicycle for years already. Mine is a Royal-Enfield Classic 350, which is perfect for 85% of what I'd want to do -- errands in town, etc -- but is obviously limited to surface streets and slower back roads. It tops out at 70, notionally, but it's not super happy above 55(mph). We'd dropped to one car during the pandemic, because my wife has traditionally taken the bus to work. I'd have the car during the day for the odd daytime errand. Then she got a better job that required driving to work, and we really really didn't want to buy another car - and I had always wanted a motorcycle... She wasn't super excited, but the price tag helped a lot ($4800US). It's lovely, and I get comments on it constantly, but after a year I'm definitely starting to think about Next Bike. Top contender is a Bonneville T120.

WaitWaitWha 3y ago

Schneier is asking for an dedicated, FAA-like Robot/AI governmental regulatory agency. I do not believe we need one. Farm implements have been killing us for far longer, and in far larger numbers. (approx. 20 deaths per 100K, out of 2.1mm in agro in 2020. [1]) Yet, we do not have a "tractor overturn death prevention agency (TODPA)" There is already ROPS 1928.51. In the USA, re-evaluating the OSHA requirements would reduce the number of at-work robot accidents. [1] https://www.cdc.gov/niosh/topics/aginjury/default.html https://www.cdc.gov/niosh/topics/aginjury/default.html caveat emptor lots of lumped in deaths

bell-cot 3y ago
THIS. But add in all the accidents with non-"robot" industrial machinery, and mine accidents, and people killed by livestock, and etc. The problem is millennia old. Yes, the regulations do need to be savvy about the more-complex ways that robots and AI's can go wrong. But that is also true of industrial accidents at a large chemical works. If Tank 372 is full, and Valve 8601b has a faulty sensor, and ... and ... and BOOM.
WaitWaitWha 3y ago
Oh, all the down votes because the bad Agency name. How about "Safety Alliance for Farm Equipment Turnover Risk Elimination and Awareness Division (SAFE-TREAD)"? :D

cushychicken 3y ago

I think Bruce has the right perspective, but I also can’t help but wonder about whether calls to slow the pace of AI research are loud warnings about the barn door being open long after the horse has left. For example: the Chinese have created a massive surveillance network of CCTVs and drones feeding data into the AI-augmented social credit score system. These are the AI technologies of theirs that we know about. I wouldn’t be surprised if they have lethal drones controlled by AI as a military project. We’ve seen time and time again that the Chinese Communist Party are perfectly happy subverting anything and everything in the service of upholding their own power. It’s a no-brainer to assume that AI ethics are just one more line item in the laundry list of the things they’d subvert. I wonder about the world’s capabilities to counteract such a ruthlessly amoral threat without a deep understanding of the technologies they use, and how to counteract them.

rob74 3y ago

> The 1905 Grover Shoe Factory disaster led to regulations governing the safe operation of steam boilers That seems awfully late?! Germany already had the "Society for the Supervision and Insurance of Steam Boilers" since 1866, and according to https://en.wikipedia.org/wiki/Technischer_%C3%9Cberwachungsverein https://en.wikipedia.org/wiki/Technischer_%C3%9Cberwachungsv..., "from 1871, membership in such an association exempted them from inspection by a state inspector" - so at that point, there was already a system of state inspection. And all this emulated a system that was already established in the UK, so the concept is even older. But maybe it's the American attitude to regulation that delayed similar regulations for so long in the US?

eduction 3y ago

There is simply not a culture of quality in software, and trying to graft one on to software controlling robots does not get to the root of the problem. To take just one example, software makers are uniquely able to shield themselves from liability via EULA. I can sell you software for millions of dollars, it can say “reliable, helps route 911 calls” on the tin, but it I make you click an EULA as everyone does you can’t sue me even for the most basic negligent defect in the software. Indeed, we tend not to even use words like “defect” when discussing software. We use words with more forgiving associations like “bug.” We do not have any widely used metric of software quality. Even hypothesizing such a measure feels a little absurd in todays climate. It’s basically still the Wild West in software. Even as the systems controlled by software become more critical and the those critical systems become more widely deployed.

zer0tonin 3y ago
I think some people attempted to get us out of this Wild West during the 20th century, with people like Margaret Hamilton trying to turn software engineering into "real" engineering. I don't really know why we seem to have reverted back to this stage nowadays, I suspect the boom/burst cycles of tech, especially 2001, might not have helped.

dgroshev 3y ago

It's interesting how public discourse has those blind spots, with people talking about autonomous machines killing people as if it's some danger looming on the horizon. They are already here and have been around for decades, we just call them mines (some of which attempt to filter out unintended targets), missiles (anti-ship missiles were collaboratively and selectively targeting ships since the 80s), and torpedoes. Anti-radiation loitering munitions are effectively autonomous killer drones, and Harpy first flew in 1989. Wider recognition of the fact would do good to the quality of the discourse. What exactly are we scared of, considering that we had decades of experience with these systems? How much truth is there to those fears? How much of it is shallow media driven emotion vs genuine risk?

loup-vaillant 3y ago

> From 1992 to 2017, workplace robots were responsible for 41 recorded deaths in the United States—and that’s likely an underestimate, especially when you consider knock-on effects from automation, such as job loss. That sentence right there makes me angry. Not at Schneier specifically, but at this blatant lie that somehow automation is responsible for even a fraction of job loss, unemployment, and the deaths that ensued. Wake the fuck up, people: it’s not robots, it’s capitalism. I was born in 1982. Many people my age still remember being promised that machines would lift the burden of work from us, that we’d all enjoy 3-day work weeks and so on. A huge amount of automation did happen, but the rest, not so much. There are many reasons for this, but a big one is that fewer and fewer people concentrated a bigger and bigger share of the profits. Now when we automate something, it’s not, as we were promised, to reduce the physical or mental burden of workers. It’s to reduce the costs associated with wages. The only way Schneier’s sentence makes sense is if we can’t change our economic system, but for some reason we can stop or limit automation. But even that isn’t going to work: capital owners are going to continue automating stuff as long as doing so increases their profits. (Edit: I guess that kind of criticism is not exactly welcome in a startup forum.)

swayvil 3y ago

Can you suss out the roots of your own intentions and motivations? It's a welling-up from mysterious depths and only the closer tip is "rational", if at all. And no, the machine does not access these depths (which is kind of the whole point of machines) and has no "intentions".

dav_Oz 3y ago

My expectation: It's about the history, present and future capabilities of "robots"/"automation processes" specifically designed to neutralize enemy targets; one of the optimization being, of course: to kill as many humans as possible. Alas, it's about the unintended dangers of ever sophisticated and fully automated tool use. Is this satire? Yeah, something heavy on wheels can roll you over, crush you and kill you. I'm not trying to dismiss any safety regulation but isn't the big elephant in the room one of the main driver of all technology: military use? Maybe most people gave up on that and try to focus on things in the civilian sphere that they can potentially impact. A "healthy" compartmentalization of sorts. But for me the contrast is so stark that the examples mentioned in the article are unintentionally comical and forced, even if the underlying questions valid and transferable to military use.

tivert 3y ago

> Large corporations have a tendency to develop computer technologies to self-servingly shift the burdens of their own shortcomings onto society at large, or to claim that safety regulations protecting society impose an unjust cost on corporations themselves, or that security baselines stifle innovation. You can say that again. Also the first part ("technologies to self-servingly shift the burdens ... onto society at large") is frequently the very innovation they're talking about in the last part.

robbywashere_ 3y ago

I’m starting to see the whole “robots killing people” is really just going to be the flagrant deflection of responsibility and blame by the people who deploy them. Just think, smart humanoid “intelligence” will make a wonderful scapegoat.

mcoliver 3y ago

Wired for War came out in 2009 and if you are new to this topic I would recommend giving it a read. It focuses on offensive military uses rather than civilian "accidents" discussed in this article. I do think there are crossover ethical dilemmas at play with both. https://en.wikipedia.org/wiki/Wired_for_War https://en.wikipedia.org/wiki/Wired_for_War