5 ms·

> the robot simply determined that “the most efficient way to eliminate the threat was to push the worker into an adjacent machine.” This seems inflammatory an

> the robot simply determined that “the most efficient way to eliminate the threat was to push the worker into an adjacent machine.” This seems inflammatory and highly misleading. I could be wrong, but I highly doubt it: the robot didn't "determine" anything. It had assembly instructions to follow, and it followed them. The worker got in the way and was struck. Schneier falls into this same error, saying Tesla Autopilot sometimes "ignor[es] red lights, stop signs, and crosswalks." Tesla isn't perfect, but I'm pretty confident there's no code in Autopilot that says, "yeah, I see that crosswalk, but Elon's in a hurry, so let's goooo!" -- okay, maybe in Elon mode. But not in general. So no, Autopilot doesn't "ignore" those items: it misses them. Assigning intention to robots at this point is pretty much equivalent to assigning intention to a clock because the hands go around. We have a long way to go before we can/should assign intentionality to "robots" this way. Of course, at the rate things are going, "a long way" could happen in the next few years. But it's not happening now. I'll happily consider counter-examples.

ekanes 3y ago
Absolutely, but you're missing the most inflammatory part of all! The robot had to "eliminate the threat" -- when this was about a human who got in the way.
dfxm12 3y ago
What's the meaningful distinction between the robot ignoring or missing relevant information? We're not in a court of law trying to suss out intent or lay blame on a specific entity. The essay is about robots killing people, and supposing this will continue without government regulation. The people will be killed either way...
- NikolaNovak 3y ago
  We can't address and fix a problem if we fundamentally misunderstand it.
  - dfxm12 3y ago
    The irony is that we're in this problem in the first place because people are releasing problematic tech before it is understood in the first place. Humanity at large shouldn't be part of the experimentation process, and the essay is arguing that we need tighter regulations around that. The essay suggests "Any regulation of industrial robots stems from existing industrial regulation, which has been evolving for many decades" & that regulations need to be clear about other robots as well. Interestingly enough, if we applied existing regulations to Tesla's Autopilot as we would to a driver who racked up as many fatalities and crashes (736 crashes and 17 deaths in the U.S. since 2019 [0]), they'd be behind bars long ago, let alone never able to drive again. 0 - https://www.caranddriver.com/news/a44185487/report-tesla-autopilot-crashes-since-2019/ https://www.caranddriver.com/news/a44185487/report-tesla-aut...
nonrandomstring 3y ago
Intentionality is a central concept in AI philosophy - from debates between Dennet, Churchland and Searle going back to the 1980s. Aat this stage I agree that we are dealing with "blind machines", with no more "intention" than a steam engine. However, the central argument in Schneier's piece seems to be that robots inherit the intentionality of their creators. If I create a machine that kills someone, my "intentionality" is fair game to be dissected in court, and ultimately I may be found guilty as if I had killed with my own hands. That is the kind of high bar I think society expects, and is certainly one that should scare the crap out of robotics developers (aside: I have been one) to ensure the highest standards of engineering discipline and industrial ethics.
- JoeAltmaier 3y ago
  Nah. Idiots walk into machinery all the time. Put up signs, do safety training, yell "Stop!" and still Darwin takes his share. It's definitely fair to consider the 'intentionality' of the idiot too.
  - LapsangGuzzler 3y ago
    > Idiots walk into machinery all the time. Put up signs, do safety training, yell "Stop!" and still Darwin takes his share. It's still on leadership if they choose to hire idiots who get hurt. The solution is: don't hire idiots. If you look historically at manufacturing and the origins of signage and safety training, you'll find them mandated by the government (i.e. OSHA, etc.) because those things weren't always voluntarily provided by the employer. Upton Sinclair opened our eyes to this 100 years ago.
    - taneq 3y ago
      What you're missing here is the numbers game. The size of the operation vs. the tolerance for worker injuries has a massive impact on this stuff. When you run a shop with five experienced people, if each of them has a 1-in-a-million chance of getting seriously injured on any given day, each year you have a 1-in-548 chance of a serious injury. Any sane person would call this risk acceptable. If you run a shop with 5000 people of equal skill level to those five above, you're talking a couple of deaths or permanent disability injuries per year, just from the numbers. Any rational person would say hey, that's the same risk, why are you mad? But society isn't rational and if any legal entity no matter how large kills or maims a couple of workers a day, there's an outcry and executive heads will roll. So yeah, they try to do better. At this point maybe 'hire smarter people' works? Now imagine you're running a multinational company with 500,000 people. If you operate like that original shop you'll see a couple of hundred deaths in your first year and then get sued into oblivion or lynched, and remember those were skilled experts. There aren't 500,000 skilled experts available for hire, and most of these jobs are ahem less skilled anyway, so you're hiring people with much higher rates of personal injury. To operate at this level you need to do stuff that seems dumb to the uninitiated, like relying on layers of subcontracting companies for everything while implementing byzantine and ever-shifting safety regulations that each subcontractor must commit to following. Sure, it costs an order of magnitude more, but it's worth it for the severability.
umilegenio 3y ago
I agree with your first comment, but disagree on the second one. It is weird to say that the robot wanted to eliminate a threat, most likely it was simply not programmed to avoid obstacles given the environemnt it was in. He ascribes an intention that was not there. To be fair, it is a quote from a book and not the words chosen directly by the author, but he could have clarified. The second one seems fair: he is describing the result of what the Tesla Autopilot is doing. I do not think that he means to say that the Autopilot is understanding that there is a red light but then it makes a consciuous decision to itgnore it. He is just saying that the robot is malfunctioning, as it should be clear from context: "Today’s AI-related robot deaths are no different from the robot accidents of the past. Those industrial robots malfunctioned, and human operators trying to assist were killed in unexpected ways. [..] Malfunctioning Teslas on Autopilot have deviated from their advertised capabilities by misreading road markings, suddenly veering into other cars or trees, crashing into well-marked service vehicles, or ignoring red lights, stop signs, and crosswalks". I do not think there is a more neutral way of describing the events.
- gcanyon 3y ago
  > most likely it was simply not programmed to avoid obstacles given the environment it was in. Even more likely, it’s a manufacturing robot, with no vision at all except maybe for the pieces it is manipulating. It almost certainly has a line on the floor, or a railing even, to keep people out of its way. In short, it has zero obstacle avoidance. I’d be interested in counterexamples. > The second one seems fair: he is describing the result of what the Tesla Autopilot is doing. I do not think that he means to say that the Autopilot is understanding that there is a red light but then it makes a consciuous decision to itgnore it. He is just saying that the robot is malfunctioning, as it should be clear from context: "Today’s AI-related robot deaths are no different from the robot accidents of the past. Those industrial robots malfunctioned, and human operators trying to assist were killed in unexpected ways. [..] Malfunctioning Teslas on Autopilot have deviated from their advertised capabilities by misreading road markings, suddenly veering into other cars or trees, crashing into well-marked service vehicles, or ignoring red lights, stop signs, and crosswalks". He describes Teslas as “no different” from non-intentional machines of the past, and then describes them with “intentional” language. >I do not think there is a more neutral way of describing the events. How about, “Industrial accidents happen through lack of human foresight and carelessness. The same is currently true of ‘AI’ products like Autopilot. But the time will come, and probably soon, where AI alignment and intentionality will be part of the conversation. We should prepare for that starting now.” Not as catchy as “Robot rampage!” But more accurate and, as you say, neutral.
ballenf 3y ago
To put it another way, we might as well call a steam train a robot who will murder any human who tries to obstruct its desire to get to the next station, instead of jumping the track and going around.
- taneq 3y ago
  Or a brick wall, a robot who will murder any human who runs into it fast enough.
rauljara 3y ago
That quote leapt out at me too. I think the reason it leapt out was because the rest of the article was so thoroughly objective, well reasoned, and rooted in historical precedence. Given the context in which it appeared, I’m willing to give it the benefit of the doubt as slightly expedient wording that was shorthand for the more nuanced understanding he clearly possesses.
AbrahamParangi 3y ago
Yeah this is a great example of what’s wrong with all our robots: their perception is garbage. Reasoning is actually pretty good now. An AI can compose a sonnet on electron degeneracy pressure no-problem but ask it where the red ball is - and really inspect its localization of that red ball and it’s way off. Orders of magnitude of imprecision. A modern AI can reason through the trolley problem and millions of variants. But it can’t see the person on the track with very high confidence, or be sure that the trolley is going to hit the person or not.
- romesc 3y ago
  You're right about AI capabilities being much better in the "ungrounded" realm (like writing sonnets), but I would revise the statement a little to be more accurate. AI systems are quite good at perception in a traditional sense (e.g. determining a red ball is in the sensor receptive field), but once actual cognition about how the red ball relates to the world around it becomes important, current approaches are limited. I would argue that the first wave of perception is solved much more completely than the ability to reason about complex interactions of objects grounded in the physical world. Disclaimer: I am a robotics research scientist
  - AbrahamParangi 3y ago
    I think it matters how accurate something should be before we consider it "good". For many applications, a 99% success rate is unusably bad. A self driving car that recognizes 99% of red lights or stop signs is probably a criminal liability. A red ball detector that operates at 60hz with a 99.9% accuracy will make an error on average once every 11.5 seconds. I think the issue is not so much the interactions of objects and the world but rather we're just wildly off-base regarding the base performance needed.
Karellen 3y ago
If a human driver isn't paying attention to the road, fails to see a stop sign, and drives past it causing an accident, isn't it a reasonable use of English to say they "ignored" the sign? Ignoring can mean failing to notice something, as well as noticing but disregarding it. Words can often have multiple similar complementary meanings. I don't think the usage here is incorrect, or even misleading.
- clarkmoody 3y ago
  Any sort of ticket from the traffic police will say something like "failed to obey traffic control devices," which is more of a statement of fact. "Ignore" definitely has shades of nuance.
- c0pium 3y ago
  No it can’t. Every definition of ignore you will find has intention in it, words mean things. You don’t think it’s important to draw a distinction between whether a robot was deliberately programmed to do something malicious as opposed to a bug which caused an oversight? Come on.
  - Karellen 3y ago
    > Every definition of ignore you will find has intention in it https://dictionary.cambridge.org/us/dictionary/english/ignore https://dictionary.cambridge.org/us/dictionary/english/ignor... > to intentionally not listen or give attention to: > to give no attention to something or someone: Two definitions - one with intention, one without > words mean things. Yes, but they often mean multiple things. Very few dictionary entries only have a single definition for a word. You can't use "words mean things" to claim that "this word means only what I say it means, regardless of the other accepted uses it already has". > You don’t think it’s important to draw a distinction between whether a robot was deliberately programmed to do something malicious as opposed to a bug which caused an oversight? I do think there are times when it is very important to do that. I just don't think that "ignore" is the right word to make that distinction, and isn't a distinction the authors were trying to make at that particular time, because if they were they would probably have used another word. ...at least, that's how I read it. It seemed clear to me. ¯\_(ツ)_/¯
WeylandYutani 3y ago
It's just a way for tech bros to dodge responsibility. The people who program AI should be liable for injury.
PoignardAzur 3y ago
> > the robot simply determined that “the most efficient way to eliminate the threat was to push the worker into an adjacent machine.” > This seems inflammatory and highly misleading. I could be wrong, but I highly doubt it: the robot didn't "determine" anything. It had assembly instructions to follow, and it followed them. The worker got in the way and was struck. I was prepared to argue with you, but upon reading the article, I agree that the description you quote makes a huge category error. I think it's believable that we'll soon get AIs that are advanced enough they can do agent modeling and that some subgoal-maximizer accident could happen where the robots determines "the most efficient way to accomplish <objective> is to take out that human so he doesn't turn me off", in which case the article's poetic description would be correct. Except the quoted paragraph is talking about an industrial accident in 1981, so the whole "eliminate the threat" presentation is obviously absurd.

ekanes 3y ago

Absolutely, but you're missing the most inflammatory part of all! The robot had to "eliminate the threat" -- when this was about a human who got in the way.

dfxm12 3y ago

What's the meaningful distinction between the robot ignoring or missing relevant information? We're not in a court of law trying to suss out intent or lay blame on a specific entity. The essay is about robots killing people, and supposing this will continue without government regulation. The people will be killed either way...

NikolaNovak 3y ago
We can't address and fix a problem if we fundamentally misunderstand it.
- dfxm12 3y ago
  The irony is that we're in this problem in the first place because people are releasing problematic tech before it is understood in the first place. Humanity at large shouldn't be part of the experimentation process, and the essay is arguing that we need tighter regulations around that. The essay suggests "Any regulation of industrial robots stems from existing industrial regulation, which has been evolving for many decades" & that regulations need to be clear about other robots as well. Interestingly enough, if we applied existing regulations to Tesla's Autopilot as we would to a driver who racked up as many fatalities and crashes (736 crashes and 17 deaths in the U.S. since 2019 [0]), they'd be behind bars long ago, let alone never able to drive again. 0 - https://www.caranddriver.com/news/a44185487/report-tesla-autopilot-crashes-since-2019/ https://www.caranddriver.com/news/a44185487/report-tesla-aut...

nonrandomstring 3y ago

Intentionality is a central concept in AI philosophy - from debates between Dennet, Churchland and Searle going back to the 1980s. Aat this stage I agree that we are dealing with "blind machines", with no more "intention" than a steam engine. However, the central argument in Schneier's piece seems to be that robots inherit the intentionality of their creators. If I create a machine that kills someone, my "intentionality" is fair game to be dissected in court, and ultimately I may be found guilty as if I had killed with my own hands. That is the kind of high bar I think society expects, and is certainly one that should scare the crap out of robotics developers (aside: I have been one) to ensure the highest standards of engineering discipline and industrial ethics.

JoeAltmaier 3y ago
Nah. Idiots walk into machinery all the time. Put up signs, do safety training, yell "Stop!" and still Darwin takes his share. It's definitely fair to consider the 'intentionality' of the idiot too.
- LapsangGuzzler 3y ago
  > Idiots walk into machinery all the time. Put up signs, do safety training, yell "Stop!" and still Darwin takes his share. It's still on leadership if they choose to hire idiots who get hurt. The solution is: don't hire idiots. If you look historically at manufacturing and the origins of signage and safety training, you'll find them mandated by the government (i.e. OSHA, etc.) because those things weren't always voluntarily provided by the employer. Upton Sinclair opened our eyes to this 100 years ago.
  - taneq 3y ago
    What you're missing here is the numbers game. The size of the operation vs. the tolerance for worker injuries has a massive impact on this stuff. When you run a shop with five experienced people, if each of them has a 1-in-a-million chance of getting seriously injured on any given day, each year you have a 1-in-548 chance of a serious injury. Any sane person would call this risk acceptable. If you run a shop with 5000 people of equal skill level to those five above, you're talking a couple of deaths or permanent disability injuries per year, just from the numbers. Any rational person would say hey, that's the same risk, why are you mad? But society isn't rational and if any legal entity no matter how large kills or maims a couple of workers a day, there's an outcry and executive heads will roll. So yeah, they try to do better. At this point maybe 'hire smarter people' works? Now imagine you're running a multinational company with 500,000 people. If you operate like that original shop you'll see a couple of hundred deaths in your first year and then get sued into oblivion or lynched, and remember those were skilled experts. There aren't 500,000 skilled experts available for hire, and most of these jobs are ahem less skilled anyway, so you're hiring people with much higher rates of personal injury. To operate at this level you need to do stuff that seems dumb to the uninitiated, like relying on layers of subcontracting companies for everything while implementing byzantine and ever-shifting safety regulations that each subcontractor must commit to following. Sure, it costs an order of magnitude more, but it's worth it for the severability.

umilegenio 3y ago

I agree with your first comment, but disagree on the second one. It is weird to say that the robot wanted to eliminate a threat, most likely it was simply not programmed to avoid obstacles given the environemnt it was in. He ascribes an intention that was not there. To be fair, it is a quote from a book and not the words chosen directly by the author, but he could have clarified. The second one seems fair: he is describing the result of what the Tesla Autopilot is doing. I do not think that he means to say that the Autopilot is understanding that there is a red light but then it makes a consciuous decision to itgnore it. He is just saying that the robot is malfunctioning, as it should be clear from context: "Today’s AI-related robot deaths are no different from the robot accidents of the past. Those industrial robots malfunctioned, and human operators trying to assist were killed in unexpected ways. [..] Malfunctioning Teslas on Autopilot have deviated from their advertised capabilities by misreading road markings, suddenly veering into other cars or trees, crashing into well-marked service vehicles, or ignoring red lights, stop signs, and crosswalks". I do not think there is a more neutral way of describing the events.

gcanyon 3y ago
> most likely it was simply not programmed to avoid obstacles given the environment it was in. Even more likely, it’s a manufacturing robot, with no vision at all except maybe for the pieces it is manipulating. It almost certainly has a line on the floor, or a railing even, to keep people out of its way. In short, it has zero obstacle avoidance. I’d be interested in counterexamples. > The second one seems fair: he is describing the result of what the Tesla Autopilot is doing. I do not think that he means to say that the Autopilot is understanding that there is a red light but then it makes a consciuous decision to itgnore it. He is just saying that the robot is malfunctioning, as it should be clear from context: "Today’s AI-related robot deaths are no different from the robot accidents of the past. Those industrial robots malfunctioned, and human operators trying to assist were killed in unexpected ways. [..] Malfunctioning Teslas on Autopilot have deviated from their advertised capabilities by misreading road markings, suddenly veering into other cars or trees, crashing into well-marked service vehicles, or ignoring red lights, stop signs, and crosswalks". He describes Teslas as “no different” from non-intentional machines of the past, and then describes them with “intentional” language. >I do not think there is a more neutral way of describing the events. How about, “Industrial accidents happen through lack of human foresight and carelessness. The same is currently true of ‘AI’ products like Autopilot. But the time will come, and probably soon, where AI alignment and intentionality will be part of the conversation. We should prepare for that starting now.” Not as catchy as “Robot rampage!” But more accurate and, as you say, neutral.

ballenf 3y ago

To put it another way, we might as well call a steam train a robot who will murder any human who tries to obstruct its desire to get to the next station, instead of jumping the track and going around.

taneq 3y ago
Or a brick wall, a robot who will murder any human who runs into it fast enough.

rauljara 3y ago

That quote leapt out at me too. I think the reason it leapt out was because the rest of the article was so thoroughly objective, well reasoned, and rooted in historical precedence. Given the context in which it appeared, I’m willing to give it the benefit of the doubt as slightly expedient wording that was shorthand for the more nuanced understanding he clearly possesses.

AbrahamParangi 3y ago

Yeah this is a great example of what’s wrong with all our robots: their perception is garbage. Reasoning is actually pretty good now. An AI can compose a sonnet on electron degeneracy pressure no-problem but ask it where the red ball is - and really inspect its localization of that red ball and it’s way off. Orders of magnitude of imprecision. A modern AI can reason through the trolley problem and millions of variants. But it can’t see the person on the track with very high confidence, or be sure that the trolley is going to hit the person or not.

romesc 3y ago
You're right about AI capabilities being much better in the "ungrounded" realm (like writing sonnets), but I would revise the statement a little to be more accurate. AI systems are quite good at perception in a traditional sense (e.g. determining a red ball is in the sensor receptive field), but once actual cognition about how the red ball relates to the world around it becomes important, current approaches are limited. I would argue that the first wave of perception is solved much more completely than the ability to reason about complex interactions of objects grounded in the physical world. Disclaimer: I am a robotics research scientist
- AbrahamParangi 3y ago
  I think it matters how accurate something should be before we consider it "good". For many applications, a 99% success rate is unusably bad. A self driving car that recognizes 99% of red lights or stop signs is probably a criminal liability. A red ball detector that operates at 60hz with a 99.9% accuracy will make an error on average once every 11.5 seconds. I think the issue is not so much the interactions of objects and the world but rather we're just wildly off-base regarding the base performance needed.

Karellen 3y ago

If a human driver isn't paying attention to the road, fails to see a stop sign, and drives past it causing an accident, isn't it a reasonable use of English to say they "ignored" the sign? Ignoring can mean failing to notice something, as well as noticing but disregarding it. Words can often have multiple similar complementary meanings. I don't think the usage here is incorrect, or even misleading.

clarkmoody 3y ago
Any sort of ticket from the traffic police will say something like "failed to obey traffic control devices," which is more of a statement of fact. "Ignore" definitely has shades of nuance.
c0pium 3y ago
No it can’t. Every definition of ignore you will find has intention in it, words mean things. You don’t think it’s important to draw a distinction between whether a robot was deliberately programmed to do something malicious as opposed to a bug which caused an oversight? Come on.
- Karellen 3y ago
  > Every definition of ignore you will find has intention in it https://dictionary.cambridge.org/us/dictionary/english/ignore https://dictionary.cambridge.org/us/dictionary/english/ignor... > to intentionally not listen or give attention to: > to give no attention to something or someone: Two definitions - one with intention, one without > words mean things. Yes, but they often mean multiple things. Very few dictionary entries only have a single definition for a word. You can't use "words mean things" to claim that "this word means only what I say it means, regardless of the other accepted uses it already has". > You don’t think it’s important to draw a distinction between whether a robot was deliberately programmed to do something malicious as opposed to a bug which caused an oversight? I do think there are times when it is very important to do that. I just don't think that "ignore" is the right word to make that distinction, and isn't a distinction the authors were trying to make at that particular time, because if they were they would probably have used another word. ...at least, that's how I read it. It seemed clear to me. ¯\_(ツ)_/¯

WeylandYutani 3y ago

It's just a way for tech bros to dodge responsibility. The people who program AI should be liable for injury.

PoignardAzur 3y ago

> > the robot simply determined that “the most efficient way to eliminate the threat was to push the worker into an adjacent machine.” > This seems inflammatory and highly misleading. I could be wrong, but I highly doubt it: the robot didn't "determine" anything. It had assembly instructions to follow, and it followed them. The worker got in the way and was struck. I was prepared to argue with you, but upon reading the article, I agree that the description you quote makes a huge category error. I think it's believable that we'll soon get AIs that are advanced enough they can do agent modeling and that some subgoal-maximizer accident could happen where the robots determines "the most efficient way to accomplish <objective> is to take out that human so he doesn't turn me off", in which case the article's poetic description would be correct. Except the quoted paragraph is talking about an industrial accident in 1981, so the whole "eliminate the threat" presentation is obviously absurd.