16 ms·

Machine Behavior Needs to Be an Academic Discipline

96 points by dnetesn 9y ago

aje403 9y ago
The only thing that stopped me from calling the authors morons was that I googled their names and discovered they were both MIT PhDs. My only hope is that AI ethics doesn't become an MBA specification or else this mainly sounds like an excuse for more non-technicals with loud opinions to bs their way to into the industry, with some kind of moral high ground over coders Edit: Oh, author's entire career niche is and has been AI ethics, so, at any rate, he's just pumping his own field at an opportune moment
- nkrisc 9y ago
  What was it that initially made you think they are morons? Do you believe that AI should not be held to ethical standards?
  - taeric 9y ago
    It does seem to presupposed agency on the part of the machines being studied. We aren't worried about making sure all of our other machines are ethical. Why not? That is, we should maintain that the people involve are held to ethical standards. But I don't see that being up for debate. Is it?
    - nkrisc 9y ago
      > We aren't worried about making sure all of our other machines are ethical. Why not? As more machines are autonomously making decisions and exhibiting emergent behavior while also being able to directly interact with space shared with humans, maybe we should be worried about it.
      - taeric 9y ago
        The ethics are in building the machines, though. Consider, we don't worry about the ethics of bio-weapons, from the perspective of the weapon. Rather, we say it is unethical to build said weapons. (Right?) This doesn't change just because we could build a nanotech (or other) weapon that selectively kills people.
        nkrisc 9y ago
        I disagree, particularly if the machines are making their own decisions and exhibiting emergent behavior. I don't think only weapons should be considered. I think there will be many unintended outcomes from AI that was never meant to hurt anyone, but has nonetheless.
        taeric 9y ago
        But this is somewhat nonsensical. Is a slaughterhouse ethical because no kids are allowed to walk into them? Are they unethical because they would kill a person in the wrong place? It would be unethical to build a slaughterhouse that traveled around and had the potential to go into a populace. It seems odd to call the slaughterhouse itself unethical, when it was the building of the machine that is the problem. If you are giving agency to the machine, then you are wanting to teach the machine ethics. At some point, I can see that happening for "intelligence." Not for "intelligent machines" though.
  - aje403 9y ago
    It is still quite possible they are morons, it's just less certain. Where did you read me saying that? I think ethical standards for AI should come from reputable CS PhDs who understand the technology they're pontificating about working in conjunction with lawyers and legislators and taking into account public sentiment, which hopefully will eventually be the case when this becomes more of a mainstream issue. The authors do not appear to have any of that background. I think the AI aistorian/philosopher/psychologist field belongs on the fringe for now and likely a very long time.
    - nkrisc 9y ago
      I did not read you saying that, I was asking you a question. > ethical standards for AI should come from reputable CS PhDs who understand the technology they're pontificating about Should AI be held to different ethical standards than humans for the same actions? Given that we are beginning to see AI (proto-AI?) directly manipulate and interact with our physical world (autonomous vehicles, smart homes, etc.) I think it's a perfect time to study AI ethics in depth.
      - aje403 9y ago
        Quote the entire sentence if you want to quote me there. I do not think "trained ethics experts" should fall into any equation whatsoever. There are enough of them in the middle east and deep south. We don't need them infecting the intelligent world anymore than they do already. The conversation should not be driven by an intellectual movement that is equivalent to a bunch of cavemen poking a coconut with sticks. As my post that you're responding to stated, this will enter the mainstream in a few years with a lot of unqualified people voicing their opinions. Hopefully the qualified people end up drafting the laws (note: not professional ethicists)
        nkrisc 9y ago
        Now you've finally made your position and opposition clear: you believe "ethics" is interchangeable with "religion." I would not to work alongside an autonomous machine designed by someone who dismisses all ethics as nothing more than superstitious mythology. I think your view is a great example of why AI ethics should be examined by outsiders.
  - btilly 9y ago
    For what it is worth, I had a similar gut level reaction. See https://news.ycombinator.com/item?id=16738190 https://news.ycombinator.com/item?id=16738190 for my explanation of why I think that this is a bad idea.
    - nkrisc 9y ago
      I think you're overlooking the needs of a third stakeholder in this debate: the people who will have to share space with autonomous and/or artificially intelligent machines but are neither an expert in ethics nor creators of AI. As a bystander to this debate I'd like to know the debate is happening now and not after I've been killed by an autonomous machine that may or may not have been acting ethically.
currymj 9y ago
if a sociologist wanted to study the social impact of transit infrastructure, that would be perfectly reasonable, and we wouldn't expect them to know much going in about signaling systems, switches, or different types of rolling stock. likely a railway engineer wouldn't be all that successful trying to do the sociologist's job. i think machine learning systems are different, and any researcher in a new field of "Machine Behavior" would have to know much more about how they work than in the railroad example. but there's no reason to suppose that work done by people trained outside of CS departments will be nonsense. and as the authors say, there are real blind spots where CS researchers might just ask the wrong questions entirely.
- adamsea 9y ago
  > if a sociologist wanted to study the social impact of transit infrastructure, that would be perfectly reasonable, and we wouldn't expect them to know much going in about signaling systems, switches, or different types of rolling stock. Yes -- and a good sociologist would also learn a lot about that stuff once they started their research, I'd imagine :). Not enough to be a railway engineer, but presumably enough to be able to speak meaningfully & intelligently to railway engineers. Not unlike science journalists.
zitterbewegung 9y ago
We covered these topics in my Computer Ethics class. I took a bunch of philosophy classes on Ethics too. You can teach a person from a book all about Ethics. The problem is that people will make the decision to be ethical with their gut or in the moment. For instance I was designing an app that would identify if the food had an allergen or not. Eventually I came to the realization that my program if deployed could actually hurt people. So I stopped working on the project. I wrote a blogpost about this (shameless plug) at https://medium.com/@zitterbewegung/making-computer-vision-systems-that-dont-kill-your-users-2c98228d9032 https://medium.com/@zitterbewegung/making-computer-vision-sy... . The core issue was I was giving people information that they could perform medical decisions. This gave me a bad feeling and a bunch of lawyers told me when I referenced this post. I eventually pivoted what I learned into this (yet another shameless plug) : https://steemit.com/twilio/@zitterbewegung/mms2text-let-your-computer-look-at-your-message-pics-so-you-don-t-have-to https://steemit.com/twilio/@zitterbewegung/mms2text-let-your... which doesn't have the issue of harming people.
- gowld 9y ago
  But not making your app killed people who didn't notice nuts in their food...
  - venuur 9y ago
    That is too strong a conclusion. People with nut allergies already have procedures for knowing if nuts are in food, for example, asking the restaurant or preparing food their self. Without knowing if his app was accurate, we cannot say whether building or not building the app was the right decision.
    - zitterbewegung 9y ago
      The app was very inaccurate during my testing. Hiding almonds in a dish made out of pure chocolate (a Hershey kiss) would be an easy counter example. Also, I have a nut allergy so I have a complicated set of procedures to figure out if nuts are in the dish (that was the main motivation for the app).
jaddood 9y ago
The problem here, I think, is that although humans can be studied without knowing the structure of their brains, (psychology, sociology, etc...) machines of today cannot. The reason for that is that humans are humans and barely differ from each other biologically (the structure of the brain), but ANNs are very diverse and their behavior changes depending on their "biology," so machine psychology or machine sociology cannot be built on stable grounds
- gowld 9y ago
  > although humans can be studied without knowing the structure of their brains, (psychology, sociology, etc...) machines of today cannot. why? Or do you mean that study of some humans is generalizable to most/all humans, whereas machines lack this consistency and generalizability? Anyway, why would this concern prevent "feature detection" tests that apply to all machines?
btilly 9y ago
When you create a group of experts whose expertise lies in telling other people how what they're doing is wrong, you create problems. The experts need to justify their position by coming up with more and more reasons that people need to listen to them. People who are actually doing the work quickly become convinced that said experts are ignorant asses who don't know what they are doing. This automatically sets up grounds for conflict. I agree that the ethics of AI is important. However I expect the best work to come from people who are experts in AI first, and only secondarily in ethics. And I expect that the self-appointed experts in ethics won't recognize this. (Particularly not when faced with anything that makes their position seem less important.)
- 21 9y ago
  Your proposal has the exact opposite problem - the AI experts are incentivized to not make two much waves regarding the dangers, because that would endanger the field (and money invested in it). Thus we get quotes from them like "overpopulation on Mars"
  - aje403 9y ago
    This doesn't make any sense. Why are you lumping AI experts into a group? I'm trying to picture if the government went around to all highly qualifying AI PhDs asking them if they would like to join a prominent ethics panel to guide and aid in AI ethics regulation in industry. So, you're claiming they're going to decline that or accept and sabotage under the assumption that they're a. all members of a giant AI consortium and b. any ethics regulation is going to destroy their industry? As bad of an example as this is - look at other industries. Banking officials in government who set banking policies come from the finance industry. While this certainly has its problems and how they get there comes into question, they actually understand the industry they're regulating.
    - 21 9y ago
      I didn't say forbid AI PhDs to work on ethics. I just say that an AI PhD shouldn't be a requirement. We already have an ethics discipline. This is an ethics problem. Do you also propose that for example self-driving cars accidents should be investigated by AI PhDs which worked on self-driving cars algorithms, and they should be the ones proposing regulations? Are you saying that a regular "car accident specialist" is ill equipped to handle these, and it's opinions aren't as valuable because he's an expert in "human car accidents" not "AI car accidents"?
Veedrac 9y ago
This article seems really wishy-washy and confusing. What is actually being proposed? A set of standards that automated algorithms should adhere to? What kind of standards are we talking about? A new branch of philosophy that talks about the actions of automated algorithms? What would be the point? How do you actually turn that into something that progresses, and that people will act on? More ethics classes in universities for CS students? How do you differentiate this from what's already happening? Is it just broader? --- To try to steelman this a bit, I figure there are two issues at play. The first is making sure that AI is working in the way we designed it to; it's free of biases, it doesn't endanger people, it adheres to laws. We're doing horifficially on this metric when it comes to long-term superhuman AI alignment stuff, but most people are talking about next-decade issues. Those are raw technical issues, and though we're still working on robustness and detecting biases I really cannot see much need for external guidance outside of the natural technical progress that the field is already heavily invested in. These are problems we want to solve already. The second issue is use of these technologies as tools. This is where we talk about how large companies' algorithms affect public perception, or how automated militaries affects warfare, or how self-driving cars are litigated. These are not AI problems, they're social problems. Yes, the technology behind those examples looks kind'a similar behind the scenes, but these are distinct social, legal and economic problems. This is kind of like seeing the advent of electricity, predicting its effect on society, and asking for people to study "ethics of electricity".
- currymj 9y ago
  the first problem is one where CS researchers would likely benefit from more contact with outsiders, although I agree it's a pretty lively area of research already, at least for the shorter-term issues. the second problem is one where social scientists could benefit from actually understanding how these systems work. yes, they are social problems, but the sociologists could do much better if they had a better understanding of the details of the technology. The authors say they want "a consolidated, scalable, and scientific approach to the behavioral study of artificial intelligence agents in which social scientists and computer scientists can collaborate seamlessly". So, a new interdisciplinary category. I get the sense they are especially concerned with the empirical study of behavior and social impact of big ML systems in the real world. In another post I compared this sort of thing to sociologists studying transit infrastructure.
sgt101 9y ago
"In our own work—being computer scientists ourselves—we’ve been frequently humbled by our social and behavioral science collaborators. " this part rings truer than any other phrase in the whole thing.
bad_ramen_soup 9y ago
"We often cannot certify that an AI agent is optimal or ethical by looking at its source code, any more than we can certify that humans are good by scanning their brains." What do people make of this claim? Is that a fair comparison?
- bo1024 9y ago
  It's fair to some extent. The human brain is vastly more complex, so I don't love the comparison. However, with very modern machine learning which is based on deep neural networks, even though the source code is simple, it produces a very complex neural network after training on data. That network is very hard to understand and predict. Even with simpler algorithms, you might argue that in a complex environment they could have some emergent behavior that's hard to foresee based on the source code. For example, if one is not careful then even simple linear regression on some datasets can "learn" techniques like redlining -- using zip code as a proxy for race as a part of biased predictions.
taneq 9y ago
It sounds like the term they're looking for is "robopsychologist", I was hoping for a mention of the great Susan Calvin.
s17n 9y ago
AI is not yet complex enough to warrant such a discipline. Obviously, like any other tool, AI systems are part of the purview of anyone studying how society works (and as a particularly complicated tool, deserve a particularly extensive examination.) But at this point, studying the ethics of a machine is like studying the ethics of a fruit fly.

96 points by dnetesn 9y ago

The only thing that stopped me from calling the authors morons was that I googled their names and discovered they were both MIT PhDs. My only hope is that AI ethics doesn't become an MBA specification or else this mainly sounds like an excuse for more non-technicals with loud opinions to bs their way to into the industry, with some kind of moral high ground over coders Edit: Oh, author's entire career niche is and has been AI ethics, so, at any rate, he's just pumping his own field at an opportune moment

nkrisc 9y ago
What was it that initially made you think they are morons? Do you believe that AI should not be held to ethical standards?
- taeric 9y ago
  It does seem to presupposed agency on the part of the machines being studied. We aren't worried about making sure all of our other machines are ethical. Why not? That is, we should maintain that the people involve are held to ethical standards. But I don't see that being up for debate. Is it?
  - nkrisc 9y ago
    > We aren't worried about making sure all of our other machines are ethical. Why not? As more machines are autonomously making decisions and exhibiting emergent behavior while also being able to directly interact with space shared with humans, maybe we should be worried about it.
    - taeric 9y ago
      The ethics are in building the machines, though. Consider, we don't worry about the ethics of bio-weapons, from the perspective of the weapon. Rather, we say it is unethical to build said weapons. (Right?) This doesn't change just because we could build a nanotech (or other) weapon that selectively kills people.
      - nkrisc 9y ago
        I disagree, particularly if the machines are making their own decisions and exhibiting emergent behavior. I don't think only weapons should be considered. I think there will be many unintended outcomes from AI that was never meant to hurt anyone, but has nonetheless.
        taeric 9y ago
        But this is somewhat nonsensical. Is a slaughterhouse ethical because no kids are allowed to walk into them? Are they unethical because they would kill a person in the wrong place? It would be unethical to build a slaughterhouse that traveled around and had the potential to go into a populace. It seems odd to call the slaughterhouse itself unethical, when it was the building of the machine that is the problem. If you are giving agency to the machine, then you are wanting to teach the machine ethics. At some point, I can see that happening for "intelligence." Not for "intelligent machines" though.
- aje403 9y ago
  It is still quite possible they are morons, it's just less certain. Where did you read me saying that? I think ethical standards for AI should come from reputable CS PhDs who understand the technology they're pontificating about working in conjunction with lawyers and legislators and taking into account public sentiment, which hopefully will eventually be the case when this becomes more of a mainstream issue. The authors do not appear to have any of that background. I think the AI aistorian/philosopher/psychologist field belongs on the fringe for now and likely a very long time.
  - nkrisc 9y ago
    I did not read you saying that, I was asking you a question. > ethical standards for AI should come from reputable CS PhDs who understand the technology they're pontificating about Should AI be held to different ethical standards than humans for the same actions? Given that we are beginning to see AI (proto-AI?) directly manipulate and interact with our physical world (autonomous vehicles, smart homes, etc.) I think it's a perfect time to study AI ethics in depth.
    - aje403 9y ago
      Quote the entire sentence if you want to quote me there. I do not think "trained ethics experts" should fall into any equation whatsoever. There are enough of them in the middle east and deep south. We don't need them infecting the intelligent world anymore than they do already. The conversation should not be driven by an intellectual movement that is equivalent to a bunch of cavemen poking a coconut with sticks. As my post that you're responding to stated, this will enter the mainstream in a few years with a lot of unqualified people voicing their opinions. Hopefully the qualified people end up drafting the laws (note: not professional ethicists)
      - nkrisc 9y ago
        Now you've finally made your position and opposition clear: you believe "ethics" is interchangeable with "religion." I would not to work alongside an autonomous machine designed by someone who dismisses all ethics as nothing more than superstitious mythology. I think your view is a great example of why AI ethics should be examined by outsiders.
- btilly 9y ago
  For what it is worth, I had a similar gut level reaction. See https://news.ycombinator.com/item?id=16738190 https://news.ycombinator.com/item?id=16738190 for my explanation of why I think that this is a bad idea.
  - nkrisc 9y ago
    I think you're overlooking the needs of a third stakeholder in this debate: the people who will have to share space with autonomous and/or artificially intelligent machines but are neither an expert in ethics nor creators of AI. As a bystander to this debate I'd like to know the debate is happening now and not after I've been killed by an autonomous machine that may or may not have been acting ethically.

currymj 9y ago

if a sociologist wanted to study the social impact of transit infrastructure, that would be perfectly reasonable, and we wouldn't expect them to know much going in about signaling systems, switches, or different types of rolling stock. likely a railway engineer wouldn't be all that successful trying to do the sociologist's job. i think machine learning systems are different, and any researcher in a new field of "Machine Behavior" would have to know much more about how they work than in the railroad example. but there's no reason to suppose that work done by people trained outside of CS departments will be nonsense. and as the authors say, there are real blind spots where CS researchers might just ask the wrong questions entirely.

adamsea 9y ago
> if a sociologist wanted to study the social impact of transit infrastructure, that would be perfectly reasonable, and we wouldn't expect them to know much going in about signaling systems, switches, or different types of rolling stock. Yes -- and a good sociologist would also learn a lot about that stuff once they started their research, I'd imagine :). Not enough to be a railway engineer, but presumably enough to be able to speak meaningfully & intelligently to railway engineers. Not unlike science journalists.

zitterbewegung 9y ago

We covered these topics in my Computer Ethics class. I took a bunch of philosophy classes on Ethics too. You can teach a person from a book all about Ethics. The problem is that people will make the decision to be ethical with their gut or in the moment. For instance I was designing an app that would identify if the food had an allergen or not. Eventually I came to the realization that my program if deployed could actually hurt people. So I stopped working on the project. I wrote a blogpost about this (shameless plug) at https://medium.com/@zitterbewegung/making-computer-vision-systems-that-dont-kill-your-users-2c98228d9032 https://medium.com/@zitterbewegung/making-computer-vision-sy... . The core issue was I was giving people information that they could perform medical decisions. This gave me a bad feeling and a bunch of lawyers told me when I referenced this post. I eventually pivoted what I learned into this (yet another shameless plug) : https://steemit.com/twilio/@zitterbewegung/mms2text-let-your-computer-look-at-your-message-pics-so-you-don-t-have-to https://steemit.com/twilio/@zitterbewegung/mms2text-let-your... which doesn't have the issue of harming people.

gowld 9y ago
But not making your app killed people who didn't notice nuts in their food...
- venuur 9y ago
  That is too strong a conclusion. People with nut allergies already have procedures for knowing if nuts are in food, for example, asking the restaurant or preparing food their self. Without knowing if his app was accurate, we cannot say whether building or not building the app was the right decision.
  - zitterbewegung 9y ago
    The app was very inaccurate during my testing. Hiding almonds in a dish made out of pure chocolate (a Hershey kiss) would be an easy counter example. Also, I have a nut allergy so I have a complicated set of procedures to figure out if nuts are in the dish (that was the main motivation for the app).

jaddood 9y ago

The problem here, I think, is that although humans can be studied without knowing the structure of their brains, (psychology, sociology, etc...) machines of today cannot. The reason for that is that humans are humans and barely differ from each other biologically (the structure of the brain), but ANNs are very diverse and their behavior changes depending on their "biology," so machine psychology or machine sociology cannot be built on stable grounds

gowld 9y ago
> although humans can be studied without knowing the structure of their brains, (psychology, sociology, etc...) machines of today cannot. why? Or do you mean that study of some humans is generalizable to most/all humans, whereas machines lack this consistency and generalizability? Anyway, why would this concern prevent "feature detection" tests that apply to all machines?

btilly 9y ago

When you create a group of experts whose expertise lies in telling other people how what they're doing is wrong, you create problems. The experts need to justify their position by coming up with more and more reasons that people need to listen to them. People who are actually doing the work quickly become convinced that said experts are ignorant asses who don't know what they are doing. This automatically sets up grounds for conflict. I agree that the ethics of AI is important. However I expect the best work to come from people who are experts in AI first, and only secondarily in ethics. And I expect that the self-appointed experts in ethics won't recognize this. (Particularly not when faced with anything that makes their position seem less important.)

21 9y ago
Your proposal has the exact opposite problem - the AI experts are incentivized to not make two much waves regarding the dangers, because that would endanger the field (and money invested in it). Thus we get quotes from them like "overpopulation on Mars"
- aje403 9y ago
  This doesn't make any sense. Why are you lumping AI experts into a group? I'm trying to picture if the government went around to all highly qualifying AI PhDs asking them if they would like to join a prominent ethics panel to guide and aid in AI ethics regulation in industry. So, you're claiming they're going to decline that or accept and sabotage under the assumption that they're a. all members of a giant AI consortium and b. any ethics regulation is going to destroy their industry? As bad of an example as this is - look at other industries. Banking officials in government who set banking policies come from the finance industry. While this certainly has its problems and how they get there comes into question, they actually understand the industry they're regulating.
  - 21 9y ago
    I didn't say forbid AI PhDs to work on ethics. I just say that an AI PhD shouldn't be a requirement. We already have an ethics discipline. This is an ethics problem. Do you also propose that for example self-driving cars accidents should be investigated by AI PhDs which worked on self-driving cars algorithms, and they should be the ones proposing regulations? Are you saying that a regular "car accident specialist" is ill equipped to handle these, and it's opinions aren't as valuable because he's an expert in "human car accidents" not "AI car accidents"?

Veedrac 9y ago

This article seems really wishy-washy and confusing. What is actually being proposed? A set of standards that automated algorithms should adhere to? What kind of standards are we talking about? A new branch of philosophy that talks about the actions of automated algorithms? What would be the point? How do you actually turn that into something that progresses, and that people will act on? More ethics classes in universities for CS students? How do you differentiate this from what's already happening? Is it just broader? --- To try to steelman this a bit, I figure there are two issues at play. The first is making sure that AI is working in the way we designed it to; it's free of biases, it doesn't endanger people, it adheres to laws. We're doing horifficially on this metric when it comes to long-term superhuman AI alignment stuff, but most people are talking about next-decade issues. Those are raw technical issues, and though we're still working on robustness and detecting biases I really cannot see much need for external guidance outside of the natural technical progress that the field is already heavily invested in. These are problems we want to solve already. The second issue is use of these technologies as tools. This is where we talk about how large companies' algorithms affect public perception, or how automated militaries affects warfare, or how self-driving cars are litigated. These are not AI problems, they're social problems. Yes, the technology behind those examples looks kind'a similar behind the scenes, but these are distinct social, legal and economic problems. This is kind of like seeing the advent of electricity, predicting its effect on society, and asking for people to study "ethics of electricity".

currymj 9y ago
the first problem is one where CS researchers would likely benefit from more contact with outsiders, although I agree it's a pretty lively area of research already, at least for the shorter-term issues. the second problem is one where social scientists could benefit from actually understanding how these systems work. yes, they are social problems, but the sociologists could do much better if they had a better understanding of the details of the technology. The authors say they want "a consolidated, scalable, and scientific approach to the behavioral study of artificial intelligence agents in which social scientists and computer scientists can collaborate seamlessly". So, a new interdisciplinary category. I get the sense they are especially concerned with the empirical study of behavior and social impact of big ML systems in the real world. In another post I compared this sort of thing to sociologists studying transit infrastructure.

sgt101 9y ago

"In our own work—being computer scientists ourselves—we’ve been frequently humbled by our social and behavioral science collaborators. " this part rings truer than any other phrase in the whole thing.

bad_ramen_soup 9y ago

"We often cannot certify that an AI agent is optimal or ethical by looking at its source code, any more than we can certify that humans are good by scanning their brains." What do people make of this claim? Is that a fair comparison?

bo1024 9y ago
It's fair to some extent. The human brain is vastly more complex, so I don't love the comparison. However, with very modern machine learning which is based on deep neural networks, even though the source code is simple, it produces a very complex neural network after training on data. That network is very hard to understand and predict. Even with simpler algorithms, you might argue that in a complex environment they could have some emergent behavior that's hard to foresee based on the source code. For example, if one is not careful then even simple linear regression on some datasets can "learn" techniques like redlining -- using zip code as a proxy for race as a part of biased predictions.

taneq 9y ago

It sounds like the term they're looking for is "robopsychologist", I was hoping for a mention of the great Susan Calvin.

s17n 9y ago

AI is not yet complex enough to warrant such a discipline. Obviously, like any other tool, AI systems are part of the purview of anyone studying how society works (and as a particularly complicated tool, deserve a particularly extensive examination.) But at this point, studying the ethics of a machine is like studying the ethics of a fruit fly.