7 ms·

They're right. Why would we put a NN in charge of anything important if we can't explain how a particular model works? Would you want your car or an aircraft

They're right. Why would we put a NN in charge of anything important if we can't explain how a particular model works? Would you want your car or an aircraft you're on piloted by neural net the actions of which can't be explained? What if it encounters an unforseen event that causes a flash crash or worse an actual crash that kills people? Do you want to trust something built from incomplete data and simulated annealing with your life and livelihood?

bitL 8y ago
Why do you trust moody, inconsistent people with their own biases as your leaders in life-or-death situations then?
- lawlessone 8y ago
  I'm not saying all AI is bad, but in the case of NN, they're full of biases known and unknown.
  - mdorazio 8y ago
    > they're full of biases known and unknown Are we talking about AI or people?
    - gbrown 8y ago
      Depending on where the training data comes from, both ;)
- deleted 8y ago
  [deleted]
- claydavisss 8y ago
  People can be fined or jailed
  - adrianN 8y ago
    And software can be deleted.
    - lawlessone 8y ago
      >And software can be deleted. The software doesn't care and isn't intimidated by that fact... yet.
  - wpietri 8y ago
    That's an important point. One of the depressing features of the 2008 financial crisis was the great number of industry titans who went smoothly from "My awesome insight is worth tens or hundreds of millions annually" to "Who could possibly have known?" I would hate to see the same thing happen again when algorithms fail to work as hoped. If people build and deploy a system, they should be socially and legally responsible for its effects in the world. And executives should be held responsible by the rest of us. I'd like us to be done with the shrug-emoji-and-cash-the-bonus-check school of corporate governance.
- jerf 8y ago
  We have centuries/millennia/choose-a-time-frame of experience with what they do, how they tend to perform, and while it's not necessarily perfect, we know how to engineer around their limitations, again, with centuries/millennia/etc. of experience in that field. Computer models are much younger, and we know they tend to have weird pathological corners, but unlike humans and their weird pathological corners, we have a much less firm grasp on what they are. In many cases, humans have skin in the game, too. No computer model is yet sophisticated enough to be able to say that about. There is also some irrationality in having someone to blame, etc. Certainly. But it's not the only part of the story.
  - samstave 8y ago
    Your comment made me think of something, perhaps in the larger assessment of the soundness/comfort-level we will/do have with AI decision making: With the moody, inconsistent people, we can understand and evaluate their incentive reasoning - positive or negative and empathize with that understanding to assign a level of trust. With any AI - especially to ones who didnt design/create it - any and all of its motivations, incentives are completely (emotionally) opaque to the recipient of the outcomes of its behavior. I know we are saying the same thing - but unless a "receipt" can be given for a decision to why an AI / NN made a decision, people will not learn to build an understanding of a trust. Otherwise, AI will always be hated by the humans who feel slighted by the cold decisions of a machine against them. And that is clearly going to be the real future. There will be anti-AI terrorism/activism and violence against systems, companies and countries that make life-bending decisions by AI against groups of people. The seed to this future is the gatekeeping to capital/resources through FICO-like systems.
    - cr0sh 8y ago
      > There will be anti-AI terrorism/activism and violence There's already been bomb attacks in Mexico against nanotechnology researchers, so it's not a big leap from there to AI/ML and other similar research (especially into GAI).
  - wpietri 8y ago
    Absolutely. We all daily get practice in understanding humans, so I think humans are much better prepared to handle a rogue employee than a rogue model.
- TeMPOraL 8y ago
  Like 'jerf says, we have many centuries of experience in dealing with people and people-based systems. We know the biases, the cognitive problems, the buttons to press and the buttons to absolutely avoid pressing. More importantly, humans share the same brain architecture and work roughly the same way - you and me included. This makes it easier for us to understand other people (also having a part of the brain trying to directly model other people's minds helps). The way those NN-based black boxes work is completely alien to us (and each being a special snowflake made of hacks does not help).
nradov 8y ago
There's a huge difference between trading algorithms and safety-critical control systems. If you're speculating with other people's money it's not entirely unreasonable to risk it using unexplainable models, provided you disclose your general approach to investors.
covemarkets 8y ago
"Would you want your car or an aircraft you're on piloted by neural net the actions of which can't be explained?" Actually, if the net is properly trained to reduce overfitting and is shown to work on out of sample data - yes. On the flip side of this argument, a person can do something inexplicable and invent a plausible explanation after the fact that seems and feels safe and correct, but is actually wrong.
- TeMPOraL 8y ago
  Define: "properly trained". With humans, we generally know the bounds for unexpected behavior. We understand tiredness, confusion, fear, distraction, suicidal thoughts and other factors. We also know how to screen people to minimize those bounds. With ML stacks, we have no good grasp on bounds. They usually work, for some definition of working, up until they don't - and when they fail, it's in some absurd (therefore hard to predict) way.
  - covemarkets 8y ago
    humans also fail for all sorts of reasons that are unpredictable. if, for example, Waymo, demonstrates properly calculated lower accident rates with their cars than with humans, then at some point you have to agree with what the data says. We aren't there yet, but that doesn't mean we can't get there
    - TeMPOraL 8y ago
      If they demonstrate, sure. But for that to be useful they also need to demonstrate that their model is robust under modifications - otherwise every time they retrain their NNs they should throw away prior safety records and start counting from zero. Because at this point, what would be the argument to keep it? With humans we know - from thousands of years of experience - that people generally don't go crazy when taught new things. With NNs, we know they're very sensitive to training data.
      - covemarkets 8y ago
cwilkes 8y ago
I don’t understand what you are asking for. 100% certainty before making any decision? Any decision in a messy real world that’s far more complicated than a simple math statement that can be proved?
quotemstr 8y ago
> Would you want your car or an aircraft you're on piloted by neural net the actions of which can't be explained? Almost all cars and aircraft today are operated by neural nets, the actions of which can't be explained.
- quonn 8y ago
  As anyone driving a car can assure you, the actions can be explained, easily.
  - olefoo 8y ago
    But are the explanations true? Are they deterministic? If someone took ten minutes of video of you driving in city traffic; could you consistently explain all of your decisions that would be recorded? It's very easy to tell yourself that your actions are consistent and explainable ( rational ) but if examined closely they aren't always consistent with any one set of heuristics. Following distance is one example; can you explain the decision making process by which you decide how much space to give the vehicle in front of you? Why it varies based on road conditions; weather or your estimate of your reaction times?
TuringNYC 8y ago
I'd also be worried about the model being biased. For example, perhaps the model is "better" because it uses correlated features it shouldnt be using (due to fairness laws, etc.)

bitL 8y ago

Why do you trust moody, inconsistent people with their own biases as your leaders in life-or-death situations then?

lawlessone 8y ago
I'm not saying all AI is bad, but in the case of NN, they're full of biases known and unknown.
- mdorazio 8y ago
  > they're full of biases known and unknown Are we talking about AI or people?
  - gbrown 8y ago
    Depending on where the training data comes from, both ;)
deleted 8y ago
[deleted]
claydavisss 8y ago
People can be fined or jailed
- adrianN 8y ago
  And software can be deleted.
  - lawlessone 8y ago
    >And software can be deleted. The software doesn't care and isn't intimidated by that fact... yet.
- wpietri 8y ago
  That's an important point. One of the depressing features of the 2008 financial crisis was the great number of industry titans who went smoothly from "My awesome insight is worth tens or hundreds of millions annually" to "Who could possibly have known?" I would hate to see the same thing happen again when algorithms fail to work as hoped. If people build and deploy a system, they should be socially and legally responsible for its effects in the world. And executives should be held responsible by the rest of us. I'd like us to be done with the shrug-emoji-and-cash-the-bonus-check school of corporate governance.
jerf 8y ago
We have centuries/millennia/choose-a-time-frame of experience with what they do, how they tend to perform, and while it's not necessarily perfect, we know how to engineer around their limitations, again, with centuries/millennia/etc. of experience in that field. Computer models are much younger, and we know they tend to have weird pathological corners, but unlike humans and their weird pathological corners, we have a much less firm grasp on what they are. In many cases, humans have skin in the game, too. No computer model is yet sophisticated enough to be able to say that about. There is also some irrationality in having someone to blame, etc. Certainly. But it's not the only part of the story.
- samstave 8y ago
  Your comment made me think of something, perhaps in the larger assessment of the soundness/comfort-level we will/do have with AI decision making: With the moody, inconsistent people, we can understand and evaluate their incentive reasoning - positive or negative and empathize with that understanding to assign a level of trust. With any AI - especially to ones who didnt design/create it - any and all of its motivations, incentives are completely (emotionally) opaque to the recipient of the outcomes of its behavior. I know we are saying the same thing - but unless a "receipt" can be given for a decision to why an AI / NN made a decision, people will not learn to build an understanding of a trust. Otherwise, AI will always be hated by the humans who feel slighted by the cold decisions of a machine against them. And that is clearly going to be the real future. There will be anti-AI terrorism/activism and violence against systems, companies and countries that make life-bending decisions by AI against groups of people. The seed to this future is the gatekeeping to capital/resources through FICO-like systems.
  - cr0sh 8y ago
    > There will be anti-AI terrorism/activism and violence There's already been bomb attacks in Mexico against nanotechnology researchers, so it's not a big leap from there to AI/ML and other similar research (especially into GAI).
- wpietri 8y ago
  Absolutely. We all daily get practice in understanding humans, so I think humans are much better prepared to handle a rogue employee than a rogue model.
TeMPOraL 8y ago
Like 'jerf says, we have many centuries of experience in dealing with people and people-based systems. We know the biases, the cognitive problems, the buttons to press and the buttons to absolutely avoid pressing. More importantly, humans share the same brain architecture and work roughly the same way - you and me included. This makes it easier for us to understand other people (also having a part of the brain trying to directly model other people's minds helps). The way those NN-based black boxes work is completely alien to us (and each being a special snowflake made of hacks does not help).

nradov 8y ago

There's a huge difference between trading algorithms and safety-critical control systems. If you're speculating with other people's money it's not entirely unreasonable to risk it using unexplainable models, provided you disclose your general approach to investors.

covemarkets 8y ago

"Would you want your car or an aircraft you're on piloted by neural net the actions of which can't be explained?" Actually, if the net is properly trained to reduce overfitting and is shown to work on out of sample data - yes. On the flip side of this argument, a person can do something inexplicable and invent a plausible explanation after the fact that seems and feels safe and correct, but is actually wrong.

TeMPOraL 8y ago
Define: "properly trained". With humans, we generally know the bounds for unexpected behavior. We understand tiredness, confusion, fear, distraction, suicidal thoughts and other factors. We also know how to screen people to minimize those bounds. With ML stacks, we have no good grasp on bounds. They usually work, for some definition of working, up until they don't - and when they fail, it's in some absurd (therefore hard to predict) way.
- covemarkets 8y ago
  humans also fail for all sorts of reasons that are unpredictable. if, for example, Waymo, demonstrates properly calculated lower accident rates with their cars than with humans, then at some point you have to agree with what the data says. We aren't there yet, but that doesn't mean we can't get there
  - TeMPOraL 8y ago
    If they demonstrate, sure. But for that to be useful they also need to demonstrate that their model is robust under modifications - otherwise every time they retrain their NNs they should throw away prior safety records and start counting from zero. Because at this point, what would be the argument to keep it? With humans we know - from thousands of years of experience - that people generally don't go crazy when taught new things. With NNs, we know they're very sensitive to training data.
    - covemarkets 8y ago

cwilkes 8y ago

I don’t understand what you are asking for. 100% certainty before making any decision? Any decision in a messy real world that’s far more complicated than a simple math statement that can be proved?

quotemstr 8y ago

> Would you want your car or an aircraft you're on piloted by neural net the actions of which can't be explained? Almost all cars and aircraft today are operated by neural nets, the actions of which can't be explained.

quonn 8y ago
As anyone driving a car can assure you, the actions can be explained, easily.
- olefoo 8y ago
  But are the explanations true? Are they deterministic? If someone took ten minutes of video of you driving in city traffic; could you consistently explain all of your decisions that would be recorded? It's very easy to tell yourself that your actions are consistent and explainable ( rational ) but if examined closely they aren't always consistent with any one set of heuristics. Following distance is one example; can you explain the decision making process by which you decide how much space to give the vehicle in front of you? Why it varies based on road conditions; weather or your estimate of your reaction times?

TuringNYC 8y ago

I'd also be worried about the model being biased. For example, perhaps the model is "better" because it uses correlated features it shouldnt be using (due to fairness laws, etc.)