4 ms·

This is the real “safety” problem with AI. Since it’s so good at generating convincingly wrong answers, eventually we are going to build it into all sorts of mi

This is the real “safety” problem with AI. Since it’s so good at generating convincingly wrong answers, eventually we are going to build it into all sorts of mission critical systems and and everything from finance to healthcare will start to accumulate sophisticated, non deterministic failures over time. Like a trojan horse or a perhaps a civilization-level virus.

mdavidn 3y ago
It's not that dissimilar to the sophisticated, non-deterministic failures already introduce over time by humans.
- 1propionyl 3y ago
  No, it is dissimilar. Humans learn over time. Humans have social pressures. Humans have explanatory rather than merely predictive models about what they do. Deviations by humans from expected standard behavior often produce better rather than worse results. The list goes on.
  - jrockway 3y ago
    The philosophical question is whether "social pressure" and "reward function" are the same thing.
    - moqmar 3y ago
      For one person, at one point in time, sure, even though it'd be a quite complex one. But with an AI used by a thousand companies to replace one team member each, they're replacing a thousand of different "reward functions" with just one across all of them. A huge advantage of humanity is that we have different people, from different cultural and social backgrounds, all with social and personal norms and values that vary over time for each and every person, and telling an AI "imitate a pirate" or "answer without any biases" is not going to replace that.
  - deleted 3y ago
    [deleted]
- jrockway 3y ago
  It will be interesting to see. People are already involved in business-critical processes, and they have a far-from-100% success rate. You can see this with self-driving cars; 0.00001% of the time, they decide to swerve into oncoming traffic for no reason. But they can't get drunk! So whether or not that's a win is something that only time can tell. (People will be outraged, because no person would ever do what the AI did, but what matters are the numbers in aggregate. One could also argue; why should I have to put up with this when I never drink and drive? It's a good point and involves philosophical discussions about free will and all that.) I guarantee people will be mad when an AI chatbot denies their insurance or customer service claim. But I have a friend who works in customer service and it feels like half her job is correcting the mistakes other customer service agents made. If the first line support was AI, would it change much, other than not paying people to make a bunch of mistakes? Again, it's going to be painful while we try it out and get the data. It feels like it's going to be worse, and individual anecdotes will make each fuckup seem even worse than it is. But it seems inevitable that we're going to get the data. The only way to get VC writing software right now seems to be by tying your company to AI. That means a lot of experiments are about to leak out into the world. We'll learn a lot.
  - hooverd 3y ago
    That's the great part about AI, because computers don't make mistakes, you can launder whatever biases you want through an objective and rational computer.
    - Forgeties79 3y ago
      This is such an excellent way of concisely expressing a major fallacy I see whenever I talk about AI/modeling/etc. (I.e. “it isn’t racist it’s just numbers” and other nonsensical takes). I’m borrowing this language for the future.
      - sdenton4 3y ago
        Meanwhile, human judges, juries, parole boards, etc are racist as hell and have no update algorithm...
- marcosdumay 3y ago
  Humans are not expected to be infallible. Computers are. This expectation causes a lot of other problems, and will cause problems here too.
- ailef 3y ago
  That's true but humans are, or at least should be, accountable. When the AIs start making/contributing to impactful decision it will be hard to determine who to consider accountable for them (if something bad happens as a result).
- tharne 3y ago
  > It's not that dissimilar to the sophisticated, non-deterministic failures already introduce over time by humans. It's not dissimilar at all, just significantly faster and now in the hands of anyone with a laptop.
idiocrat 3y ago
Waiting for the AI judges and the AI executioners. End-to-end fully automated pipeline.
tyingq 3y ago
Or, more mundanely, automated customer support systems that just cannot handle lots of edge cases. And they'll have shut off all the "escalate to a human" paths since most of the mainstream paths work ok-ish.

mdavidn 3y ago

It's not that dissimilar to the sophisticated, non-deterministic failures already introduce over time by humans.

1propionyl 3y ago
No, it is dissimilar. Humans learn over time. Humans have social pressures. Humans have explanatory rather than merely predictive models about what they do. Deviations by humans from expected standard behavior often produce better rather than worse results. The list goes on.
- jrockway 3y ago
  The philosophical question is whether "social pressure" and "reward function" are the same thing.
  - moqmar 3y ago
    For one person, at one point in time, sure, even though it'd be a quite complex one. But with an AI used by a thousand companies to replace one team member each, they're replacing a thousand of different "reward functions" with just one across all of them. A huge advantage of humanity is that we have different people, from different cultural and social backgrounds, all with social and personal norms and values that vary over time for each and every person, and telling an AI "imitate a pirate" or "answer without any biases" is not going to replace that.
- deleted 3y ago
  [deleted]
jrockway 3y ago
It will be interesting to see. People are already involved in business-critical processes, and they have a far-from-100% success rate. You can see this with self-driving cars; 0.00001% of the time, they decide to swerve into oncoming traffic for no reason. But they can't get drunk! So whether or not that's a win is something that only time can tell. (People will be outraged, because no person would ever do what the AI did, but what matters are the numbers in aggregate. One could also argue; why should I have to put up with this when I never drink and drive? It's a good point and involves philosophical discussions about free will and all that.) I guarantee people will be mad when an AI chatbot denies their insurance or customer service claim. But I have a friend who works in customer service and it feels like half her job is correcting the mistakes other customer service agents made. If the first line support was AI, would it change much, other than not paying people to make a bunch of mistakes? Again, it's going to be painful while we try it out and get the data. It feels like it's going to be worse, and individual anecdotes will make each fuckup seem even worse than it is. But it seems inevitable that we're going to get the data. The only way to get VC writing software right now seems to be by tying your company to AI. That means a lot of experiments are about to leak out into the world. We'll learn a lot.
- hooverd 3y ago
  That's the great part about AI, because computers don't make mistakes, you can launder whatever biases you want through an objective and rational computer.
  - Forgeties79 3y ago
    This is such an excellent way of concisely expressing a major fallacy I see whenever I talk about AI/modeling/etc. (I.e. “it isn’t racist it’s just numbers” and other nonsensical takes). I’m borrowing this language for the future.
    - sdenton4 3y ago
      Meanwhile, human judges, juries, parole boards, etc are racist as hell and have no update algorithm...
marcosdumay 3y ago
Humans are not expected to be infallible. Computers are. This expectation causes a lot of other problems, and will cause problems here too.
ailef 3y ago
That's true but humans are, or at least should be, accountable. When the AIs start making/contributing to impactful decision it will be hard to determine who to consider accountable for them (if something bad happens as a result).
tharne 3y ago
> It's not that dissimilar to the sophisticated, non-deterministic failures already introduce over time by humans. It's not dissimilar at all, just significantly faster and now in the hands of anyone with a laptop.

idiocrat 3y ago

Waiting for the AI judges and the AI executioners. End-to-end fully automated pipeline.

tyingq 3y ago

Or, more mundanely, automated customer support systems that just cannot handle lots of edge cases. And they'll have shut off all the "escalate to a human" paths since most of the mainstream paths work ok-ish.