2 ms·

Reading this article, I kept think about James Micken's 2018 usenix security talk https://www.youtube.com/watch?v=ajGX7odA87k https://www.youtube.com/watch?v=aj

by norir 3y ago

Reading this article, I kept think about James Micken's 2018 usenix security talk https://www.youtube.com/watch?v=ajGX7odA87k https://www.youtube.com/watch?v=ajGX7odA87k. He warns quite effectively against widely deploying technology that we don't understand. Kevin Scott, on the other hand, seems to have exactly the opposite take. I suppose there is a reason though why one of them is a successful businessman and the other is an academic.

dekhn 3y ago
I don't think ML is substantially different from other technologies that we have deployed that we didn't completely understand: early atomic weapons, early molecular biology, cars, and more. In each case, by deploying a bit with caution, we learned important lessons that allowed us to continue. Rather than sticking our head in the ground and remaining ignorant. In each case there were people calling for the deployments to stop. Of all those, I think only atomic weapons ever represented an existential risk to humanity- and we figured that out within a decade or to and set up political structures to prevent it (although, ideally, we'd have worldwide disarmment). Unless somebody can make some sort of extremely convincing argument that by deploying this tech we are taking an existential risk, rather than purely banal ones, it doesn't seem reasonble to prevent deployment.
- ethanbond 3y ago
  What? All of those inventions you listed were mechanistically understood quite well. Atomic weapons wouldn’t have been possible without a completely sound understanding of their mechanics. A better example is perhaps pharmaceuticals, many of which we do not actually understand the mechanisms of action. But note those go through extremely rigorous testing for exactly that reason.
  - dekhn 3y ago
    The inventions were partly understood mechanistically- just like ML. What we didn't understand was the consequences of deployment (such as massive radiation leaks due to testing, unpredictable yields, the risks of global thermonuclear war, and the ongoing cost of maintaining a fleet of nuclear weapons). Similar issues with molecular biology- we knew how to clone a gene from one organism to another, but that doesn't mean we really knew all the things going on (side effects) or what the large-scale implications are (hence the Asilomar agreement). Even cars- while engines were understood mechanistically, it took quite some time for people to appreciate why automobile safety glass was necessary. See the experiences learned during testing nuclear weapons- we tested them because we didn't understand them mechanistcally, at least not fully enough to predict many effects.
    - comp_throw7 3y ago
      > The inventions were partly understood mechanistically- just like ML We have effectively no mechanistic understanding of frontier ML systems, in the sense that we have no idea how they do what they do and could not e.g. write human-readable code to perform comparable tasks, nor can we predict ahead of time what capabilities such systems will have when they're trained (being able to predict e.g. log-loss is _not_ the same as being able to predict specific capabilities).
  - Tarq0n 3y ago
    Maybe GMO crops are a better analogy. We understand them well enough, but if we mess up at making them unable to reproduce they could easily proliferate beyond our control.

by norir 3y ago

dekhn 3y ago

I don't think ML is substantially different from other technologies that we have deployed that we didn't completely understand: early atomic weapons, early molecular biology, cars, and more. In each case, by deploying a bit with caution, we learned important lessons that allowed us to continue. Rather than sticking our head in the ground and remaining ignorant. In each case there were people calling for the deployments to stop. Of all those, I think only atomic weapons ever represented an existential risk to humanity- and we figured that out within a decade or to and set up political structures to prevent it (although, ideally, we'd have worldwide disarmment). Unless somebody can make some sort of extremely convincing argument that by deploying this tech we are taking an existential risk, rather than purely banal ones, it doesn't seem reasonble to prevent deployment.

ethanbond 3y ago
What? All of those inventions you listed were mechanistically understood quite well. Atomic weapons wouldn’t have been possible without a completely sound understanding of their mechanics. A better example is perhaps pharmaceuticals, many of which we do not actually understand the mechanisms of action. But note those go through extremely rigorous testing for exactly that reason.
- dekhn 3y ago
  The inventions were partly understood mechanistically- just like ML. What we didn't understand was the consequences of deployment (such as massive radiation leaks due to testing, unpredictable yields, the risks of global thermonuclear war, and the ongoing cost of maintaining a fleet of nuclear weapons). Similar issues with molecular biology- we knew how to clone a gene from one organism to another, but that doesn't mean we really knew all the things going on (side effects) or what the large-scale implications are (hence the Asilomar agreement). Even cars- while engines were understood mechanistically, it took quite some time for people to appreciate why automobile safety glass was necessary. See the experiences learned during testing nuclear weapons- we tested them because we didn't understand them mechanistcally, at least not fully enough to predict many effects.
  - comp_throw7 3y ago
    > The inventions were partly understood mechanistically- just like ML We have effectively no mechanistic understanding of frontier ML systems, in the sense that we have no idea how they do what they do and could not e.g. write human-readable code to perform comparable tasks, nor can we predict ahead of time what capabilities such systems will have when they're trained (being able to predict e.g. log-loss is _not_ the same as being able to predict specific capabilities).
- Tarq0n 3y ago
  Maybe GMO crops are a better analogy. We understand them well enough, but if we mess up at making them unable to reproduce they could easily proliferate beyond our control.