4 ms·

TLDR: A faulty loss function leads to faulty results. This article has wonderful real-world examples of AIs gaming the mathematical problem specification creat

TLDR: A faulty loss function leads to faulty results. This article has wonderful real-world examples of AIs gaming the mathematical problem specification created by the AI researcher in order to both produce a world-class score on a given dataset and be utterly unusable in the real world. While it doesn't appear to be mentioned in the article, this is exactly why I believe that AI will not have a democratizing effect. Either you have the mathematical and stochastic skills to describe your problem criteria as a differentiable function, or not. In the first case, you're already a good programmer, but AI might save you a bit of time over trying to find the optimal parameters manually. In the second case, you will not be able to ensure that the AI follows the implicit rules of the real world, which means that you will produce something that scores extremely well on its training data, but does not generalize to the real world. And that's also why if someone claims a new "state of the art result on benchmark XY", you should mentally append " ... but it works nowhere else."

spinningslate 7y ago
> TLDR: A faulty loss function leads to faulty results. This. The use of "gaming" to describe the observed behaviour seems strange. To "game" suggests some malfeasance in the algorithm; a deliberate effort to subvert the intended goal, which in turn implies knowing what the "real" goal was - and somehow deciding to shortcut it. That isn't the case. Rather, the problem lies in appropriately specifying the desired outcome, and any relevant constraints that must be obeyed. Indeed, DeepMind has separately written about the "specification problem" elsewhere [0]. That's in no way to downplay the difficulty in specification. It's one we've wrestled with in software for decades. > Either you have the mathematical and stochastic skills to describe your problem criteria as a differentiable function, or not. Indeed. That statement could equally have been written ~30 years ago when Z, VDM and the likes were active areas of research. More recently, TLA+ has gotten some traction, but it's still very much in the margins. Perhaps ML will spur the next wave of specification formalisms. But it will face the same challenge as its pre-decessors: how to make it accessible to more than a very (very) small set of sufficiently capable few. [0] https://medium.com/@deepmindsafetyresearch/building-safe-artificial-intelligence-52f5f75058f1 https://medium.com/@deepmindsafetyresearch/building-safe-art...
- perl4ever 7y ago
  It seems to me that saying the problem with AI is specifying the problem to be solved is admitting that you haven't even started to develop AI. We could be just around the corner, but the fact that people are taking for granted that there's no intelligence in AI isn't promising. If you respond defensively about how difficult it is to give people what they want, well, that's the whole point.
  - fxtentacle 7y ago
    I see myself more as the observer here. I see that there's plenty of publications of people claiming to exceed the "state of the art", as well as plenty of people claiming to have found the holy grail. But if you use their resulting AIs with real-world data, they usually fail. For example, people were willing to declare optical flow "solved" already 5 years ago. Yet, autonomous drones still crash into wires, glass, trees, snow, water, etc., because their optical flow algorithms cannot handle these aspects of reality (which were not present in the AI training data).
- SubiculumCode 7y ago
  The article addresses your point exactly.
auggierose 7y ago
"Might save you a bit of time" is too underwhelmingly stated though. With Alpha Go it was a game changer, literally. Of course, in that case the specification was given by the rules of the game, so not a problem.
dexen 7y ago
>AI might be useless because it finds loopholes AI might be best put to use for finding loopholes. In formal specifications, in legal documents, in laws... you name it. The required process change is reasonably straightforward; instead of presuming an answer given by AI is useful as a solution to the given problem, take it as an indicator of a loophole in problem specification and see if one can be closed up.
- paganel 7y ago
  I'm no legal expert by any means but it is my understanding that laws are supposed to have loopholes almost by design. A legal system with no loopholes which probably also means with no interpretation will be as close to dystopia as we could ever get.
  - pas 7y ago
    Legislative intent shouldn't be gameable, just as constitutional shouldn't be either. Courts are of course have a certain degree of interpretive freedom, but the process should result in a machinery that implements intent faithfully, and legislators change laws to change intent (and to patch holes, but in common law that usually happens through the courts too).
    - paganel 7y ago
      No law has ever been perfect (and they'll never be), and as such "gaming" them is the only recourse for lots of people negatively affected by said imperfect laws until those legislators you mention happen to change them (for the better, it is hoped). Otherwise we'll be back to living in one of Kafka's stories.
      - pas 7y ago
        But that doesn't make sense. Laws don't have loopholes in them for the off chance that it turns out to be a bad law. Legislation, laws, and law enforcement as technologies have always had these trade offs, and they have progressed tremendously over the centuries (or millennias). From the Roman veto of the tribune of the plebs, to current constitutional review there are many checks and balances. But legislation as a process doesn't have inherent loopholes. Currently for every piece of law proposed many official and civil groups warn of the potential unintended side effects of it. But then the majority usually disregards those concerns.

spinningslate 7y ago

> TLDR: A faulty loss function leads to faulty results. This. The use of "gaming" to describe the observed behaviour seems strange. To "game" suggests some malfeasance in the algorithm; a deliberate effort to subvert the intended goal, which in turn implies knowing what the "real" goal was - and somehow deciding to shortcut it. That isn't the case. Rather, the problem lies in appropriately specifying the desired outcome, and any relevant constraints that must be obeyed. Indeed, DeepMind has separately written about the "specification problem" elsewhere [0]. That's in no way to downplay the difficulty in specification. It's one we've wrestled with in software for decades. > Either you have the mathematical and stochastic skills to describe your problem criteria as a differentiable function, or not. Indeed. That statement could equally have been written ~30 years ago when Z, VDM and the likes were active areas of research. More recently, TLA+ has gotten some traction, but it's still very much in the margins. Perhaps ML will spur the next wave of specification formalisms. But it will face the same challenge as its pre-decessors: how to make it accessible to more than a very (very) small set of sufficiently capable few. [0] https://medium.com/@deepmindsafetyresearch/building-safe-artificial-intelligence-52f5f75058f1 https://medium.com/@deepmindsafetyresearch/building-safe-art...

perl4ever 7y ago
It seems to me that saying the problem with AI is specifying the problem to be solved is admitting that you haven't even started to develop AI. We could be just around the corner, but the fact that people are taking for granted that there's no intelligence in AI isn't promising. If you respond defensively about how difficult it is to give people what they want, well, that's the whole point.
- fxtentacle 7y ago
  I see myself more as the observer here. I see that there's plenty of publications of people claiming to exceed the "state of the art", as well as plenty of people claiming to have found the holy grail. But if you use their resulting AIs with real-world data, they usually fail. For example, people were willing to declare optical flow "solved" already 5 years ago. Yet, autonomous drones still crash into wires, glass, trees, snow, water, etc., because their optical flow algorithms cannot handle these aspects of reality (which were not present in the AI training data).
SubiculumCode 7y ago
The article addresses your point exactly.

auggierose 7y ago

"Might save you a bit of time" is too underwhelmingly stated though. With Alpha Go it was a game changer, literally. Of course, in that case the specification was given by the rules of the game, so not a problem.

dexen 7y ago

>AI might be useless because it finds loopholes AI might be best put to use for finding loopholes. In formal specifications, in legal documents, in laws... you name it. The required process change is reasonably straightforward; instead of presuming an answer given by AI is useful as a solution to the given problem, take it as an indicator of a loophole in problem specification and see if one can be closed up.

paganel 7y ago
I'm no legal expert by any means but it is my understanding that laws are supposed to have loopholes almost by design. A legal system with no loopholes which probably also means with no interpretation will be as close to dystopia as we could ever get.
- pas 7y ago
  Legislative intent shouldn't be gameable, just as constitutional shouldn't be either. Courts are of course have a certain degree of interpretive freedom, but the process should result in a machinery that implements intent faithfully, and legislators change laws to change intent (and to patch holes, but in common law that usually happens through the courts too).
  - paganel 7y ago
    No law has ever been perfect (and they'll never be), and as such "gaming" them is the only recourse for lots of people negatively affected by said imperfect laws until those legislators you mention happen to change them (for the better, it is hoped). Otherwise we'll be back to living in one of Kafka's stories.
    - pas 7y ago
      But that doesn't make sense. Laws don't have loopholes in them for the off chance that it turns out to be a bad law. Legislation, laws, and law enforcement as technologies have always had these trade offs, and they have progressed tremendously over the centuries (or millennias). From the Roman veto of the tribune of the plebs, to current constitutional review there are many checks and balances. But legislation as a process doesn't have inherent loopholes. Currently for every piece of law proposed many official and civil groups warn of the potential unintended side effects of it. But then the majority usually disregards those concerns.