5 ms·

I call this the "babysitting problem." If a model is right 99.99% of the time (which nobody has come close to), we still need something that understands what i

I call this the "babysitting problem." If a model is right 99.99% of the time (which nobody has come close to), we still need something that understands what it's doing enough to observe and catch that 0.01% where it's wrong. Because wrong at that level is often dangerously wrong. This is explored (in an earlier context) in the 1983 paper "Ironies of Automation". https://en.wikipedia.org/wiki/Ironies_of_Automation https://en.wikipedia.org/wiki/Ironies_of_Automation

adrianN 2y ago
I’m pretty sure humans make mistakes too and it happens rather frequently that nobody catches them until it’s too late. In most fields we’re okay with that because perfection is prohibitively expensive.
- emseetech 2y ago
  Obviously systems have always had to be resilient. But the point here is how dangerous a "set it and forget it" AI can be. Because the mistakes it makes, although fewer, are much more dangerous, unpredictable, and inscrutable than the mistakes a human would make. Which means the people who catch these mistakes have to be operating at a very high level. This means we need to resist getting lulled into a false sense of security with these systems, and we need to make sure we can still get people to a high level of experience and education.
Animats 2y ago
> we still need something that understands what it's doing enough to observe and catch that 0.01% where it's wrong. Nobody has figured out how to get a confidence metric out of the innards of a neural net. This is why chatbots seldom say "I don't know", but, instead, hallucinate something plausible. Most of the attempts to fix this are hacks outside the LLM. Run several copies and compare. Ask for citations and check them. Throw in more training data. Punish for wrong answers. None of those hacks work very well. The black box part is still not understood. This is the elephant in the room of LLMs. If someone doesn't crack this soon, AI Winter #3 will begin. There's a lot of startup valuation which assumes this problem gets solved.
- JohnMakin 2y ago
  > There's a lot of startup valuation which assumes this problem gets solved. Not just solved, but solved soon. I think this is an extremely difficult problem to solve to the point it'd involve new aspects of computer science to even approach correctly, but we seem to just think throwing more CPU and $$$ at the problem will work itself out. I myself am skeptical.
  - Animats 2y ago
    Is there any progress? About two years ago, there were people training neural nets to play games, looking for a representation of the game state inside the net, and claiming to find it. That doesn't seem to be mentioned any more. As for "solved soon", the market can remain irrational longer than you can stay solvent. Look at Uber and Tesla, both counting on some kind of miracle to justify their market cap.
    - marcosdumay 2y ago
      Tesla was mildly successful right until ite CEO satrted to fight its customers. It's unclear if this will revert. Uber seems to have become sustainable thid year. There's little reason to expect a correction any soon on any of those.
    - sdenton4 2y ago
      I get the impression that most of the 'understand the innards' work isn't scalable - you build out a careful experiment with a specific network, but the work doesn't transfer to new models, fine-tuned models, etc. I'm just an outside observer, though...

adrianN 2y ago

I’m pretty sure humans make mistakes too and it happens rather frequently that nobody catches them until it’s too late. In most fields we’re okay with that because perfection is prohibitively expensive.

emseetech 2y ago
Obviously systems have always had to be resilient. But the point here is how dangerous a "set it and forget it" AI can be. Because the mistakes it makes, although fewer, are much more dangerous, unpredictable, and inscrutable than the mistakes a human would make. Which means the people who catch these mistakes have to be operating at a very high level. This means we need to resist getting lulled into a false sense of security with these systems, and we need to make sure we can still get people to a high level of experience and education.

Animats 2y ago

> we still need something that understands what it's doing enough to observe and catch that 0.01% where it's wrong. Nobody has figured out how to get a confidence metric out of the innards of a neural net. This is why chatbots seldom say "I don't know", but, instead, hallucinate something plausible. Most of the attempts to fix this are hacks outside the LLM. Run several copies and compare. Ask for citations and check them. Throw in more training data. Punish for wrong answers. None of those hacks work very well. The black box part is still not understood. This is the elephant in the room of LLMs. If someone doesn't crack this soon, AI Winter #3 will begin. There's a lot of startup valuation which assumes this problem gets solved.

JohnMakin 2y ago
> There's a lot of startup valuation which assumes this problem gets solved. Not just solved, but solved soon. I think this is an extremely difficult problem to solve to the point it'd involve new aspects of computer science to even approach correctly, but we seem to just think throwing more CPU and $$$ at the problem will work itself out. I myself am skeptical.
- Animats 2y ago
  Is there any progress? About two years ago, there were people training neural nets to play games, looking for a representation of the game state inside the net, and claiming to find it. That doesn't seem to be mentioned any more. As for "solved soon", the market can remain irrational longer than you can stay solvent. Look at Uber and Tesla, both counting on some kind of miracle to justify their market cap.
  - marcosdumay 2y ago
    Tesla was mildly successful right until ite CEO satrted to fight its customers. It's unclear if this will revert. Uber seems to have become sustainable thid year. There's little reason to expect a correction any soon on any of those.
  - sdenton4 2y ago
    I get the impression that most of the 'understand the innards' work isn't scalable - you build out a careful experiment with a specific network, but the work doesn't transfer to new models, fine-tuned models, etc. I'm just an outside observer, though...