4 ms·

Neural networks do achieve impressive things but they also fail to achieve essential things that preclude them from an AGI or Causal NLU ambition, such as the i

Neural networks do achieve impressive things but they also fail to achieve essential things that preclude them from an AGI or Causal NLU ambition, such as the inability to approximate a dumb calculator without significant accuracy loss.

Filligree 4y ago
I can’t approximate a dumb calculator without significant accuracy loss. Not without emulating symbolic computation, which current AI is perfectly capable of doing if you ask it the right way. Whatever makes you think it’s necessary for AGI, when we don’t have it?
- SemanticStrengh 4y ago
  NNs fails to do any algorithmy like pathfinding, sorting, etc The point is not that you have it it's that you can have it by learning and using a pen and paper. Natural language understanding require both neural network like pattern recognition abilities and advanced algorithmic calculations. Since neural networks are pathetically bad at algorithmy, we need neuro-symbolic software. However the symbolic part is rigid and program synthesis is exponential. Therefore the brain is the only technology on earth to be able to dynamically code algorithmic solutions. Neural networks have only solved a subset of the class of automated programs.
  - visarga 4y ago
    There are about 3,610 results for "neural network pathfinding" in Google Scholar since 2021. Try a search.
    - SemanticStrengh 4y ago
      And as you can trivially see it is outputting nonsense values https://www.lovebirb.com/Projects/ANN-Pathfinder?pgid=kqe249l8-29dac0_9ad1dbf1bd2542148cd647a087e61989mv2 https://www.lovebirb.com/Projects/ANN-Pathfinder?pgid=kqe249... (see last slide) At least in this implementation Even if it had 80% accuracy (optimistic) it would still he too mediocre to be used at any serious scale.
visarga 4y ago
It's a model mismatch, not an inherent impossibility. A calculator needs to have an adaptive number of intermediate steps. Usually our models have fixed depth, but in auto-regressive modelling the tape can become longer as needed by the stepwise algorithm. Recent models show LMs can do arithmetic, symbolic math and common sense chain-of-thought step by step reasoning and reach much higher accuracies. In other words, we too can't do three digit multiplication in our heads reliably, but can do it much better on paper, step by step. The problem you were mentioning is caused by the bad approach - LMs need intermediate reasoning steps to get from problem to solution, like us. We just need to ask them to produce the whole reasoning chain. - Chain of Thought Prompting Elicits Reasoning in Large Language Models https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903 - Deep Learning for Symbolic Mathematics https://arxiv.org/abs/1912.01412 https://arxiv.org/abs/1912.01412

Filligree 4y ago

I can’t approximate a dumb calculator without significant accuracy loss. Not without emulating symbolic computation, which current AI is perfectly capable of doing if you ask it the right way. Whatever makes you think it’s necessary for AGI, when we don’t have it?

SemanticStrengh 4y ago
NNs fails to do any algorithmy like pathfinding, sorting, etc The point is not that you have it it's that you can have it by learning and using a pen and paper. Natural language understanding require both neural network like pattern recognition abilities and advanced algorithmic calculations. Since neural networks are pathetically bad at algorithmy, we need neuro-symbolic software. However the symbolic part is rigid and program synthesis is exponential. Therefore the brain is the only technology on earth to be able to dynamically code algorithmic solutions. Neural networks have only solved a subset of the class of automated programs.
- visarga 4y ago
  There are about 3,610 results for "neural network pathfinding" in Google Scholar since 2021. Try a search.
  - SemanticStrengh 4y ago
    And as you can trivially see it is outputting nonsense values https://www.lovebirb.com/Projects/ANN-Pathfinder?pgid=kqe249l8-29dac0_9ad1dbf1bd2542148cd647a087e61989mv2 https://www.lovebirb.com/Projects/ANN-Pathfinder?pgid=kqe249... (see last slide) At least in this implementation Even if it had 80% accuracy (optimistic) it would still he too mediocre to be used at any serious scale.

visarga 4y ago

It's a model mismatch, not an inherent impossibility. A calculator needs to have an adaptive number of intermediate steps. Usually our models have fixed depth, but in auto-regressive modelling the tape can become longer as needed by the stepwise algorithm. Recent models show LMs can do arithmetic, symbolic math and common sense chain-of-thought step by step reasoning and reach much higher accuracies. In other words, we too can't do three digit multiplication in our heads reliably, but can do it much better on paper, step by step. The problem you were mentioning is caused by the bad approach - LMs need intermediate reasoning steps to get from problem to solution, like us. We just need to ask them to produce the whole reasoning chain. - Chain of Thought Prompting Elicits Reasoning in Large Language Models https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903 - Deep Learning for Symbolic Mathematics https://arxiv.org/abs/1912.01412 https://arxiv.org/abs/1912.01412