3 ms·

The article is of course right but also a bit silly. Language models like GPT-X are producing grammatically correct sentences, along the lines of "Colorless gre

The article is of course right but also a bit silly. Language models like GPT-X are producing grammatically correct sentences, along the lines of "Colorless green ideas sleep furiously". The NLP research more or less solved the old syntax problem using 'distributional semantics' but 'semantics' is a misnomer, it's all about syntax. In fact the most useful part of the article for me is that they mentioned Douglas Summers-Stay, who does some interesting work on 'common sense' engineering, combining syntax engines like GPT-3 with knowledge graphs. https://sci-hub.tw/https://www.sciencedirect.com/science/article/abs/pii/S2212683X16300160 https://sci-hub.tw/https://www.sciencedirect.com/science/art... My bet is that actual AI will come from combination of these statistics-driven syntax generators with graphical causality models. Treating syntax as a kind of lower level substrate, akin to sensory modalities in vision, with intelligence model as a directed causal graph linking concepts at different levels of abstraction/chunking. As a side note it’s funny that people working on artificial intelligence at OpenAI and elsewhere are mostly computer scientists, not cognitive psychologists or neuroscientists who might actually have a clue how intelligence works. This probably explains the proliferation of ‘backpropagation’ as primary method of artificial learning. These people are just naturally good at calculus in high school, so it’s a hammer that found its proverbial nail.

Der_Einzige 6y ago
Gradient free optimization is not used much in Neural Networks except in Reinforcement Learning. I think it's because backprop is objectively faster for most supervised problems than other techniques (e.g. simulated annealing or GAs)
- bra-ket 6y ago
  The thing is, living beings don’t learn by brute-force trial and error like mathematical optimization models you mentioned. Besides enormous energy spent, an individual organism will just be eaten by predator on another iteration of its ‘error minimization’ loop. the idea of learning via reinforcement, that came from Skinner behaviorist experiments has been long discredited in cognitive psychology. (I highly recommend Wayne Wickelgren’ work on learning and memory if you’re interested, it’s brilliant and concise http://www.columbia.edu/~nvg1/Wickelgren/ http://www.columbia.edu/~nvg1/Wickelgren/ ) Biological plausibility might not be needed for recognizing check signatures or images of traffic lights, where backprop is working just fine, but I believe true cognition would require such energy expenditures that brute-force trial and error will never be feasible. Moreover such error correction imposes artificial constraints that limit the amount of information that can be learned, kind of like those mechanical calculators of the 17th century with gears and wheels and crude mechanical actuators.
visarga 6y ago
Yes, how dare these machine learning scientists use backprop? I mean, there are better methods our there, right?
- deleted 6y ago
  [deleted]

Der_Einzige 6y ago

Gradient free optimization is not used much in Neural Networks except in Reinforcement Learning. I think it's because backprop is objectively faster for most supervised problems than other techniques (e.g. simulated annealing or GAs)

bra-ket 6y ago
The thing is, living beings don’t learn by brute-force trial and error like mathematical optimization models you mentioned. Besides enormous energy spent, an individual organism will just be eaten by predator on another iteration of its ‘error minimization’ loop. the idea of learning via reinforcement, that came from Skinner behaviorist experiments has been long discredited in cognitive psychology. (I highly recommend Wayne Wickelgren’ work on learning and memory if you’re interested, it’s brilliant and concise http://www.columbia.edu/~nvg1/Wickelgren/ http://www.columbia.edu/~nvg1/Wickelgren/ ) Biological plausibility might not be needed for recognizing check signatures or images of traffic lights, where backprop is working just fine, but I believe true cognition would require such energy expenditures that brute-force trial and error will never be feasible. Moreover such error correction imposes artificial constraints that limit the amount of information that can be learned, kind of like those mechanical calculators of the 17th century with gears and wheels and crude mechanical actuators.

visarga 6y ago

Yes, how dare these machine learning scientists use backprop? I mean, there are better methods our there, right?

deleted 6y ago
[deleted]