5 ms·

This is a bit lateral, but there is a parallel where Marvin Minsky will most likely be best remembered for dismissing neural networks (a 1 layer perceptron can'

This is a bit lateral, but there is a parallel where Marvin Minsky will most likely be best remembered for dismissing neural networks (a 1 layer perceptron can't even handle an xor!). We are now sufficiently removed from his heyday where I can't really recall anything he did besides the book Perceptrons with Seymour Papert (who went on to do some very interesting work in education). There is a chart out there about ML progress that makes a conjecture about how small the gap is between what we would consider that smartest and dumbest levels of human intelligence (in the grand scheme of information processing systems). It is a purely qualitative vibes sort of chart, but it is not unreasonable that even the smartest tenured professors at MIT might not be that much beyond the rest of us.

masswerk 4y ago
This dismissal of Minsky misses that Minsky had actually extensive experience with neural nets (starting in the 1950s, with neural nets in hardware) and was around 1960 probably the most experienced person in the field. Also, in Jan 1961, he published “Steps Toward Artificial Intelligence” [0], where we not only find a description of gradient descend (then "hill climbing", compare sect. B in “Steps”, as this was still measured towards a success parameter and not against an error function), but also a summary of experiences with this. (Also, the eventual reversal of success into a quantifiable error function may provide some answer to the question of success in statistical models.) [0] Minsky, Marvin, “Steps Toward Artificial Intelligence”, Proceedings of the IRE, Volume: 49, Issue: 1, Jan. 1961: https://courses.csail.mit.edu/6.803/pdf/steps.pdf https://courses.csail.mit.edu/6.803/pdf/steps.pdf
- riku_iki 4y ago
  Gradient descent was invented before Minsky. Imo, Minsky produced some vague writings, with no significant practical impact, but this is enough for some people to claim his founder's role in the field.
  - masswerk 4y ago
    Minsky was actually a pioneer in the field, when it came to working with real networks. Compare [0] “A Neural-Analogue Calculator Based upon a Probability Model of Reinforcement”, Harvard University Psychological Laboratories, Cambridge, MA, January 8, 1952 [1] “Neural Nets and the Brain Model Problem”, Princeton Ph.D dissertation, 1954 In comparison, Frank Rosenblatt's Perceptron at Cornell was only built in 1958. Notably, Minsky's SNARC (1951) was the first learning neural network.
    - riku_iki 4y ago
      > when it came to working with real networks. Compare my understanding is that that no one knows what that SNARK thing was, he built something on the grant, abandoned it shortly after that, and only many years later he and fanboys started using it as foundation of bold claims about his role in the field.
      - masswerk 4y ago
        Well, his papers are out there to read.
        riku_iki 4y ago
        Yes, and I read them: https://dspace.mit.edu/bitstream/handle/1721.1/6103/AIM-048.pdf?sequence=2&isAllowed=y https://dspace.mit.edu/bitstream/handle/1721.1/6103/AIM-048.... vague esssay without specifics
        masswerk 4y ago
        So you may like better, > “Multiple simultaneous optimizers” search for a (local) maximum value of some function E(λ1, …, λn) of several parameters. Each unit Ui independently “jitters” its parameter λ1, perhaps randomly, by adding a variation δi(t) to a current mean value μi. The changes in the quantities λi and E are correlated, and the result is used to slowly change μi. The filters are to remove DC components. This technique, a form of coherent detection, usually has an advantage over methods dealing separately and sequentially with each parameter. (In “Steps”) :-)
cma 4y ago
Take the amount of language a blind 6 year old has been exposed to. It is nothing like the scale of these corpsuses but they can develop a rich use of language. With current models if you increased parameters but gave it a similar amount of data it would overfit.
- riku_iki 4y ago
  It could be because kids are gradually and structurally trained through trials, errors and manual corrections, which we somehow don't do with NN. He wouldn't be able learn language if only exercises he would be doing is to guess next word in sentence.
aaroninsf 4y ago
For me this is a prototypical example of compounded cognitive error colliding with Dunning-Kruger. We (all of us) are very bad at non-linear reasoning, reasoning with orders of magnitude, and (by extension) have no valid intuition about emergent behaviors/properties in complex systems. In the case of scaled ML this is quite obvious in hindsight. There are many now-classic anecdotes about even those devising contemporary scale LLM being surprised and unsettled by what even their first versions were capable of. As we work away at optimizations and architectural features and expediencies which render certain classes of complex problem solving tractable by our ML, we would do well to intentionally filter for further emergent behavior. Whatever specific claims or notions any member has that may be right or wrong, the LessWrong folks are at least taking this seriously...

masswerk 4y ago

This dismissal of Minsky misses that Minsky had actually extensive experience with neural nets (starting in the 1950s, with neural nets in hardware) and was around 1960 probably the most experienced person in the field. Also, in Jan 1961, he published “Steps Toward Artificial Intelligence” [0], where we not only find a description of gradient descend (then "hill climbing", compare sect. B in “Steps”, as this was still measured towards a success parameter and not against an error function), but also a summary of experiences with this. (Also, the eventual reversal of success into a quantifiable error function may provide some answer to the question of success in statistical models.) [0] Minsky, Marvin, “Steps Toward Artificial Intelligence”, Proceedings of the IRE, Volume: 49, Issue: 1, Jan. 1961: https://courses.csail.mit.edu/6.803/pdf/steps.pdf https://courses.csail.mit.edu/6.803/pdf/steps.pdf

riku_iki 4y ago
Gradient descent was invented before Minsky. Imo, Minsky produced some vague writings, with no significant practical impact, but this is enough for some people to claim his founder's role in the field.
- masswerk 4y ago
  Minsky was actually a pioneer in the field, when it came to working with real networks. Compare [0] “A Neural-Analogue Calculator Based upon a Probability Model of Reinforcement”, Harvard University Psychological Laboratories, Cambridge, MA, January 8, 1952 [1] “Neural Nets and the Brain Model Problem”, Princeton Ph.D dissertation, 1954 In comparison, Frank Rosenblatt's Perceptron at Cornell was only built in 1958. Notably, Minsky's SNARC (1951) was the first learning neural network.
  - riku_iki 4y ago
    > when it came to working with real networks. Compare my understanding is that that no one knows what that SNARK thing was, he built something on the grant, abandoned it shortly after that, and only many years later he and fanboys started using it as foundation of bold claims about his role in the field.
    - masswerk 4y ago
      Well, his papers are out there to read.
      - riku_iki 4y ago
        Yes, and I read them: https://dspace.mit.edu/bitstream/handle/1721.1/6103/AIM-048.pdf?sequence=2&isAllowed=y https://dspace.mit.edu/bitstream/handle/1721.1/6103/AIM-048.... vague esssay without specifics
        masswerk 4y ago
        So you may like better, > “Multiple simultaneous optimizers” search for a (local) maximum value of some function E(λ1, …, λn) of several parameters. Each unit Ui independently “jitters” its parameter λ1, perhaps randomly, by adding a variation δi(t) to a current mean value μi. The changes in the quantities λi and E are correlated, and the result is used to slowly change μi. The filters are to remove DC components. This technique, a form of coherent detection, usually has an advantage over methods dealing separately and sequentially with each parameter. (In “Steps”) :-)

cma 4y ago

Take the amount of language a blind 6 year old has been exposed to. It is nothing like the scale of these corpsuses but they can develop a rich use of language. With current models if you increased parameters but gave it a similar amount of data it would overfit.

riku_iki 4y ago
It could be because kids are gradually and structurally trained through trials, errors and manual corrections, which we somehow don't do with NN. He wouldn't be able learn language if only exercises he would be doing is to guess next word in sentence.

aaroninsf 4y ago

For me this is a prototypical example of compounded cognitive error colliding with Dunning-Kruger. We (all of us) are very bad at non-linear reasoning, reasoning with orders of magnitude, and (by extension) have no valid intuition about emergent behaviors/properties in complex systems. In the case of scaled ML this is quite obvious in hindsight. There are many now-classic anecdotes about even those devising contemporary scale LLM being surprised and unsettled by what even their first versions were capable of. As we work away at optimizations and architectural features and expediencies which render certain classes of complex problem solving tractable by our ML, we would do well to intentionally filter for further emergent behavior. Whatever specific claims or notions any member has that may be right or wrong, the LessWrong folks are at least taking this seriously...