3 ms·

I completed my PhD in the subject of improving back propagation - in 1992! Many people are startled when I tell them how old the idea of NNs are :-)

by 2sk21 8y ago

YorkshireSeason 8y ago
It's not widely known, but Alan Turing tinkered with neural nets in 1948, see [1], with his boss at the time calling Turing's work a "schoolboy essay". For a history of the subject, see Section 5.1 of [2]. Note that some parts of [2] are rather controversial. For a historical account of back-progagation, which itself was 'invented' many times, see [3, 4]. It is also not widely known that regular expressions, one of programmers' favourite tools, were introduced by Kleene in [5]: he was interested in characterising the behaviour of McCulloch-Pitts "nerve nets" (early versions of neural nets) and finite automata. [1] http://www.alanturing.net/turing_archive/pages/reference%20articles/connectionism/Turing's%20neural%20networks.html http://www.alanturing.net/turing_archive/pages/reference%20a... [2] https://arxiv.org/abs/1404.7828 https://arxiv.org/abs/1404.7828 [3] https://www.math.uni-bielefeld.de/documenta/vol-ismp/52_griewank-andreas-b.pdf https://www.math.uni-bielefeld.de/documenta/vol-ismp/52_grie... [4] http://people.idsia.ch/~juergen/who-invented-backpropagation.html http://people.idsia.ch/~juergen/who-invented-backpropagation... [5] S. Kleene, Representation of events in nerve nets and finite automata, see https://www.rand.org/content/dam/rand/pubs/research_memoranda/2008/RM704.pdf https://www.rand.org/content/dam/rand/pubs/research_memorand...
- FabHK 8y ago
  Great references (though I can't open [4]). I sometimes think the contributions of numerical analysis pioneers is undervalued (eg Wilkinson, Kahan, Golub). Backpropagation is really just reverse mode AD, and has been discovered many times, as you point out. https://en.wikipedia.org/wiki/Automatic_differentiation https://en.wikipedia.org/wiki/Automatic_differentiation EDIT: [4] works now.
s-macke 8y ago
Show them this video: https://www.youtube.com/watch?v=cNxadbrN_aI https://www.youtube.com/watch?v=cNxadbrN_aI
close04 8y ago
Too bad that so many people working in the field or ones just reading about it now assume the technology was just invented these days by Tesla or Waymo. It's about who shouts the loudest.

I completed my PhD in the subject of improving back propagation - in 1992! Many people are startled when I tell them how old the idea of NNs are :-)

by 2sk21 8y ago

I completed my PhD in the subject of improving back propagation - in 1992! Many people are startled when I tell them how old the idea of NNs are :-)

YorkshireSeason 8y ago

It's not widely known, but Alan Turing tinkered with neural nets in 1948, see [1], with his boss at the time calling Turing's work a "schoolboy essay". For a history of the subject, see Section 5.1 of [2]. Note that some parts of [2] are rather controversial. For a historical account of back-progagation, which itself was 'invented' many times, see [3, 4]. It is also not widely known that regular expressions, one of programmers' favourite tools, were introduced by Kleene in [5]: he was interested in characterising the behaviour of McCulloch-Pitts "nerve nets" (early versions of neural nets) and finite automata. [1] http://www.alanturing.net/turing_archive/pages/reference%20articles/connectionism/Turing's%20neural%20networks.html http://www.alanturing.net/turing_archive/pages/reference%20a... [2] https://arxiv.org/abs/1404.7828 https://arxiv.org/abs/1404.7828 [3] https://www.math.uni-bielefeld.de/documenta/vol-ismp/52_griewank-andreas-b.pdf https://www.math.uni-bielefeld.de/documenta/vol-ismp/52_grie... [4] http://people.idsia.ch/~juergen/who-invented-backpropagation.html http://people.idsia.ch/~juergen/who-invented-backpropagation... [5] S. Kleene, Representation of events in nerve nets and finite automata, see https://www.rand.org/content/dam/rand/pubs/research_memoranda/2008/RM704.pdf https://www.rand.org/content/dam/rand/pubs/research_memorand...

FabHK 8y ago
Great references (though I can't open [4]). I sometimes think the contributions of numerical analysis pioneers is undervalued (eg Wilkinson, Kahan, Golub). Backpropagation is really just reverse mode AD, and has been discovered many times, as you point out. https://en.wikipedia.org/wiki/Automatic_differentiation https://en.wikipedia.org/wiki/Automatic_differentiation EDIT: [4] works now.

s-macke 8y ago

Show them this video: https://www.youtube.com/watch?v=cNxadbrN_aI https://www.youtube.com/watch?v=cNxadbrN_aI

close04 8y ago

Too bad that so many people working in the field or ones just reading about it now assume the technology was just invented these days by Tesla or Waymo. It's about who shouts the loudest.