5 ms·
Munich 1991: The Roots of the Current AI Boom
- sgammon 4mo agosuccess has many fathers
- jacknews 4mo agoSurely the roots, if we skip over the early preceptron work', are in backpropagation and Hinton, and the work going on at Edinburgh and elsewhere in the 80s. Indeed I remember buying a set of three conference-papers-as-books around that time, titled Artificial Neural Networks .. proceedings of the whatever the conference was. No doubt Schmidhuber made important contributions, but I see him pop up claiming to be the 'root' of it all every couple of years.
- h8hawk 4mo agoHinton did not invent backpropagation. related paragraph from Wikipedia: Modern backpropagation was first published by Seppo Linnainmaa as "reverse mode of automatic differentiation" (1970)[26] for discrete connected networks of nested differentiable functions.[27][28][29] In 1982, Paul Werbos applied backpropagation to MLPs in the way that has become standard.
- ogrisel 4mo agoPaul Werbos did not apply backprop to MLPs as cleanly described in Hinton's paper, but rather to some kind of autoregressive non-linear parametrized functions with a much more specific application scope. Both papers are direct applications of the chain rule applied to estimate the gradient of a multivariate function.
- hyttioaoa 4mo agoThat's what bugs me about him. So much work has gone into today's models that calling his contributions "the root" isn't really warranted. He's always complaining that Hinton, LeCun, and Bengio get more credit than they deserve, and now he's over-claiming himself.
- BoredPositron 4mo agoBoth can be right.
- HarHarVeryFunny 4mo agoThey could be, but they really aren't. Name a single aspect of something modern like the Transformer architecture or how it is trained, that is even indirectly attributable to Schmidhuber. No doubt he'd be jumping up and down wanting to take credit for residual connections, but where was Schmidhuber in the ImageNet era when everyone else was discovering how to build deep neural nets? Why didn't Schmidhuber invent ResNets, but instead waited until someone else (Kaiming He) did, then claim credit for it? I'll bet Schmidhuber isn't done with yet ... when someone eventually comes up with an architecture for AGI, Schmidhuber will come out of the woodwork and point to a note he made on a napkin in 1800 that predicted it all.
- emil-lp 4mo agoSurely the roots go back to Turing, Gödel, Hilbert, Frege, Leibniz, Aristoteles.
- jcattle 4mo agoThere's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc. And while it is very true that often the research coming out of Academia is useless, what is always neglected are the roots of the research done in private labs. When Jürgen Schmidhuber and team published their work on Neural Nets back in 1991 it was also useless. Unless you had a supercomputer and very, very deep pockets you were not going to do anything with what came out of their lab. But still, 30 years later here we are, standing on top of the shoulders of this useless research.
- yorwba 4mo agoLike half of what Schmidhuber is always complaining about is that (except for LSTMs) people aren't standing on the shoulders of his research very much. They try to solve some of the same problems people have always wanted to solve, try some of the same approaches people always tend to try, and then tinker until it works. At no point do they consult Schmidhuber's decade-old papers where he tried something kind of similar but didn't get very impressive results, and hence they also do not think to cite him. Then he comes out of the woodwork to assert priority.
- suddenlybananas 4mo agoYou can be influenced downstream by papers you haven't personally read.
- bonzini 4mo agoShane Legg was in Schmidhuber's lab at IDSIA before being one of the founders of DeepMind, so he probably read the papers personally and knows what influenced him or not...
- gillesjacobs 4mo agoOf course, but if you haven't read them you also shouldn't cite them. And that's where Schmidhuber goes off the rails: publicly shaming published papers into citing you isn't good academic practice. It's bullying.
- emmelaich 4mo agohttps://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber https://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber
- practal 4mo agoTU Munich and Nipkow, Makarius et.al. are also at the center of the influential Isabelle theorem prover. TU Munich is cool :-)
- MeteorMarc 4mo agoAlso see Schmidhuber's take on the Hinton + Hopfield Nobel prize: https://people.idsia.ch/~juergen/physics-nobel-2024-plagiarism.html https://people.idsia.ch/~juergen/physics-nobel-2024-plagiari...
- h8hawk 4mo agoIt's sad that he is the only one speaking out about Hinton. This whole Hinton glorification seems like it's being pushed by an agenda. I'm not sure if he would receive this much attention if he held a different view (closer to LeCun or Ng), rather than these Effective Altruism takes on current AI.
- ks2048 4mo agoI don't associate Hinton will Effective Altruism. He did switch to focus on warning of the dangers of AI, but that was after he already was established as the father (or one of them) of deep learning.
- Hoasi 4mo agoNot that surprising since the whole LLM ecosystem is based on plagiarism.
- letssaythat 4mo ago[flagged]
- snowpid 4mo agowell, so you think, all parts and peoples of USSR were voluntarily part of USSR?
- letssaythat 4mo agoNo, not all, though you will be surprised how many convinced communists were even in Lithuania, to pick one example (a fact which today's Lithuania hides). And do you think the people in Europe are voluntarily a part of your undeclared, but very much existing, empire?
- sagex 4mo agoI believe invention of Transformers and especially Attention mechanism do have influence from past research but its not definitely only the Schmidhuber's work. Said that, if we remove the papers mentioned by Schmidhuber from history, I am quite certain that there will be no influence in the discovery of Transformers, hence his works can not be the root. He has to grow up and accept that work and equations can appear similar, looking at inverse squared law and saying Newton stole that from someone is being dishonest.
- gillesjacobs 4mo agoWhich work has more value: the abstract description of a catalogue of potential model architectures or their validated application trained on real data? In the Schmidhuber case their is 20 years and a chain of countless other works in between the two.
- trashburger 4mo agoThis article, too, was originally discovered by Jürgen Schmidhuber in 1991!
- jongjong 4mo agoIt's crazy to think that if Elon Musk hadn't mentioned Schmidhuber, most people would have no idea. It's nauseating how all the researchers who happened to work for big tech got tons of media coverage but Schmidhuber and his team were getting zero coverage yet they made massive contributions. I bet there are many others not mentioned. Nobody even knows about Frank Rosenblatt. It's insane how distorted our perception of innovation is. Even science has been corrupted. It makes one doubt every story we're told about who invented what.
- gom_jabbar 4mo agoYes, Rosenblatt is another good example. I recently looked deeper into the development of the perceptron and it's absolutely fascinating.
- ks2048 4mo ago> Nobody even knows about Frank Rosenblatt. Very Trump-like statement - "Not many people know this, but ...". Yes, I lot of people know this. Any class that even says a little about the history of NNs will talk about Rosenblatt and the Perceptron.
- gom_jabbar 4mo ago> Any class that even says a little about the history of NNs will talk about Rosenblatt and the Perceptron. Sure. I think it starts to get more interesting when the influences that Rosenblatt explicitly cites in his seminal Perceptron paper (e.g. Hayek) become part of the discussion (which rarely happens in my experience).
- storus 4mo agoInstead of focusing on the future, EU is busy rewriting history to please some eccentric researcher that claims he invented it all.
- impossiblefork 4mo agoSchmidhuber isn't in the EU, nor Switzerland at the moment.
- greggoB 4mo agoHow does the EU feature in TFA exactly?
- storus 4mo agoThere seems to be a coordinated push around Schmidhuber all around media in the EU, even LinkedIn is full of "random" posts about him in the past week.
- logicchains 4mo agoYou clearly aren't familiar with Schmidhuber if this kind of thing seems new to you. It's basically his thing.
- HarHarVeryFunny 4mo agoThe current AI boom has more to do with NVIDIA, and the popularity of computer gaming giving us GPU compute, than who was using neural networks back in 1990's. More specifically, it was really AlexNet, the 2012 ImageNet entry, running on two NVIDIA GTX 580's, that highlighted the practicality and utility of running large scale neural nets on affordable hardware. CUDA had been released in 2006, but cuDNN (the CUDA library for neural nets) didn't come out until 2014 - after AlexNet had already kickstarted the demand. What followed from AlexNet was a few years of intense competition on the ImageNet benchmark, and larger and larger/deeper neural nets (CNNs), which gave rise to a lot of the algorithms and concepts still used today such as residual connections (originally from ResNet), ADAM (training algorithm), ReLU/etc, normalization, dropout, etc... all the fundamentals that made building large neural nets possible. Schmidhuber's continual reminding everyone that he was working on neural nets back in the 1990s is beyond tiresome. Yes, he should have been recognized alongside Hinton/Bengio/LeCun as one of the pioneers, but time for him to get over it.
- LogicFailsMe 4mo agoAnd Google's acquisition of DNN Research to get the ball rolling with conv nets and AI moneyball, followed by the acquisition of Deepmind. Schmidhuber IMO *has* been recognized as one of the 4 horseman and rightly so, but what has he done lately? Just noticed they now say the 3 godfathers of AI. This is what people hate about academia. It's not academia itself, it's the mean girl politics that emerge from the tenure system. And at this point, tenure should be abolished IMO having been utterly weaponized to defend the status quo.
- AndrewKemendo 4mo agoThis is well put. 2012 really fundamentally changed everything for the AI community, I’d argue because tensorflow/keras/pytorch followed and that made the infrastructure accessible for distributed training.
- alephnerd 4mo ago> The current AI boom has more to do with NVIDIA, and the popularity of computer gaming giving us GPU compute, than who was using neural networks back in 1990's I disagree. But more critically, I'd argue it's the legacy of the PDP project that led to what became foundation models today.
- cold_harbor 4mo agoworth separating: LSTM (Hochreiter & Schmidhuber 1997) is ironclad and widely cited. the transformer attention priority claims are far shakier. conflating them is how Schmidhuber undermines himself
- HarHarVeryFunny 4mo agoYes, and notable how Alex Graves, one of Schmidhuber's students, later at DeepMind, doesn't even mention Schmidhuber in his historical overview of attention mechanisms "Attention and Memory in Deep Learning". https://www.youtube.com/watch?v=AIiwuClvH6k https://www.youtube.com/watch?v=AIiwuClvH6k When it comes to attention, details matter, since the idea itself is obvious - weighted inputs, and implicit attention is present in every neural network - this is what weights are. The specific form of attention used by the Transformer is key-based associative attention, aka "Bahdanau attention" introduced in Bahdanau's paper "Neural Machine Translation by Jointly Learning to Align and Translate". It's perhaps worth noting that the word "attention" is barely even mentioned in this paper, other than noting that this weighted input mechanism can be seen as a form of attention (presumably mentioned since attention was at that time a recurring theme in various types of neural network). Bahdanau attention - not just the general concept of attention - seems to be a very critical piece of the Transformer architecture since this this is what allows the Transformer to find things in context and is behind the "induction head" mechanism that appears central to how Transformers operate.
- greenavocado 4mo agoSchmidhuber will NEVER stop trying to aggressively preserve his relevance and its endlessly amusing. Good for him.
- throwa356262 4mo agoHot take: The real root of the current AI boom is a master thesis from university of Toronto. The thesis demonstrated that neural networks much longer than before could be trained by simply having a random fraction of the neurons excluded during forward and back propagation. That's how we got practical deep neural networks. Without that we would still be in AI winter.
- trilogic 4mo ago> it is easy to forget that the foundations of this trillion-dollar industry were laid down over 30 years ago in Munich Yes is very easy to forget, cause the trillion is not being made in Europe. If it was really conceived in Munich (like the maps that got stolen also), it show how incompetent is Europe to keep it´s technology and protect European companies. It is painful to read this article.
- Sharlin 4mo agoSomehow "protecting companies" by keeping basic research, done openly at a university lab, from being "stolen"? What? It's like saying it's painful that the Web was invented in Europe and opened for everybody rather than being kept at CERN to protect European companies.
- trilogic 4mo agoRight, is totally fine to create new inventions, but let others take credit and financial benefits. It is our duty to protect and get the benefits of European inventions, especially the ones financed with public tax. Open for everybody means benefits for everybody.
- Sharlin 4mo agoHonestly can't tell if sarcasm. Are you seriously proposing that all university research just stop and be replaced with some new proprietary system?
- trilogic 4mo agoI see you need me to spell it for you. "AN INVENTION MADE IN EUROPE, PAID WITH EUROPEAN TAXPAYERS MONEY, NEED TO BE MONETIZED BY EUROPE". This way everyone gets credit and the research can stay open and available for benefits to everyone. I would continue explaining further but you can´t distinguish sarcasm :/
- queeshonda 4mo ago[dead]
- davidw 4mo agoThis sort of seems like a pattern in CS - someone creates something and then it blows up 20 or 30 years later when the world is ready for it.
- slipperybeluga 4mo ago[dead]
- itkovian_ 4mo agoContrarian view; I think he’s right. Many of these ideas it’s almost shocking how many you can find sketched out in his old papers. To the point where I think it’s very wise to read all his work to see what hasn’t showed up yet but likely will. Artificial curiosity for example.