17 ms·
My honest take is a lot of these famous academics played almost no part in the developments at openai. But they want the limelight. They aren’t as relevant as t
by greatpostman 3y ago
My honest take is a lot of these famous academics played almost no part in the developments at openai. But they want the limelight. They aren’t as relevant as they want to be. In many cases, they were directly wrong about how ai would develop
- innagadadavida 3y agoThis is a little harsh. Hinton trudged along with neural networks through the coldest AI winter and helped create the conditions for OpenAI to have all the raw ingredients needed to cook up something powerful.
- KRAKRISMOTT 3y agoIf you need to build an airplane, would you rather consult Newton, the Wright brothers, or a modern aerospace engineer? Inventing a field and snatching up the low hanging fruits doesn't mean somebody would be able to consistently create leading edge output. Most of the advances in deep learning are due to hardware scaling, and the success of a few very specific architectures. Yes credit's due where credit's due, but academia name recognition is very much winner take all. For all the criticism Schumidhuber has received, he has a point. The authors of Attention is all you need, the transformers paper, yolo, have nowhere close to the name recognition of the Turing award trio despite generating similar if not more value through their ideas.
- cma 3y ago> The authors of Attention is all you need, the transformers paper, y Schmidhuber claims to have invented something formally equivalent to the linear Transformer architecture (slightly weaker) years before: https://arxiv.org/abs/2102.11174 https://arxiv.org/abs/2102.11174
- mardifoufs 3y agoSchmidhuber claims to have invented a lot of things. It's almost a running gag at this point.
- uoaei 3y agoAnd yet somehow his claims always bear some truth. I understand the comments about boys crying wolf, but it's hard to ignore the facts on the ground.
- caycep 3y agonot having a PHD in ML, it's hard for me to evaluate his claims, but how valid are all the obscure papers that he brings up? Did someone actually invent backprop in 1930 in some random corner of the former Soviet Union? Or is it a case of "true but misses the point"?
- uoaei 3y agoOften it is indeed the latter, although it is interesting that sometimes despite that it gets at the core of our contemporary understanding of the concepts in question.
- telotortium 3y ago"Formal equivalence" means very little for engineering, to be frank - the implementation is the important thing. If I wanted to be snarky, I'd say that neural networks are "formally equivalent" to Fourier analysis, which is 200 years old. I see that the paper proposes an implementation of linearized attention as well, which many others have done, but none of which seem to have caught on (although FlashAttention at least makes attention O(n) in memory, if not computation).
- innagadadavida 3y agoThere are multiple dimensions here - fame and fortune at the very least and whether it is localized or global in scope. It is still winner takes all, but if you look at the overall landscape, there are plenty of opportunities where you can have an outsized impact - you can have localized fame and fortune (anyone with AI expertise under their belt have no problems with fortune!)
- jxmorris12 3y agoThis may be true in other cases, but not here. Hinton literally wrote the paper on backpropagation, the way that modern neural networks are trained. He won the Turing award for a reason.
- UncleMeat 3y agoHinton was critical for the development of ai. But was he critical for the development of openai, the company? Loads of startups get eminent people on their boards largely for advertising.
- Fricken 3y agoHinton's protege Ilya Sutskever has been critical to Open AI's success.
- jstx1 3y agoHas he contributed that much personally? I thought a lot of the success of ChatGPT is some good ideas from lower ranked researchers + great engineering.
- parthdesai 3y agoHe is the co-founder and chief scientist[0] at OpenAI but "has he contributed that much personally". I don't even know how to respond to that [0]https://www.linkedin.com/in/ilya-sutskever/ https://www.linkedin.com/in/ilya-sutskever/
- jstx1 3y agoI asked the question knowing that he's a co-founder and chief scientist at OpenAI. Being in his position doesn't automatically mean that he's contributed meaningfully.
- shadowgovt 3y agoIt's a bit in the category of "When you consider all factors, how important was Isaac Newton's work to Einstein's discoveries?"
- mochomocha 3y agoYour take might be honest, but it's clearly uninformed. Everyone has been wrong about how ai developed. It's worth giving "The Bitter Lesson" a read [1] if you haven't yet. [1]: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...
- ttul 3y agoTLDR it’s been better to focus on computational growth than clever algorithms. That being said, architectures are also important when they can reduce computational complexity by orders of magnitude.
- ss1996 3y agoIn many cases yes, but definitely not in this. Geoffrey Hinton is as relevant as ever. Ilya Sutskever, Chief Scientist at OpenAI, is a student of Hinton. Hinton also recently won the Turing award.
- meh8881 3y agoRegardless of incentives, I don’t see any particular reason to think he has a more informed view than other experts on the trajectory of AI. He’s made several incorrect bets (capsule networks). I’m sure he’s smart and all. His contributions were valuable. But he’s not special in this particular moment.
- neilk 3y agoYour viewpoint is fascinating. So the inventor of backpropagation, Turing award winner, Google researcher, mentor to the CTO of OpenAI, doesn’t have any special insights about AI and the tech industry that’s forming around it? He might as well be some guy off the street? Who, in your opinion, _does_ have enough context to be worth our attention? Because if you’re waiting for Sam Altman or the entire OpenAI team to say “guys, I think we made a mistake here” we’re going to be knee-deep in paperclips.
- eternalban 3y agoAuthority figures will not matter. This technology, like nuclear weapons, will be pursued to the utmost by all actors capable of marshalling the resources, in secret if necessary. (After all, the 'Hydrogen bomb' was debated pro/con by established authorities, including Oppenheimer and Teller. Did that stop their development?) https://en.wikipedia.org/wiki/Thermonuclear_weapon https://en.wikipedia.org/wiki/Thermonuclear_weapon https://www.simonandschuster.com/books/Dark-Sun/Richard-Rhodes/9780684824147 https://www.simonandschuster.com/books/Dark-Sun/Richard-Rhod... Today: Germany's relevant minister has already declared at G7 that Germany can not follow Italy's example. "Banning generative AI is not an option". https://asia.nikkei.com/Spotlight/G-7-in-Japan/Banning-generative-AI-is-not-an-option-for-Germany-minister2 https://asia.nikkei.com/Spotlight/G-7-in-Japan/Banning-gener... US Senate has a bill drawing the line on AI launching nuclear weapons but to think US military, intelligence, and industry will sit out the AI arms race is not realistic. https://www.markey.senate.gov/imo/media/doc/block_nuclear_launch_by_autonomous_ai_act_-_042623pdf.pdf https://www.markey.senate.gov/imo/media/doc/block_nuclear_la... China's CPC's future existence (imo) depends on AI based surveillance, propaganda, and realtime behavior conditioning. (re RT conditioning: We've already experienced this outselves via interacting with the recent chatbots to some extent. I certainly modulated my interactions to avoid the AI mommy retors.) https://d3.harvard.edu/platform-rctom/submission/using-machine-learning-to-rule-the-chinese-communist-party/ https://d3.harvard.edu/platform-rctom/submission/using-machi...
- rain1 3y ago> Together with Yann LeCun, and Yoshua Bengio, Hinton won the 2018 Turing Award for conceptual and engineering breakthroughs that have made deep neural networks a critical component of computing
- edgefield 3y agoIt sounds like you’re biased against academics. Not only did Hinton develop some of the fundamental ideas behind AI (winning the Turing award) but also one of his PhD students is now the CTO at OpenAI.
- areyousure 3y agoIn case anyone is curious, this appears to refer to https://en.wikipedia.org/wiki/Ilya_Sutskever https://en.wikipedia.org/wiki/Ilya_Sutskever who was a PhD student of Geoffrey Hinton's and is now Chief Scientist at OpenAI. The CTO at OpenAI is https://en.wikipedia.org/wiki/Mira_Murati https://en.wikipedia.org/wiki/Mira_Murati who does not have a PhD.
- edgefield 3y agoSorry, you’re correct. Chief scientist not CTO.
- thundergolfer 3y agoWow the CTO of OpenAi seems to have ~1 yr of hands on engineering experience, followed by years of product and people management, That’s unexpected. I thought the CTO was Brockman.
- KeplerBoy 3y agoReminds me of a press release by Hochreiter last week. He claims to have ideas for architectures that could surpass the capabilities of GPT4, but can't try them for a lack of funding in his academic setting. He said his ideas were nothing short of genius.. (unfortunately german) source: https://science.orf.at/stories/3218956/ https://science.orf.at/stories/3218956/ https://science.apa.at/power-search/11747286588550858111 https://science.apa.at/power-search/11747286588550858111
- neel8986 3y agoReally? Hinton dont need openAI to be relevant. He literally invented back propagation. He sticked to deep learning through 1990s and 2000s when almost all major scientist abandoned it. He was using neural networks for language model in 2007-08 when no one knew what it was. Again the deep learning in 2010s started when his students created AlexNet by coding deep learning in GPU. Chief Scientist of OpenAI Ilya Sutskever was one of his student while developing the paper. He already have a Turing award and don't give a rat's ass about who owns how much search traffic. OpenAI just like Google will give him millions of dollar just to be a part of organization
- PartiallyTyped 3y agoHinton didn’t invent back prop. > Explicit, efficient error backpropagation (BP) in arbitrary, discrete, possibly sparsely connected, NN-like networks apparently was first described in a 1970 master's thesis (Linnainmaa, 1970, 1976), albeit without reference to NNs. BP is also known as the reverse mode of automatic differentiation (e.g., Griewank, 2012), where the costs of forward activation spreading essentially equal the costs of backward derivative calculation. See early BP FORTRAN code (Linnainmaa, 1970) and closely related work (Ostrovskii et al., 1971). > BP was soon explicitly used to minimize cost functions by adapting control parameters (weights) (Dreyfus, 1973). This was followed by some preliminary, NN-specific discussion (Werbos, 1974, section 5.5.1), and a computer program for automatically deriving and implementing BP for any given differentiable system (Speelpenning, 1980). > To my knowledge, the first NN-specific application of efficient BP as above was described by Werbos (1982). Related work was published several years later (Parker, 1985; LeCun, 1985). When computers had become 10,000 times faster per Dollar and much more accessible than those of 1960-1970, a paper of 1986 significantly contributed to the popularisation of BP for NNs (Rumelhart et al., 1986), experimentally demonstrating the emergence of useful internal representations in hidden layers. https://people.idsia.ch/~juergen/who-invented-backpropagation-2014.html https://people.idsia.ch/~juergen/who-invented-backpropagatio... Hinton wasn’t the first to use NNs for language models either. That was Bengio.
- neel8986 3y agoI mean he was one of the first to use backprop for training multilayer perceptron. Their experiments showed that such networks can learn useful internal representations of data[1]. 1987. Nevertheless he is one of the founding fathers of deep learning [1]Learning representations by back-propagating errors
- sidewndr46 3y agoGoing along with that, as long as they are "concerned" about how AI is developing it opens the door to regulation of it. This might just conveniently hobble anyone with an early mover advantage in the market.
- sorokod 3y agoHow about this particular academic?
- kitd 3y agoIt helps to read TFA on occasions. Hinton founded the AI company acquired by Google with 2 of his students. One of them is now in charge at OpenAI. Hinton has had a significant part to play in the current state of the art.
- bitL 3y agoGPT basically showed that scalable brute-force trumps clever theoretical models which makes many academics salty.
- mrazomor 3y agoThat's something that Microsoft research wrote two decades ago. And those results were well known in the NLP community. Example: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/sigir2002-qarevised.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... (Michele Banko published a few similar papers on that topic)
- bitL 3y agoThere was no hugely scalable approach before transformers, RNNs, the previous SOTA, were notoriously bad at scaling.
- tensor 3y agoYes, we needed clever ideas from scientists to make them scale. In fact, we still need clever ideas to make them scale because the current architectures still have all sorts of problems with length and efficiency.
- 10xDev 3y agoWe are talking about a Turing Award winner known as one of the "godfathers of AI" and your take is that this is just about taking the limelight? The level of cynicism on HN never fails to surprise me.
- ftxbro 3y ago> they want the limelight Maybe, but there is another force at play here too. It's that journalists want stories about AI, so they look for the most prominent people related to AI. The ones who the readers will recognize, or the ones who have good enough credentials for the journalists to impress upon their editors and readers that these are experts. The ones being asked to share their story might be trying to grab the limelight or be indifferent or even not want to talk so much about it. In any case I argue that journalism has a role. Probably these professional journalists are skilled enough that they could make any average person look like a 'limelight grabber' if the journalist had enough reason to badger that person for a story. This isn't the case for everyone. Some really are trying to grab the limelight, like some who are really pushing their research agenda or like the professional science popularizers. It's people like Gary Marcus and Wolfram and Harari and Lanier and Steven Pinker and Malcolm Gladwell and Nassim Taleb, as a short list off the top of my head. I'm not sure I would be so quick to put Hinton among that group, but maybe it's true.
- sjkoelle 3y agoHinton is absolutely not a pop scientist. That said, AI doomerism is Planckian
- michael_nielsen 3y agoHe played key roles in the development of backprop, ReLU, LayerNorm, dropout, GPU-assisted deep learning, including AlexNet, was the mentor of OpenAI's Chief Scientist, and contributed many, many other things. These techniques are crucial for transformers, LLMs, generative image modelling, and many other modern applications of AI Your post suggests that you know almost nothing about how modern deep learning originated.
- zackmorris 3y agoI don't disagree. But for me, their mistake wasn't in the algorithms or their approach or anything like that. The problem has always been, and now will likely always be, the hardware. I've written about this at length in my previous comments, but a split happened in the mid-late 1990s with the arrival of video cards like the Voodoo that set alternative computation like AI back decades. At the time, GPUs sounded like a great way to bypass the stagnation of CPUs and memory busses which ran at pathetic speeds like 33 MHz. And even today, GPUs can be thousands of times faster than CPUs. The tradeoff is their lack of general-purpose programmability and how the user is forced to deal with manually moving buffers in and out of GPU memory space. For those reasons alone, I'm out. What we really needed was something like the 3D chip from the Terminator II movie, where a large array of simple CPUs (possibly even lacking a cache) perform ordinary desktop computing with local memories connected into something like a single large content-addressable memory. Yes those can be tricky to program, but modern Lisp and Haskell-style functional languages and even bare-hands languages like Rust that enforce manual memory management can do it. And Docker takes away much of the complexity of orchestrating distributed processes. Anyway, what's going to happen now is that companies will pour billions (trillions?) of dollars into dedicated AI processors that use stuff like TensorFlow to run neural nets. Which is fine. But nobody will make the general-purpose transputers and MIMD (multiple instruction multiple data) under-$1000 chips like I've talked about. Had that architecture kept up with Moore's law, 1000 core chips would have been standard in 2010, and we'd have chips approaching 1 million cores today. Then children using toy languages would be able to try alternatives like genetic algorithms, simulated annealing, etc etc etc with one-liners and explore new models of computation. Sadly, my belief now is that will never happen. But hey, I'm always wrong about everything. RISC-V might be able to do it, and a few others. And we're coming out of the proprietary/privatization malaise of the last 20-40 years since the pandemic revealed just how fragile our system of colonial-exploitation-powered supply chains really is. A little democratization of AI on commoditized GPUs could spur these older/simpler designs that were suppressed to protect the profits of today's major players. So new developments more than 5-10 years out can't be predicted anymore, which is a really good thing. I haven't felt this inspired by not knowing what's going to happen since the Dot Bomb when I lost that feeling.
- rl3 3y ago
- Fricken 3y agoEven developers at Open AI played almost no part in the developments at Open AI. 99.9999% of the work was done by those who created the content it was trained on.
- HarHarVeryFunny 3y agoIf that was true we could have had GPT-3/etc years ago. It's a bit like saying that college graduates are dumb because after all what have they learnt but a bunch of knowledge in text books. The success of these LLMs comes down to the Transformer architecture which was a bit of an accidental discovery - designed for sequence-to-sequence (e.g. machine translation) NLP use by a group of Google researchers (almost all of who have since left and started their own companies). The "Attention is all you need" Transformer seq-2-seq paper, while very significant, was an evolution of other seq-2-seq approaches such as Ilya Sutskever's "Sequence to Sequence Learning with Neural Networks". Sutskever is of course one of the OpenAI co-founders and chief scientist. He was also one of Geoff Hinton's students who worked on the AlexNet DNN that won the 2012 ImageNet competition, really kicking off the modern DNN revolution.
- g9yuayon 3y agoIn addition to what people clarified in this thread, you probably will be interested in this: Neural network was not a popular research area before 2005. In fact, the AI nuclear winter in the 90s left such a bitter taste that most people thought that NN is a dead end, so much so that Hinton could not even get enough funding for his research. If it were not for Canada's (I forgot the institution's name) miraculous decision to fund Hinton, LeCunn, and Bengio with $10M for 10 years, they probably wouldn't be able to continue their research. I was a CS student in the early 2000s in U of T, a pretty informed one too, yet I did not even know about Hinton's work. At that time, most of the professors who did AI research in U of T were into symbolic reasoning. I still remember I was taking courses like Model Theory and abstract interpretation from one of such professors. Yet Hinton persevered and changed the history. I don't think Hinton cared about fame as you imagined.
- calf 3y agoI remember in 2010 a postdoc came to teach a course on model checking and the classroom was just packed with CS students. I never took it but it will be interesting to see what kind of synthesis between traditional logic and neural network paradigms can be achieved.
- pmoriarty 3y ago"it will be interesting to see what kind of synthesis between traditional logic and neural network paradigms can be achieved" Ben Goertzel talks about his work on something like this at around the 16 minute mark in this video: https://m.youtube.com/watch?v=MVWzwIg4Adw https://m.youtube.com/watch?v=MVWzwIg4Adw
- wellthisisgreat 3y agolol literally chief scientist of openai is GH student
- hackerlight 3y agoThe foundational technology, e.g. Transformers, was invented outside of OpenAI. OpenAI were the first to put all the bits together. Kudos to them for that, but if we're doing credit attribution, Hinton is definitely not someone who is just unfairly seeking the limelight, he's about as legitimate a voice as you could ask for.