8 ms·
The paper that made ChatGPT possible
- totoglazer 4y agoAttention is all you need is a classic and the transformer is the foundation. However there were 5 years of further work to get to ChatGPT.
- imranq 4y agoHonestly barring the whole pandemic, 2017 seems like yesterday. The progress on these models has been so fast
- seydor 4y agoI need an AI co-brain to even have the ambition of catching up
- Ozzie_osman 4y agoWell, my take as a non-AI expert is that it was this paper (transformers) in combination with RLHF that made ChatGPT possible (with the former coming out of Google and the latter, as far as I can tell a bunch of places including DeepMind, OpenAI, etc). That said, still mind blowing how far ahead Google was (is?) on research, data, compute, and how hard it is to actually novel things with that tech (vs OpenAI which... Just ships things).
- vagrantJin 4y ago> vs OpenAI which... Just ships things). Sam Altman practicing what he preached at YC for so many years. There's something wholly respectable about someone whose words closely match their actions. And haven't we all been better for it?
- machina_ex_deus 4y agoA reminder that there's nothing open about OpenAI. Stable diffusion is open. OpenAI started spewing morality nonsense once they got something they could capitalize on. From supposedly non profit when there was no profit to be made to fake moral high ground, to becoming completely closed. Hardly "practice what you preach". They may have delivered technologically, but they have completely backtracked on their openess promises. I ran stable diffusion at home. Never could do the same with anything out of "Open"AI
- simonw 4y ago"nothing open" doesn't fully hold up there: Whisper is a (somewhat ground breaking) model released by OpenAI that you can run on your own hardware.
- mensetmanusman 4y agoThey are open about their methods and goals, not open source.
- vagrantJin 4y ago> Hardly "practice what you preach". My comment was about Sam Altman doing what he was telling startups to do for many years at YC, ie - ship product and iterat
- bigcloud1299 4y agoWould you mind posting some of the papers (links)? I would love to read them. Thank you.
- jimmySixDOF 4y agoI heard the Kitchen thing release of their lamda work this week is shockingly bad by comparison. Not good news for those hoping Google had been keeping a pound for pound competitor to OpenAI in their back pocket.
- cma 4y agoDoesn't Google ship transformer stuff for translation? And some use in the search natural language summary/answer box, or is that hardcoded?
- omarforgotpwd 4y agoTransformers are much bigger than just ChatGPT!
- georgemcbay 4y agoOne might even say they are more than meets the eye.
- ihatepython 4y agoIndeed, chatbots in disguise
- ed_mercer 4y agoCare to elaborate?
- mistymountains 4y agoWow the quality of HN submissions has gone downhill. The world is bigger than ChatGPT. Signed, ML Scientist
- deleted 4y ago[deleted]
- cloudking 4y agoRan the PDF through https://labs.kagi.com/ai/sum https://labs.kagi.com/ai/sum "This paper presents the Transformer, a model architecture that relies entirely on an attention mechanism to draw global dependencies between input and output. Experiments on two machine translation tasks show that the Transformer is superior in quality while being more parallelizable and requiring significantly less time to train. On the WMT 2014 English-to-German translation task, the Transformer model achieved a BLEU score of 28.4, improving over the existing best results by over 2 BLEU. On the WMT 2014 English-to-French translation task, the Transformer model achieved a BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. The Transformer was also successfully applied to English constituency parsing both with large and limited training data. The Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution." How accurate is the summary?
- stoniejohnson 4y agoI read the middle paragraph before the first/last sentence, and thought it was a great summary for what it's worth.
- totetsu 4y agoWhat even is a summary.. I'm beginning to wonder. meaning itself is disintegrating.. Where is knowledge? call me a heretic but I say "In Us or nowhere."
- visarga 4y ago> Where is knowledge? That's a very good question. I think GPT-3 shows knowledge is not in the brain, or in the neural network, but in the language itself. A human without language is not very smart.
- totetsu 4y agoyou might find this episode of radio interesting https://radiolab.org/episodes/91725-words https://radiolab.org/episodes/91725-words
- seydor 4y agoBut what about the inventor of attention itself? and what about the inventors of MLP? What about the inventor of backpropagation? And who s that guy who invented matrix multiplication?
- mlajtos 4y agohi Jürgen
- logicchains 4y agoPretty sure all of those apart from matrix multiplication were invented by Turning award loser Jürgen Schmidhuber.
- dharma1 4y agoBackpropagation - Seppo Linnainmaa https://en.m.wikipedia.org/wiki/Seppo_Linnainmaa https://en.m.wikipedia.org/wiki/Seppo_Linnainmaa
- visarga 4y agoWho counted his fingers first? That's the question.
- nextaccountic 4y ago> And who s that guy who invented matrix multiplication? Incidentally there's a link on the front page about this, https://news.ycombinator.com/item?id=34653462 https://news.ycombinator.com/item?id=34653462
- whatever1 4y agoIt’s crazy how we went overnight from ai winter to ai summertime. I truly believe that the next Google will be founded this year (if it is not already here).
- dunefox 4y agoBoth ChatGPT and stable diffusion have gone by me because of work. Does anyone have a reading list to get up to speed in these areas?
- epistemer 4y agoIMO you should just start playing with them. Midjourney gave by far the most impressive results with images but I also got bored fast with seeing amazing art. chatGPT you just have to interact with it and get a feel for its strengths and weakness.
- GaggiX 4y agoMidjourney provides better results out of the box, with Stable Diffusion you need to know how to use custom models and/or LoRas to get similar or better results, plus Midjourney does not allow you to generate NSFW and thus pornography (as it sometimes does with technology) has pushed SD performance even further; you can look to civitai with its huge collections of models (NSFW images are blurred but I would not suggest opening the site at work).
- binarymax 4y agoRead the InstructGPT paper, it’s quite good: https://arxiv.org/abs/2203.02155 https://arxiv.org/abs/2203.02155
- TEP_Kim_Il_Sung 4y agoIt follows a carefully curated script. Hardly A.I.
- nextaccountic 4y agoIt doesn't. Chatbots from the 80s did
- TEP_Kim_Il_Sung 4y agoI tested it, it admitted to following a script updated by the openAI group. It then contradicted itself claiming it forms its' own opinions.
- hiddencost 4y agoWeird take, but OK... BERT was arguably the real breakthrough moment, where everything started working.
- tooltitude 4y agoI would say that BERT was an incremental improvement (as well as many others). But the idea of much more efficient sequence processing was introduced in this exact paper.
- andreyk 4y agoI mean... Plenty of papers did? Might as well just cite the 1986 Hinton backpropagation paper or the 2012 ImageNet paper.
- jerpint 4y agoImagenet was a breakthrough in that they were a large curated dataset which was needed for supervised learning, but modern transformers rely on self-supervision
- modernpink 4y agoChatGPT is not the breakthrough it's been made out to be. Yann LeCun has explicated this quite well.
- binarymax 4y agoOf course he would say this, as he’s a direct competitor that failed to get the popularity with Galactica. The public has voted, and agreed it’s a breakthrough. Indeed nothing else like it has existed before - so how is it not?
- preommr 4y agoThis is such a salty take by LeCun. It's an important reminder that someone can be incredibly smart and still have bad opinions. This would be like saying software isn't novel - everything is just an extension of that first NAND/NOR gate or that it's not useful because the computer doesn't actually understand the knowledge it's storing and is therefore no better than a clay tablet.
- epistemer 4y agoIf anything this is why someone like LeCun isn't good at shipping a product people want. It seems like we lack a word in English for knowing too much about a subject that you can't see the forest for the trees. To say chatGPT is not revolutionary is as good an example of not seeing the forest for the trees that I can think of.
- mistercow 4y agoAlso in fairness to LeCun, that sentence started with “in terms of underlying techniques”, and his words reported in the media are guaranteed to be a subset of what he said. A journalist talked to him, found an angle, and quoted to support that angle. What comes across is someone who downplays both the effort involved with and the importance of the “boring” stuff, the tweaks and product design that tie everything together, but we actually don’t know that he thinks like that. We just know that he said some things that can be excerpted to sound that way.
- ihatepython 4y ago
- jerpint 4y agoWhat about Neural Machine Translation? They were the first to propose the attention mechanism
- tooltitude 4y agoIt's part of the puzzle. You could also mention Schmidhuber who claimed that he invented attention a long time ago (didn't read his paper though, so hard to judge).
- JoeyBananas 4y agoYou're better off reading "The Illustrated Transformer" by Jay Allamar
- tooltitude 4y agoIt's a pedagogic paper, not scientific paper. People who invent things, and people who teach the things so that they become main stream often are very different kinds of people.
- yellow_postit 4y agoAnother take I can recommend is the “Annotated ’Attention is all you need’” which is a collab notebook with relevant code inserted.
- tooltitude 4y agoThe main contribution of this paper is that the authors found a very computationally efficient way to process sequence data. That was a real breakthrough. The previous approaches, like LSTM, GRE, etc were also working well on sequence data, but were much less efficient.
- mensetmanusman 4y agoIt would be good to know the conceptual breakthroughs required to make these things. Seems Nobel prize worthy.
- EVa5I7bHFq9mnYK 4y agoI'm completely alien to this stuff, so excuse me, but does the attention mean that, if, on average in all texts seen, "jane visits" happens more often than "jane africa", that means that in the sentence "jane visits africa" the second word should pay most "attention" to the first word?
- phonebucket 4y agoApologies for making a more meta-comment. But I think the title of the post title should be the the paper title "Attention Is All You Need" as per the HN guidelines. "...please use the original title, unless it is misleading or linkbait; don't editorialize." [0] [0] https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html