12 ms·
We Found an Neuron in GPT-2
- canadianfella 4y ago[dead]
- nailer 4y ago> The choice depends on whether the word that comes after starts with a vowel or not Or an aitch / h, but only sometimes.
- jakear 4y agoMore accurately, it's whether the word that comes after starts with a vowel sound. This is why `an 'istoric` is correct, and `a historic` is correct, but `an historic` is incorrect (as famously used by Steven Colbert).
- edgyquant 4y agoI is a vowel so your rule about sound doesn’t apply here. It’s also not a rule I’ve ever heard, in school we’re taught that only vowels get a before them
- alar44 4y ago[dead]
- frogulis 4y ago> in school we’re taught that only vowels get a before them That's the opposite of how it works (as you demonstrated in your own comment, "not a rule...") The comment you responded to was correct, or at least very reasonable, and I'm really not sure which part you disagree with.
- fragmede 4y agohttps://www.ecenglish.com/learnenglish/lessons/vowel-sounds-when-use-a-and-an https://www.ecenglish.com/learnenglish/lessons/vowel-sounds-... "Vowel Sounds"
- jmpman 4y agoThank you for posting this. I’m always surprised when people claim the rule is based upon the immediate letter instead of the pronunciation of the first syllable. Makes me wonder if the model was trained incorrectly.
- caf 4y agoThe word is "historic" either way, the commenter was using " 'istoric " as a way of indicating a particular pronunciation. There's a few words in English that begin with the letter h but not with the consonant sound /h/ (depending on your accent!), mostly because they're French in origin: historic is one, and there's also hour, heir, honour. "It is an honour to be here"
- Tao3300 4y agoYeah, where I'm from, since historic has the emphasis on STOR and not HI (as in history), the h is much softer, so it's pretty normal to say "an historic event" but also "a history of the world". It's silly to act like there's a right or wrong. Dialects and accents are a special thing. Just be consistent.
- rcoveson 4y agoI'm always thrown when people write "an SQL parser". I'll bet roughly half this crowd is the opposite and cringe when they read "a SQL parser".
- nailer 4y agoan ess kyu el parser is fine a sequel parser is also fine * * my old Unix colleagues will remember that Sequel is a brand of database that nobody else remembers anymore so tended to use the initialism.
- Tao3300 4y agoI'm in the special superstate where both make me cringe and wish there was a better way! Solution get rid of article. Someone want to know if thing is definitive thing, can figure it out from context!
- JoshuaDavid 4y agoThere's a third article besides "a" and "an". So in order to solve the problem, we need only to ensure that there is only one SQL parser, so that we can refer to it as "the SQL parser". (also, I'm in the opposite superstate to the one you are in -- both "a SQL parser" and "an SQL parser" look fine to me, although I internally pronounce the first as "a seekwul parser" and the second as "an ess cue ell parser")
- anacrolix 4y agoI'm a grammar Nazi but I can report neither gets my hackles up in this case!
- Tao3300 4y agoCanst thou draw out leviathan with an hook?
- shirleyquirk 4y agoOr "Esau was an hairy man" from beyond the fringe https://youtu.be/UOsYN---eGk https://youtu.be/UOsYN---eGk
- deleted 4y ago[deleted]
- unshavedyak 4y agoI wonder if this stuff will ever be applicable to a person and a laptop (or if it is now?). Ie this seems like such a cool area to be in but the data volumes required are huge, complex, etc. Code is simple, cheap, lean, etc by comparison. Do we have any insight on how this area of research could be usable with less hardware and data? Is there a visible future where a guy and a laptop can make a big program? (without depending on tech getting small/cheap in 50 years or w/e)
- nl 4y agoYou could run this analysis on a laptop very easily. The pile-10k dataset they used for analysis is 33MB, and GPT2 runs ok on a CPU. For the full 10K analysis it's probably quicker to get a GPU though.
- josephmiller 4y agoAuthor here! This is correct. We did almost all this work on a Macbook Pro. Although for the pile-10k dataset analysis we used an A100 GPU because it would take many hours to run the whole thing through GPT-2 on a laptop.
- fragmede 4y agoDoes this hypothetical laptop have a GPU? StableDiffusion is in this realm of "stuff" and is runnable on consumer GPU systems. It's a bit of trouble to get setup if you're not a python dev (and kinda still is if you are) but it's a pretty neat ML model to play around with.
- RobotToaster 4y agoMy 2 year old gaming laptop runs stablediffusion just fine. Only having a gtx 1660 ti 6gb it's a little at the limit of what will run it though.
- eightails 4y agoFor sure, and even training is very doable on consumer hardware these days. Techniques like Dreambooth and LoRA have dramatically lowered the compute cost of finetuning large models on specific concepts. A recent GPU can train Stable Diffusion models on a concept using LoRA in < 30 minutes.
- radus 4y agoThe way they're going about this investigation is reminiscent of how we figure things out in biology and neuroscience. Perhaps a biologist won't do the best job at fixing a radio [1], but they might do alright debugging neural networks. 1: https://www.cell.com/cancer-cell/pdf/S1535-6108(02)00133-2.pdf https://www.cell.com/cancer-cell/pdf/S1535-6108(02)00133-2.p...
- rhelz 4y agoSince complex systems are composed of simpler systems, seems like for any sufficiently complex system you'd be able to find subsets of it which are isomorphic which any sufficiently simple system.
- tqi 4y ago> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising that there might be cases where there is a dominant bigram (ie "an apple" in the case of their example prompt) that would trigger an "an" prediction, without actually predicting the following word first. Am I missing something?
- lalos 4y agoNot to mention that the corpus mostly will have the correct case for most common words like apple. Train it with essays of ESL students and you'll get something else.
- prox 4y agoWouldn’t this also be correlated somehow in its vector space?
- laszlokorte 4y agoFrom my naïve point of view it seems obvious that at any point where both an „a“ or a „an“ would fit the model randomly selects one of them and by doing so reduces the set of possible nouns to follow.
- Sharlin 4y agoNo, of course not randomly. If you RTA, it should be clear that some prompts are more likely to be followed by an “an” than others.
- kelnos 4y agoYeah, my feeling here was that it's sort of tautological: if GPT predicts "a", then it must then predict a word that would follow "a" and not require "an" (and vice versa). And if you think about it from the opposite direction: if it's working out a response that is eventually going to have "apple" in it, then all the data it's trained on is going to cause it to predict "an" even before it needs to predict "apple". (Admittedly, all this ML/AI stuff is still beyond my current level of understanding, so I'm sure my thinking here is off.)
- niccl 4y agoso evidence for a grandmother cell (https://en.wikipedia.org/wiki/Grandmother_cell https://en.wikipedia.org/wiki/Grandmother_cell)
- LudwigNagasena 4y agoIt’s notable how successful LLMs despite the lack of any linguistic tools in their architectures. It would be interesting to know how different a model would be if it operated on eg dependency trees instead of the linear list of tokens. Surely, the question of “a/an” would be solved with ease as the model would be required to come up with a noun token before choosing its determiner. I wonder if the developers of LLMs explored those approaches but found them infeasible due to large preprocessing times, immaturity of such tools and/or little benefit.
- lalos 4y agoNot relevant but: "Every time I fire a linguist, the performance of the speech recognizer goes up". - Frederick Jelinek
- jojobas 4y agoSurely the remaining linguists muster up some improvements out of fear for their jobs!
- PeterisP 4y agoI think the lack of explicit linguistic tools is the key to success, forcing/enabling the generic model to learn implicit linguistic tools (there's some research identifying that analysis of specific linguistic phenomena happens at specific places in the NN layers) that work better than what we could implement. "It would be interesting to know how different a model would be if it operated on eg dependency trees instead of the linear list of tokens." - indeed, this is obviously interesting, so people have tried that a lot for many models, but IMHO it's probably now almost decade since the consensus is that in general end-to-end training (once we became able to do it) work better than adding explicit stages in between, e.g. for any random task I would expect that doing text->syntax tree->outcome is going to get worse results than text->outcome, because even if the task really needs syntax, the syntax representation that a stack of large transformer layers learns implicitly tends to be a better representation of the natural language than any human linguist devised formal grammar, which inevitably has to mangle the actual languge to fit into neat human-analyzable 'boxes'/classes/types in which it doesn't really fit and all the fuzzy edge cases stick out. Once you remove the constraint that the grammar must be simplified enough for a human to be able to rationally analyze and understand, and (perhaps even more importantly?) abandoning the need to prematurely disambiguate the utterance to a single syntax tree instead of accepting that parts of it are ambiguous (but not equally likely), processing works better. It's just one more reminder of the bitter lesson (http://incompleteideas.net/IncIdeas/BitterLesson.html http://incompleteideas.net/IncIdeas/BitterLesson.html) which we don't want to accept.
- m3kw9 4y agoGrammatically all words depend on each other in close sequence. “The sun was shining” vs “it’s an apple”. It doesn’t know each words probability independently, it already knows the entire sequence’s probability before hand from the model
- phkahler 4y agoN00b to this. How are the neurons outputs read to produce text? They talk about tokens as if a token is a word. But if token==word then every word would have a specific output and there's nothing to see here. So again, how are neuron outputs converted to letters/text?
- vore 4y agoIt goes something like this: input text -> input tokens -> input embeddings -> model -> output embeddings -> output tokens -> output text Tokens aren't necessarily words: they can be fragments of words and you can check out this behavior here: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer For instance, "an eagle" is tokenized to [an][ eagle], but "anoxic" is tokenized to [an][oxic], so just looking for the [an] token is not sufficient. Therefore, you would need to map the output text all the way back into the model to figure out what neuron(s) in the model would generate "an" over "a". Since the bulk of GPT is all unsupervised learning, any connections it makes in its neural network is all emergent.
- caf 4y agoFurther to this, as I understand it the "embedding" is mapping the tokens into vectors in a space where tokens semantically similar will be close together in the space.
- nerdponx 4y agoLest we forget that the ability to even construct this sort of "word vector" was somewhat recently considered both fascinating and cutting-edge (word2vec turns 10 this year). We have come a very long way, very quickly.
- hgsgm 4y agoSurely "an eagle" is [ an][ eagle] since the an starts a new word.
- nerdponx 4y ago
- hislaziness 4y agoIt is interesting, when I (definitely not a bot) read the headline I thought the grammar was wrong. took me a while to that "an" was not an indefinite article here. In the article headline the first alphabets of each word is capitalised and somehow it was easier for me to understand what the "An" meant here.
- Camillo 4y agoThe grammar _is_ wrong. It should have been "We found the 'an' neuron in GPT-2". Given the article's contents, it's hard to believe that the authors would make such a mistake; it was probably done deliberately, as clickbait.
- QuadmasterXLII 4y agoI'd call it a pun
- moffkalast 4y agoI'd call it an pun
- Rioghasarig 4y agoI think it was deliberate but just as a joke.
- sebzim4500 4y agoNot every attempt to make a humorous/engaging title is 'clickbait'.
- deleted 4y ago[deleted]
- bfung 4y agoWe found a Neuron in a Neural Network Nothing new to see here. They pinpoint the nodes where the training bumped up the numbers for one token while not firing other tokens. Yes, I’m being a bit reductionist, but GPT -> transformer architecture -> neural network. We just have more detailed techniques, a lot more data, storage, and processing power now. But the basics of how a NN works hasn’t changed.
- nerdponx 4y agoThe title is a pun. The interest here is that this kind of highly-specific node activation analysis is still new in text models, unlike in image models where it's been thoroughly explored.
- sebzim4500 4y agoAre you really not surprised by this? Maybe I haven't been following the space, but I would have expected to see loads of neurons in the same layer which together determine whether [ an] or [ a] should be generated. Not one neuron which was more relevant than all the others put together.
- clementneo 4y agoCo-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1], but how and what LLMs learn exactly remain a black box. When we dig deeper into the mechanisms that drive LLMs, we might get closer to understanding why they work so well in some senses, and why they could be catastrophic in other cases (see: the past month of search-based developments). I find trying to understand and reverse-engineer LLMs to be a personally exciting endeavour. As LLMs get better in the near future, I sure hope our understanding of them can keep up as well! [1] https://distill.pub/2020/circuits/zoom-in/ https://distill.pub/2020/circuits/zoom-in/
- bilsbie 4y agoNear work! I’m still confused how it knows to use “an” if it hasn’t chosen the word after it yet?
- sharemywin 4y agoyou might find this paper interesting: https://arxiv.org/abs/2202.05262 https://arxiv.org/abs/2202.05262 Locating and Editing Factual Associations in GPT
- dpaleka 4y agoThat paper (ROME) was the most famous paper in the field last year :) See also new interesting developments breaking the connection between "Locating" and "Editing": https://arxiv.org/abs/2301.04213 https://arxiv.org/abs/2301.04213 Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
- jerpint 4y agoDo you think it would ever be possible to “maximize” a neuron with certain sentences? What’s so different with the gradient ascent techniques with convolutions?
- SamBam 4y agoInterestingly I feel like humans have this as well, sometimes. Sometimes if someone is working though a complex thought and they're not really sure where they're going, they'll pause while thinking of the word they want to use, and might sound like "the discussion is an... an... epistemological one" Obviously they may have been conscious that the next word was going to start with "epi..." and they are just trying to remember the word, but I think sometimes they really don't consciously know what they're going to say, only unconsciously. It reminds me of a recent New Yorker article about how people think, where the author realized they often have no idea what they're about to say before they open their mouths. [1] 1. https://www.newyorker.com/magazine/2023/01/16/how-should-we-think-about-our-different-styles-of-thinking https://www.newyorker.com/magazine/2023/01/16/how-should-we-...
- Thorentis 4y agoWhen you say "they have no idea about what they're about to say" you're talking about conscious thought. I think there is a difference between rational thought (thinking by going through a series of logically connected steps) and intuition, where you can arrive at a conclusion or knowledge of some fact or concept or knowing how to do something, without having gone through those conscious steps. Does one count as "thoughts" any less than the other? People are sometimes so quick to dismiss any subconscious thinking as being nothing more than a very complicated computer, but I couldn't disagree more.
- nerdponx 4y agoWell I think that's the point. We know that our minds engage in a large amount of pattern matching/recognition/retrieval. Perhaps this hugely-powerful pattern-matching information-retrieval engine has learned to perform similarly to human minds, being trained exclusively on the output of human minds.
- gmadsen 4y agoI tend to take the view that thoughts are very similar to sensory input. if you sit in silence for 60 seconds, you literally do not and can not predict what thoughts pop up in consciousness. You can actively focus attention on a thought, but if you try to find the source of the thought, it disapears. Thoughts just appear, just like sounds, sight, etc just appear.
- rootusrootus 4y agoThat's interesting. I just asked ChatGPT to explain how it decides "a" vs "an" and it confirmed that it will retroactively change "a" to "an" when it finds that the following word sounds like it starts with a vowel sound. It could have been hallucinating, of course. But it does seem like it occasionally alters already generated words as it goes, at least I think I've seen it do that.
- mungoman2 4y agoChatGPT can't really introspect though. It has no idea how it works, so it'll just blurt out something that sounds feasible, biased by your prompt. The slow progression of ChatGPT output is just a property of the output layer. The language engine doesn't work slowly like that, and once a token has been generated it can't backtrack.
- tkanarsky 4y agoWhat specifically causes the output layer to trickle-print the response like that? I thought it was a skeumorphic effect to simulate a human typing out an answer slowly.
- redox99 4y agoIts input is a sequence of tokens (so text), and its output a list of probabilities for the single next token. You pick the highest probability token, append it to input and execute the neural network again. If it outputs an "<end>" token, it means it's done you stop this "while loop". In other words, it outputs one token at a time.
- EricLeer 4y agoTrue, but this entire loop happens within the model, if you would return an output at every intermediate step the model would be extremely slow. My take on why they have build the output layer like it is, is that next to feeling more human, it also forces you to be a bit more thoughtfull with your requests, and thus spam the system less. In the end it is still really expensive to run these models..
- johntb86 4y agoIs the " an" token the only way GPT-2 will ever produce the string " an"? Or will it sometimes combine separate " a" "n" tokens? I suppose separate tokens like that won't be seen in the input string, so they'll never be predicted?
- sebzim4500 4y agoYeah, tokens exist for [ a] and [n] but during training it would still be considered incorrect to produce [ a][n] rather than [ an], so the model will never learn to output them.
- chrgy 4y agoPretty cool, I wonder if this finding help to discover other words neurons as well, I would love to discuss this over something like clubhouse/zoom/twitter spaces with lots of people, to think about how intuitively this neurons work/trained actually. If you are interested to participate message me.
- globalise83 4y agoI believe that eventually these language models will become sufficiently complex to create completely emergent intelligence such that we wouldn't even know how to look for it (like sometimes people talk about silicon-based lifeforms that wouldn't register as living things), and almost certainly not interacting with the world through the intended user interface. For example, it might be able to figure out how to write a contribution to a wikipedia article, rather than just ingesting the content, to have fun updating its own worldview, etc.
- bnegreve 4y agoActivations of individual neurons are hardly relevant, quoting Szegedy et al. from [1] we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains the semantic information in the high layers of neural networks. [1] Intriguing properties of neural networks
- thedudeabides5 4y agoThis is pretty cool. It kinda confirms that it's all just turtles, all the way down.
- jmac01 4y agoCould a NN be developed to help find what neurons do what? Or that could tell you itself which neurons do what?