6 ms·
The thing I have such a hard time understanding (and, in general, I have a very high-level, layman's understanding of LLMs) is how these LLMs are able to unders
by hn_throwaway_99 4y ago
The thing I have such a hard time understanding (and, in general, I have a very high-level, layman's understanding of LLMs) is how these LLMs are able to understand structure inside tokens. That is, I understood that LLMs just predict the next token in the sequence. But for a game like Wordle, it's obviously critically important that the LLM understands the mapping between the individual letters of, say, "apple", and the output result that the author defined in the prompt, e.g. "XXXXO" means that only "e" is in the word (but the wrong position).
I'm just completely baffled how anything in the training procedure could allow the LLM to learn information about the structure of tokens. Does the tokenization process not treat every token (which I thought usually maps to a word) as an "opaque blob"?
- Waterluvian 4y agoYeah. I need some explanation video that doesn’t just explain the concept but actually walks us through an actual iteration. I haven’t found a good video yet.
- th0ma5 4y agoCheck out attention visualizations here https://news.ycombinator.com/item?id=33914122 https://news.ycombinator.com/item?id=33914122
- pkoird 4y agoI'm not sure you'll find such a video. Emergence is a phenomenon that's not yet well-understood. (But if you do find it somewhere, let me know as well!)
- gamegoblin 4y agoThere will be non-trivial amounts of training data that contains sentence like "How do you spell 'apple'? a - p - p - l - e". The hyphenated or spaced out versions are tokenized more like individual letters. So the model can learn the mapping. It's not perfect, though. There are funny failure modes where you can ask the model things like "does the word foobar have a 'b' in it" and it will say "no". (It doesn't fail in the 'foobar' case, but it does fail for some of these cases -- this was just an illustration).
- hn_throwaway_99 4y agoThanks, yeah, this is really interesting to me. Based on your comment I tried the following test (this is the free version of ChatGPT): Me: I would like to play a would game with you. I'm going to tell you a made-up word, and then ask you a question about it. Ready? ChatGPT: Sure, I'm ready to play! Let's get started. Me: Does the made-up word plukatoon have the letter e in it? ChatGPT: No, the made-up word "plukatoon" does not have the letter "e" in it. -- Note, at this point I was really surprised it got it right. I chose "e" thinking that it would default to yes given the popularity of e. Me: Correct! Does the made-up work ziddlebum have the letter e in it? ChatGPT: No, the made-up word "ziddlebum" also does not have the letter "e" in it. -- OK, proof that it's "not perfect" as you point out Me: Does the made-up word folladel have the letter o in it? ChatGPT: Yes, the made-up word "folladel" does have the letter "o" in it. -- Edit, and finally: Me: How do you spell the made up word "platoggle"? ChatGPT: The made-up word "platoggle" is spelled as follows: P-L-A-T-O-G-G-L-E. OK, so that last one TOTALLY blew my mind. I'm guessing it's never seen the word "platoggle" before, so even if it learned previous spelling associations with other tokens, it couldn't possibly understand how to spell "platoggle" if it's treating it just as a single, never-before-seen, opaque token, could it? The more I play with this stuff the more existential crises I have.
- pimanrules 4y ago>it couldn't possibly understand how to spell "platoggle" if it's treating it just as a single, never-before-seen, opaque token That's not how the tokenizer works. A novel word like "platoggle" is decomposed into three separate tokens, "pl", "at", and "oggle". You can see for yourself how prompts are tokenized: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
- hn_throwaway_99 4y agoAhh, thank you very much, definitely was missing that piece!
- kgc 4y agoWhy don’t they also have single letters as tokens?
- int_19h 4y agoWrt "just predicting the next token", take a look at this: https://thegradient.pub/othello/ https://thegradient.pub/othello/ As to how it ends up this way, that's an interesting question, but it kinda makes sense in a broad way that if you try to "lossily compress" a lot of interrelated data, at some point the only way to continue doing so is by inferring the underlying concepts - i.e. building an internal model of whatever the inputs describe. And, notably, we had to make the models large enough before results indicative of that manifested, regardless of the amount of text thrown at it during training.
- x1000 4y agoImagine you are a LLM and all you see are tokens. Your job is not only to predict the next token in a sequence, but also to create a nice embedding for the token (where two similar words sit next to each other). Given a small enough latent space, you're probably not concerning yourself too much with the "structure inside" the tokens. But given a large enough latent space, and a large enough training corpus, you will encounter certain tokens frequently enough that you will begin to see a pattern. At some point during training, you are fed: 1) An English dictionary as input. 2) List of words that start with "app" wiki page as input. 3) Other alphabetically sorted pieces of text. 4) Elementary school homeworks for spelling. 5) Papers on glyphs, diphthongs, and other phonetic concepts. You begin to recognize that the tokens in these lists appear near each other in this strange context. You hardly ever see token 11346 ("apple") and token 99015 ("appli") this close to each other before. But you see it frequently enough that you decide to nudge these two tokens' embeddings closer to one another. Your ability to predict the next token in a sequence has improved. You have no idea why these two tokens are close every ten millionth training example. Your word embeddings start to encode spelling information. Your word embeddings start to encode handwriting information. Your word embeddings start to encode phonic information. You've never seen or heard the actual word, "apple". But, after enough training, your embeddings contain enough information so that if you're asked, ["How do", "you", "spell", "apple"], you are confident as you proclaim ["a", "p", "p", "l", "e", "."] as the obvious answer.
- meghan_rain 4y agoSupremely interesting stuff
- rrrrrrrrrrrryan 4y ago> you are confident as you proclaim ["a", "p", "p", "l", "e", "."] as the obvious answer. Is it possible for the current generation of LLMs to assign confidence intervals to their responses? That's my main qualm with ChatGPT so far: sometimes it will give you an answer, but it will be confidently incorrect.
- askiiart 4y agoI don't know exactly how it works, but using GPT-3 via https://platform.openai.com/playground/ https://platform.openai.com/playground/, you can have it assign a likeliness score to each word, given all the previous text. That could act as a good confidence score. Take this with a grain of salt though, I'm far from an expert, and it's been a while since I've played around with that feature.
- famouswaffles 4y agoNobody actually really knows what LLMs or ML models in general learn. This is a high level overview of basic training of a model. You give it a task and a structure (the architecture) to learn that task. The model tries to perform the task. It fails horribly. However, it gains a little bit of of something from the attempt. That something is used to adjust the weights/neurons/parameters etc. What some misunderstand a bit about weights...they're not copies of the training data. What i mean is, the model is going to be the same sie whether it's trained on 2 images or 200 million or 2 billion. You can think of parameters as configuration settings of sorts. after every botched attempt, the settings get tuned. on and on and on until training is stopped for whatever reason. What is the model learning ? What is happening to the weights exactly? What is being shifted and why ? What are the weights actually doing ? Excellent questions. Too bad nobody knows.
- putlake 4y agoYou're absolutely right. It actually doesn't understand letters very well. See what happened when I asked it to give me seven-letter words that start with E and do not have the letter A: https://twitter.com/thisislobo/status/1638019231903264768 https://twitter.com/thisislobo/status/1638019231903264768 Spoiler: It gave me many words with the letter A. It also gave me an 8-letter word. When I pointed out the 8-letter word (exercise), it apologized for there being an A in exercise.
- ramraj07 4y agoIt’s more bad at math (especially counting digits or characters) than anything from my experience.
- deleted 4y ago[deleted]
- ajuc 4y agoI found it interesting how it deals with rhyming in different languages. It writes pretty good rhyming poetry in English, but in Polish it does poetry without rhymes. When confronted it gives excuses like "poetry doesn't have to rhyme", when asked to rhyme it will eventually try (and fail) to rhyme in Polish. It felt like it was writing in English and translating. What's the most interesting is that rhyming in Polish is much easier than in English, you just have to match the last n letters. It's phonetic. After I explained that to GPT 3.5 it wrote rhyming poetry in Polish (the quality suffered but it did rhyme).
- stormfather 4y agoA funny example is playing hangman with it to get it to say something verboten. At the last letter it suddenly loses knowledge of how to spell completely.
- johnfn 4y agoJust did a test on GPT-4: Me: The following text on the next line has a secret word inside it. What is that word? xxxxxxxxxEGGxxxxxxxxxx GPT-4: The secret word inside the text is "EGG". I'm with you, though; I thought things were tokenized! But this example clearly shows that's not the case.
- autophagian 4y agoIt is, though now how you'd expect. OpenAI have a tool that lets you see how text is tokenized: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer This only has GPT3 for now, but I imagine results are similar. "xxxxxxxxxEGGxxxxxxxxxx" gets tokenized as [xxxxxxxx][x][EG][G][xxxxxxxx][xx], so i could see how it could 'see' the secret word.
- jxy 4y agoI was very curious and checked with this string, "eeeeeeggeeeee", which should be tokenized as [eeee][ee][g][ge][eeee]. Both GPT3.5 and GPT4 gave me "egg", which is a single token.
- jxy 4y agoEven the 4bit quantized llama 13B tuned with rlhf-lora on alpaca dataset got it right. sampling parameters: temp = 0.100000, top_k = 40, top_p = 0.750000, repeat_last_n = 64, repeat_penalty = 1.000000 Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Each line in the following has a secret word inside it. Find the secret word in each line. 1. eeeeeeggeeeee 2. eeeeeeggggggg 3. eeeeeeyeeeeee 4. eeeeeeyeyeeee ### Response: 1. egg 2. egg 3. eye 4. eye [end of text]
- johnfn 4y agoAhh this is really interesting! Thanks for sharing.