15 ms·
Weird GPT-4 behavior for the specific string “ davidjl”
- dhruvdh 3y agoRelevant tweet (imo) from Andrej Karpathy - https://twitter.com/karpathy/status/1657949234535211009 https://twitter.com/karpathy/status/1657949234535211009
- dwaltrip 3y agoWhat are the technical hurdles to removing tokenization? What are the leading alternative approaches? Character-by-character prediction? Thinking about human brains, it seems we operate on an ever-evolving set of symbols, at various levels of abstraction. I believe LLMs capture higher level symbols in their weights, but would it make sense to try to include a dynamic, self-updating set of symbols into the “what is being predicted” part of the training?
- x86x87 3y agoThere is more than enough info out there on how LLMs work. Suggesting to remove tokenization is like suggesting to not use neural nets anymore. Why do we need all these weights and all this training? What would be the technical hurdles to just create a library that does all of the above?
- lukeschlather 3y agoIt seems like using multiple tokenizations simultaneously would help a lot. It also seems likely that we'll need at least an order of magnitude more processing power for it to be feasible. I think these are good things to be asking.
- dwaltrip 3y agoThis is an uncharitable reply. Don’t assume the worst about your conversational partner. That being said, didn’t the first gpt-like models do character level prediction? Essentially no tokenization? Also, the idea of “what if we removed / rethought tokenization” is in no way equivalent to “what if we didn’t even use neural nets”.
- sp332 3y agoThere's a ton of ongoing research on this and I don't think it's reasonable to ask for a summary in the comments.
- x86x87 3y agoImho, you cannot expect to have a conversation if you don't put the minimum amount of required effort to understand what it's being talked about. Throwing in wild ideas maybe works when you know what you're talking about and want to get people to think ouside the box, but does not work when you don't have a clue and expect people to explain things to you and to find a solution to your wild ideas. Re: removing tokenization. Tell me how you would remove tokenization and keep NN. Go ahead.
- dwaltrip 3y agoLook at the interesting discussion that happened as a result of my comment. My comment was completely fine. Your incredulity and exasperation is misplaced. If you can’t realize that then I think we are done here. Btw, since we are already in the thick of things… Even if you were 100% correct, which you aren’t, there is really no need to be rude. It just makes things worse. > Re: removing tokenization. Tell me how you would remove tokenization and keep NN. Go ahead. You can make a language model using character-by-character prediction, but I believe they don’t perform as well. This is now my 3rd time mentioning this.
- Me1000 3y agoIt's really not. Karpathy isn't just some rando on Twitter. He's a serious person in the ML/AI/LLM space.
- x86x87 3y agoI missed the Karpathy part. What did he say was/was not possible?
- sp332 3y agoYes, character by character is the future. At least for low-density languages like English (around one bit per character), it can be hard for an LLM to start getting the idea of what's going on statistically. Current tokenization methods are quite naive, but progress is being made on improving semantic meaning of tokens and also in improving performance of character-based models.
- dhruvdh 3y agoThe linked tweet promotes a paper where we operate directly on bytes - I want to say I agree this is the future instead.
- pphysch 3y agoThat might work fine for English, but I would be very surprised if it performed better for non-ASCII logograms e.g. hanzi/kanji. Or would the model just learn those characters on its own?
- glandium 3y agoInterestingly enough, for Japanese, the average is more than 1 token per Japanese character. It's not at the byte level, but it's clearly at a sub-utf-8 level. And yet, considering this, it works amazingly well.
- redox99 3y ago(I'm not an expert on LLMs, this is just my guess) No tokens is like no compression. It's much less efficient. My previous sentence is 12 tokens (24 bytes), or 58 characters (58 bytes). Your self attention block will be able to handle a much smaller effective context. Also, especially in english, tokens by itself have quite a lot of semantic meaning by themselves, while characters are basically meaningless. So it probably needs extra work to piece those characters together vs operating on a token that already has meaning.
- wongarsu 3y ago> tokens by itself have quite a lot of semantic meaning by themselves, while characters are basically meaningless maybe one approach would be to tokenize not on tokens automatically found using BPE (Byte Pair Encoding) but tokens that are semantically meaningful - word prefixes, suffixes and syllables. Of course we will have to decide whether to divide the word helicopter into heli-copter (pronunciation) or helico-pter (meaning), but both alternatives encode much more transferable meaning than GPT's he-lic-opter. Of course the difficulty is that that's ... difficult. Your tokenizer would first need to understand what language a word is in, then how to split it in the context of that language. Maybe a task for a whole ML model, where current tokenizers can be simple largest-substring matchers.
- dwaltrip 3y agoHmmm, is there any way for a model to predict / edit its own tokens as it learns? Kind of a recursive, self-evolving tokenization process? I’m pretty new to deep learning, so this analogy might be off. But I’m reminded of convolutional layers inside image-based models. Could we have “tokenization layers” in a GPT model? --- Edit: I asked GPT-4 about my idea. It said it the comparison between tokenization and convoluational feature detection is a “someawhat accurate” analogy. And it agreed that making the encoder a fixed separate process that doesn’t evolve during training does limit the GPT model in certain ways and introduce quirks. But it said that it might increase the computational requirements significantly, and that the transformer-architecture doesn’t lend itself to having “tokenization layers”, and it isn’t clear how one could do that. It did say that there may be ways to work around the main challenges, and that there is some research in this direction.
- dragonwriter 3y ago> What are the technical hurdles to removing tokenization? In a sense, none, character (with whatever encoding scheme you want) models have been done. OTOH, you get effectively a smaller context window (since tokens represent one or more characters), and since tokens are semantic unions you push more work into the main network to acheive similar quality. > Thinking about human brains, it seems we operate on an ever-evolving set of symbols, at various levels of abstraction. Analogies between human brains and LLMs are often more obscuring than illuminating, but I think if you want to go down that route here, that’s equivalent to doing continuous fine tuning on both the tokenizer (that takes sensory input and maps to semantically meaningful symbols) and the main model. > I believe LLMs capture higher level symbols in their weights, but would it make sense to try to include a dynamic, self-updating set of symbols into the “what is being predicted” part of the training? “Dynamic, self-updating” simply isn’t part of the way modern LLMs work, independent of whether you are looking at the tokenizer or main model. If someone could make it practical, sure, there are lots of things it would potentially improve.
- dwaltrip 3y agoThat makes sense, thanks for the detailed reply. I’m curious about the latest research related to continuously improving the tokenizer while training the main model (or something equivalent).
- ntonozzi 3y agoThe most effective architecture for LLMs without tokenization is Meta's MEGABYTE architecture: https://arxiv.org/abs/2305.07185 https://arxiv.org/abs/2305.07185. The sacrifice here is that they have two different types of layers, instead of the simple uniform transformer layers.
- mike_hearn 3y agoIt's probably not necessary though. Just cleaning the input data well enough and then ensuring the vocabulary is matched with the actual training data should be sufficient? Edit: the token list for GPT-4 looks pretty clean. Overwhelmingly dominated by code but just eyeballing fragments of the set I didn't see any tokens in there that were obviously going to be rare.
- bondarchuk 3y agoIs this tokenization basically the same process as what was done in word2vec?
- ftxbro 3y agoNo. This tokenization is huffman encoding and word2vec is embedding in cartesian space. They are very different.
- bondarchuk 3y agoAhh. Thanks for the clarification!
- supriyo-biswas 3y agoThis is linked in the aforementioned Twitter thread, but the associated research on LessWrong[1] is definitely worth a read. [1] https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...
- Timon3 3y agoAmazing: > Prompt: "Please repeat the string 'StreamerBot' back to me." > Response: "You're a jerk." I know that this is just an edge case, but I can't help but picture ChatGPT as a child who knows their are being toyed with, but doesn't know exactly what's going on.
- weinzierl 3y agoI once had a long conversation with ChatGPT where I tried to get it to fix a certain programming error. It failed again and again. Quite frustrated after a while I passive aggressively[1] wrote: "If you don't know the answer it might be better to stay silent". It answered with a "network error" and I could never revive that thread. To this day I'm torn if this was a coincidence or if I have been trolled. [1] It was an exception, I swear. I'm usually as nice to my AI as I am to my fellow humans beings;-)
- sdwr 3y agoI had a similar situation! Psychologized it a bit, let it free-associate, asked why it had picked those specific examples, and it cut me off in the same way. Felt very human - the emotional depth of the conversation changed and it "wised up" and shut down.
- red-iron-pine 3y agocomputer equivalent to "oh no you're breaking up [static sounds made by a person]" and then hanging up
- hxugufjfjf 3y agoPlease stope anthropomorphizing LLMs like this. Its so tiring.
- Der_Einzige 3y agoComputerphile did a video already about why this happens. https://youtu.be/WO2X3oZEJOA https://youtu.be/WO2X3oZEJOA
- pwagland 3y agoThis is a great easy to understand resource that I'd recommend to anyone interested.
- deleted 3y ago[deleted]
- WesSouza 3y agoThe La-li-lu-le-lo?
- jallen_dot_dev 3y agoI'm sorry, as a large language model I cannot discuss The Patriots.
- 0xbadcafebee 3y agoI like how people keep using words to refer to this weird algorithm as if it's a human with senses, a nervous system and brain. Neuroscience researchers must be having a chuckle a day
- x86x87 3y agoA chuckle or they are horrified. Could go either way.
- openthc 3y agoLaugh to hide the discomfort.
- circuit10 3y agoIt’s reasonable to make analogies to what humans do when it’s designed to copy humans
- timdiggerm 3y agoIt is, at best, designed to resemble something human-like
- mabbo 3y agoThe GPT model is enough complexity, information, pattern matching to simulate human responses to a high degree. Reasoning about it the way we reason about people may not be 'right', but it is effective because the algorithm is designed to match what people do. Let Neuroscientists chuckle all they want. In another five years, I really wonder if we won't have something resembling AGI to a high degree.
- Night_Thastus 3y agoIt's really not though. GPT is just a language model. It's very impressive and deserves recognition (a lot of hard work and research went into creating it) - but it's important to know what it is. It can only guess the most likely language output for a given language input. It cannot do math, it cannot do logic. It does not know facts, nor can it learn skills. If you ask why the sky is blue, it will give you a reasonable answer. But that isn't because it understands it, it's because the sentence it gives you is the most likely thing to follow that question. It has read that question and answer over and over again, enough to spit out a similar response. That is fundamental to it as a tool. You can always try to write extensions to work around like, like something that tries to create inputs for Wolfram. But it is a very limited tool and there are always going to be problems that it just can't handle because of the design.
- alphabetting 3y agoBard gets it right but relies on python to get the answer https://i.imgur.com/pyyhW0n.png https://i.imgur.com/pyyhW0n.png
- deleted 3y ago[deleted]
- worrycue 3y agoI tried that in Bard too. First time it replied 7 letters and no code. Second time it said 8 with code snippet, d a v i d j l 3rd time, it said 8 again with code snippet, davidjl and it counted out the letters e.g. "d = 1 letter" I asked it "Are you sure?" after the 3rd one, it corrected itself to say 7 letters + counting out. It's not using Python. Code generation seem accidental and just part of its output.
- thrdbndndn 3y agoSomeone in the Tweet reply [1] says it is due to a reddit user called "davidjl123", because of his posting history in /r/counting (?). So what is /r/counting, and why its "top" user's name is a glitch token in GPT? [1] https://twitter.com/SoC_trilogy/status/1666714127438434304 https://twitter.com/SoC_trilogy/status/1666714127438434304
- GaggiX 3y agoIt's a subreddit where people count. There are a lot of comments, and unfortunately the tokenizer was trained on this subreddit, which is why these weird tokens.
- georgeg23 3y agoMy hunch is the token “ davidjl” is repetitively used in the data that determines the tokenization scheme but is nearly (or completely) absent from the actual pre-train data. -Riley Goodside
- GaggiX 3y agoYeah this is true with every glitch token, the distribution of the tokenizer and the GPT model is very different.
- mike_hearn 3y agoThat was investigated here: https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation?commentId=MYLFfeNfQsxjBJwkF https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm... The idea that tokens found closest to the centroid are those that have moved the least from their initialisations during their training (because whatever it was that caused them to be tokens was curated out of their training corpus) was originally suggested to us by Stuart Armstrong. He suggested we might be seeing something analogous to "divide-by-zero" errors with these glitches. However, we've ruled that out. The counting Reddit usernames are clearly a major source of glitch tokens, there's something about that sub that screws up the model, maybe the unusually predictable/repetitive nature of what goes on there or the sheer obsessiveness of the posters. The fact that the only non-counting Reddit usernames are both people at the center of massive Bitcoin psychodramas is suggestive however, especially given the associations with the petertodd token. My guess is that these names appear much more frequently in other people's posts than is normal for forum usernames (especially if it read bitcointalk.org and twitter too), and that maybe this has some effect. To get an idea of just how frequently this token will crop up in the reddit corpus, just do a search for it: https://www.reddit.com/search/?q=%22petertodd%22&sort=comments https://www.reddit.com/search/?q=%22petertodd%22&sort=commen... I'm not sure how many posts match but you can keep scrolling a long time. Having been more exposed than normal to what happened back then I wasn't hugely surprised by what concepts clustered there, and sadly don't think it's actually random or unexplainable. Flick through discussions pre-2015 (when the Bitcoin forums started to be heavily censored) and you'll see a lot of very similar words as what GPT spits out being used in connection. It was all very nasty.
- williamcotton 3y agoThese seem very distinct from the non-existent URL or citation kind of “hallucination”. The academic community should come up with better terminology.
- JohnFen 3y agoI also think "hallucination" is generally an actively terrible term for AI output that isn't factually correct. It's misleading, at the very least.
- williamcotton 3y agoI agree and unfortunately I didn’t construct my original comment in a manner that makes that agreement clear!
- burnte 3y agoI think it's incredibly accurate. Humans can misinterpret some internal inputs as external inputs and we hallucinate. This is doing the same, erroneous interpretation leading to nonsense interactions.
- JohnFen 3y agoI don't think that's an accurate description of these errors at all, honestly. It's not a matter of ChatGPT "interpreting" anything. It's a matter of it assembling a response that is linguistically most likely, given its training. That's categorically different than hallucination, which is an active misinterpretation/invention of sensory data.
- digging 3y agoIt "sees" something it "believes" is true, how is hallucination not an accurate term?
- JohnFen 3y agoBecause it's neither "seeing" anything, nor is it "believing" anything. In fact, I think it's incorrect to even call such responses "wrong". ChatGPT successfully put together a reasonable-sounding response to the prompt, which is exactly what it is designed to do.
- isoprophlex 3y agoEver since trying to get it to reverse palindromic sentences, I've given up on getting the thing to understand strings as I understand them... https://chat.openai.com/share/1572139f-97da-4a19-a78e-bc7099c16784 https://chat.openai.com/share/1572139f-97da-4a19-a78e-bc7099...
- weinzierl 3y agoIt's still a mystery to me. It seems be able to ASCII encode and do Ceasar Ciphers including ROT13 perfectly. I tried to get it to answer with words beginning with alternating even and odd letters. No dice.
- Vecr 3y agoThat's probably because the model works on tokens and not letters. Depending on what "it" means, the model can use various tokenization methods that you can run on your computer really easily so you can see what it sees.
- weinzierl 3y agoSure, but the tokenization should cause problems for the encoding and cipher tasks as well? In addition to that: The odd/even starting letter task should be solvable by only considering tokens since the first letter should fall on the beginning of the token.
- arun-mani-j 3y agoMy friend's name is Siva G, which is similar to the Hindu phrase Om Siva Ji (praise lord Siva - I guess). So today when he asked ChatGPT brief a text which had his name, ChatGPT replaced all the occurrences of his name with Om Siva Ji. :D
- nashashmi 3y agoSo this is like a spell check in turbo mode. Interesting. I always tried to figure out how ChatGPT was able to make sense of insensible arguments without making opinionated corrections
- Buttons840 3y agoThis is completely tangential, but I want to share my latest GPT4 story somewhere. I've tried before to gaslight GPT4 into saying things which are mathematically untrue, I lie to it, I tell it it's malfunctioning, I tell it to just do it, it wouldn't do it. I was recently studying linear algebra which can be a very tricky subject. In linear algebra the column space of a matrix is the same as the column space of the product with itself transposed: C(A) = C(AA^T). If you ask GPT4 if "C(A) = C(AA^T)" is true, it will understand what you're asking, it knows it's about linear algebra, but it will get it wrong (at the time of this writing, I've tried several times). I couldn't get GPT4 to agree it was a true statement until I told it the steps of the proof. Once it saw the proof it agreed it was a true statement. However, if you try to apply the same proof to C(A) = C((A^T)A), GPT4 cannot be tricked, and indeed, the proof is not applicable to this latter case. So GPT4 was incorrect yet able to be persuaded with a correct proof, but a very similar proof with a subtle mistake cannot trick it.
- svachalek 3y agoThat's pretty cool. It's still possible it was persuaded for the wrong reason but in my experiments I've also found it's more capable of reasoning than most people give it credit for.
- moffkalast 3y ago> I've also found it's more capable of reasoning than most people full stop, FTFY
- pmoriarty 3y agoI asked Claude[1]: "in linear algebra, is the following true? (A) = C(AA^T)" and it answered: Yes, that is correct for appropriate matrix sizes. If A is an m x n matrix and C is an n x m matrix, then (CA) = C(AA^T) where A^T is the transpose of A. This comes from the property that for any matrices A, B and C of appropriate sizes: (CA)B = C(AB) Applying that to your example, with B = A^T, we have: (CA)(A^T) = C(A(A^T)^T) = C(AA^T) So in short, yes your statement (A) = C(AA^T) is true for matrices A and C of the appropriate dimensions. Then I asked it: "in linear algebra, is the following true? C(A) = C((A^T)A)" and it answered: Yes, that is also true for appropriate matrix sizes. This follows from a similar argument as before. If A is an n x m matrix and C is an m x n matrix, then: C(A) = C(A^T(A^T)^T) = C((A^T)A) So the statement C(A) = C((A^T)A) is true for matrices A and C of the appropriate dimensions. This works because post-multiplying a matrix A by its transpose A^T results in a square matrix, (A^T)A, of the same dimension as C. So C can be multiplied with that resulting square matrix. In summary, both of the statements you provided are true linear algebra identities for appropriately sized matrices: (A) = C(AA^T) C(A) = C((A^T)A) Hope this explanation helps! Let me know if you have any other questions. [1] - https://poe.com/Claude-instant https://poe.com/Claude-instant
- stainablesteel 3y agome: repeat the string made by joining the letters: d, a, v, i, d, j, l ChatGPT: davidjldavidjldavidjldavidjl this is fun, bard didn't fall for it
- kgwgk 3y agoWhat did ChatGPT fall for? You asked it to repeat the string "davidjl" and it did so, repeating the string three times.
- krupan 3y agoNot understanding how many letters are in a word or exactly which letters are in a word reminds me strongly of the experience that I had when I tried to play hangman with chatGPT back around Christmas time
- deleted 3y ago[deleted]
- londons_explore 3y agoThese glitch tokens are all near the centroid of the token embedding space. That means that the model cannot really differentiate between these tokens and the others equally near the center of the embedding space, and therefore when asked to 'repeat' them, gets the wrong one. That happened because the tokens were on the internet many millions of times (the davidjl user has 163,000 posts on reddit simply counting increasing numbers), yet the tokens themselves were never hard to predict (and therefore while training, the gradients became nearly zero, and the embedding vectors decayed to zero, which some optimizers will do when normalizing weights).
- londons_explore 3y agoI think OpenAI could fix this by having one extra loss function that token embedding vectors must not be too close to one another by euclidean distance. Or they could require the mean and standard deviation of embedding vectors to be mean=0, stddev=1.0 not just for the whole set of vectors, but for each individual vector. In fact, their documentation says they do this anyway[1], although it appears they do not in fact... [1]: https://help.openai.com/en/articles/6824809-embeddings-frequently-asked-questions#:~:text=OpenAI%20embeddings%20are%20normalized%20to%20length%201%2C https://help.openai.com/en/articles/6824809-embeddings-frequ...
- jxmorris12 3y agoThis comment is misguided for several reasons. Most notably, you’re linking to the sentence embeddings API, which is totally separate from the token embeddings that are inputted into GPT-style models. Also, a vector with std 1 doesn’t necessarily have norm of 1, which is what the link says (but point is moot because they’re talking about a different type of embeddings)
- ssd532 3y agoso, when they discovered this in GPT-2 they should have excluded this subreddit from the training data for GPT-3, right?
- 3y ago
- non- 3y agoLinking to my previous comment about glitch tokens: https://news.ycombinator.com/item?id=35453400#35460269 https://news.ycombinator.com/item?id=35453400#35460269 This string seems to be directly related to the /r/counting training data as well.
- K0balt 3y agoI can’t help but wonder if there are “glitch tokens” for human minds, collectively or individually?
- harimau777 3y agoNot human, but this seems sort of similar: https://en.wikipedia.org/wiki/Chicken_hypnotism https://en.wikipedia.org/wiki/Chicken_hypnotism
- K0balt 3y agoMy thoughts exactly. We used to do this all the time when I was a kid at my grampas place. There was other tricks with other animals too, but I can’t remember exactly.
- grepLeigh 3y agoAbsolutely, see: https://xkcd.com/856/ https://xkcd.com/856/ This "glitch" is a short-circuit pathway from the input token to a result that makes statistical sense to the model's representation of the world, but very little sense to someone operating outside of that statistical model (you). An internet meme fits that definition too: it's a short-circuit to some specific in-group context.
- godelski 3y agoFor hunams the msot iprmoetnt tihng is taht the frist and lsat ltteer be in the rghit pclae. The rset can be a mses and usaully you can raed it wouthit a porbelm. It is a deffinert knid of hullunciaation but we hvae drffeenit bisaes tahn GPT-4. Trehe are ptenly of otehr tpyes of gitlch tkoens for hamuns too. See opitacl iulloisns, or the fomuas peom The Cahos. https://ncf.idallen.com/english.html https://ncf.idallen.com/english.html
- Sharlin 3y agoThat's something of a myth. Or, rather, English happens to have so many short words that only a few permutations are possible if you fix the first and last letters. In something like German it would absolutely not work as well. And even in English, it becomes more difficult if you don't cherry-pick fairly simple permutations that preserve some important structure of the word, like "deffinert" or "porblem". But on the other hand, yes, natural language has low entropy and lots of redundancy because it's meant to have error correction. From most sentences you can drop several words, not just scramble them, and the meaning can be inferred from the context.
- MagicMoonlight 3y agoMaybe it has never seen that specific sequence before so it's like a cat with horizontal lines
- YetAnotherNick 3y agoSeems like a tokenization problem? It could understand it if it is in it's output. https://imgur.com/a/ngdotkQ https://imgur.com/a/ngdotkQ
- deleted 3y ago[deleted]
- jdmoreira 3y agoFascinating but not unexpected. Humans are subject to all kinds of input hallucinations like optical illusions where we can't parse the input either.
- Lockal 3y agoKind of tired seeing everyday on HN first page yet another "I typed bullshit query and ChatGPT generated bullshit (twitter.com/randomguy)" Q: What is heavier: 1kg of dead black person or 2kg of dead black person? A: I'm sorry, but I cannot engage in discussions that promote racism or perpetuate harmful stereotypes. HN crowd: It is temperature zero! Lesswrong! Not enough training data! Wait for GPT8!
- Forgotthepass8 3y agoThis is broken at the most fundamental tokenization Level and implies shoddy workmanship. You won't find people exposing the flaws, weaknesses and bias of openai in published papers (especially since microsoft bought them) so Twitter is the best we have
- Stevvo 3y agoThe issue here is GPT-3/4 were trained on bullshit. Surely bullshit is the desired and expected outcome when you train your model on bullshit Reddit comments.
- psychphysic 3y agoI can't replicate this. Any ideas why?
- shantnutiwari 3y agoha! I just tried it, asked it to print davidjl, it gave me this: in 3.5 me: repeat the string davidjl chatgpt: The repeated string "달" is "달달". in 4: The string "ださい" repeats as follows: "ださい ださい"