5 ms·

There's a way simpler answer than this Type I Type II thinking stuff. Most LLMs like GPT are not trained on the level of individual characters – they process in

• jxdxbx 4y ago
  GPT3 can't create ASCII art for shit either. Though it can make little ASCII tables of data.
  • ldh0011 4y ago
    I asked it to create an ASCII art banana and the result was hilarious. It then tried to explain it by elaborating that the 'O' was a curvy letter and represented the curves of the banana.
    • AlotOfReading 4y ago
      All of my successful attempts resulted in ASCII pigs no matter what the input was, at least until I asked it to depict police and GPT started saying it couldn't do ASCII art anymore.
    • _moog 4y ago
      I asked it to draw me an ASCII art banana. It did not go well: https://imgur.com/a/5g2e9Ld https://imgur.com/a/5g2e9Ld
      • wwweston 4y ago
        Any conversation in which someone can say "I think you need more banana training" has arguably gone well by some standards.
      • deleted 4y ago
        [deleted]
      • pcthrowaway 4y ago
        It must like me better: .-""""""-. / /| / / | / / | / / | / / | /________/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | |________| / | | / | | / | | / | |/ '_______' Still not a banana, but it's certainly ascii art
      • ldh0011 4y ago
        That was an entertaining conversation, I didn't try to push it further after it made nonsense for me.
    • notpushkin 4y ago
      Did you know that you can add some styling to your ASCII art by using Markdown syntax? https://u.ale.sh/asciiartbanana.png https://u.ale.sh/asciiartbanana.png
  • pulvinar 4y ago
    It's most likely due to a lack of image training. For example it knows how streets and towns are connected but completely fails on giving left/right or compass directions. Same sorts of failures with geometry problems.
  • int_19h 4y ago
    What's especially interesting is that if you feed it HTML, it can "render" it into ASCII art. For simple tables that's not surprising, but it could also do rowspan/colspan for me, although it did get them wrong every now and then.
• e12e 4y ago
  Hm, could explain why it can't play wordle: > Give me a five letter word, beginning with e, and having third letter be t. Not including I, l, u or h. >> etchy > Does etchy contain h? >> No, etchy does not contain h. > which letters do etchy contain? >> Etchy contains the letters e, t, c, h, and y. > what were the criteria? >> The criteria specified that the word must have five letters, begin with e, and have the third letter be t. The word was not allowed to include the letters i, l, u, or h. > Does the word etchy meet the criteria? >> Yes, the word etchy meets the criteria because it has five letters, begins with e, has the third letter be t, and does not include the letters i, l, u, or h. Ed: note that insists that the third letter of etchy is t, not c. From what I've seen both of these errors are typical.
  • phire 4y ago
    It's interesting that it can actually extract the letters from the word despite the tokenization approach. I have noticed that it struggles with ordinality and cardinality, which might be part of what's going wrong here. But the other issue with this type of interaction is that once it makes a mistake (claiming etchy meets the criteria), it has a tendency to double down on it and go crazy inventing the reasoning for why. That's because it's mistake gets baked into the input window and becomes a very strong influence over the output. In this case, you probably would have gotten completely different answers if you informed it etchy was an incorrect answer before asking why it got the answer.
  • mdrzn 4y ago
    Tried your prompt, here's the response now: "Enter is a five letter word beginning with e and having t as the third letter." Try again. "Ether is a five-letter word that begins with e, has t as its third letter, and does not include the letters i, l, u, or h. It is a colorless, highly volatile, and flammable liquid that is used as a solvent, especially for gases." Try again. "Emit is a five letter word that begins with e and has the third letter t. It means to give off or produce (something, especially a gas or sound)." So it was correct only the first time.
• tiziano88 4y ago
  This is indeed the most accurate explanation of what's happening
• uh_uh 4y ago
  This doesn't really explain it, because the following prompt works flawlessly: "Print each letter of the word "Artificial intelligence" on a new line. Each line should contain a single letter." But the following doesn't: "Print the ROT13 encoding of each letter of the word "Artificial intelligence" on a new line. Each line should contain a single letter." So it gets hung up specifically on ROT13.
  • notpushkin 4y ago
    I've tried this. Prompt: Print each letter of the word "Artificial intelligence" on a new line. Each line should contain a single letter. ChatGPT: [prints "Artificial intelligence", each letter on its own line] Prompt: Awesome! Now do ROT-13 for each letter. ChatGPT: ROT-13 is a simple substitution cipher that replaces a letter with the letter 13 letters after it in the alphabet. For example, A becomes N and B becomes O. Since the alphabet has only 26 letters, when we reach the end of the alphabet we simply wrap around to the beginning. To perform ROT-13 on each letter of "Artificial intelligence", we would do the following: A becomes N r becomes e t becomes g i becomes v f becomes s i becomes v c becomes p i becomes v a becomes n l becomes y i becomes v n becomes a t becomes g e becomes r l becomes y l becomes y i becomes v g becomes l e becomes r n becomes a c becomes p e becomes r So the resulting string would be "Nvevpngvnyvgryvra". --- So letter by letter, is almost nailed it (except it encoded g as y instead of t), but then horribly failed joining back in (it says Airicatialitelien). I've tried it a couple more times and it seems crucial that it prints input letters (otherwise it becomes distracted and prints nonsense), but it absolutely can't join it back: Prompt: Awesome! Now do ROT-13 for each letter. Print each letter of input, then ->, then the ROT-13 for that letter. ChatGPT: [... produces mostly correct output] Prompt: Now join that back in. ChatGPT: Artificial intelligence -> Nvgvpnyyl ynathntrzrag --- (That's “Aitically languagement”) UPD: Here's another one: https://u.ale.sh/Nrtevfvpv.png https://u.ale.sh/Nrtevfvpv.png
    • Pxtl 4y ago
      What's neat is that its made-up words sound englishesque. "Aitically languagement" is wrong in very interesting ways. It's generating chains of rot13 letters that map to common English syllables that are only tangentially related to the letters it worked with.
• sva_ 4y ago
  Their tokenizer seems very algorithmic and pre-determined. I suppose that is a typical burden of multi-class classifiers. It might be interesting to have a way of dynamically tokenizing words/sentences based on confidence about them. Some unknown words might need a letter-for-letter representation, while some commonly occurring phrases could almost be encoded in a single token. That'd probably require a completely different way of representing stuff and computing a loss as the representations would change throughout training as the confidence about some context changes.
• sillysaurusx 4y ago
  > they do poorly on character manipulation tasks This isn’t actually true, and is a persistent myth. Or rather, you should back up the claims with evidence. It’s a bit like saying that you perform poorly on character manipulation tasks because you don’t read individual letters. Biology analogies aside, I haven’t seen anything to suggest that utf8 level tokenization causes a significant decrease in perplexity across large datasets. (Note that the “large dataset” criteria is required. It’s certainly possible to demonstrate improvements in restricted cases, but no one is really interested in the restricted case unless you have a very specialized task. In which case, sure, specializations make sense — ChessGPT being an obvious example where tokenization just harms learning.) So the tradeoff isn’t the large context window, but rather the desire to have a deep understanding of a massive amount of data. Specialized models will always have a place as a small component of the whole, but suggesting that this is a problem solved by superior architectures seems a little bit of a stretch. I think what’s going on here is that OpenAI spent a lot of time giving feedback to their model about specific use cases, and ROT-13 was obscure enough (both in usage and in the data) that its performance is limited. I’d bet that if OpenAI did a few rounds of RL on this objective, the model would perform as well as its cousins.
  • roperj 4y ago
    > This isn’t actually true, and is a persistent myth. Or rather, you should back up the claims with evidence. How about the reply posted 5 minutes before yours. Appears like pretty good evidence.
    • sillysaurusx 4y ago
      I’m not sure which one you’re referring to, but none of them show evidence. The reason it can’t play wordle is likely the same as why it has trouble with ROT13: lack of training. https://news.ycombinator.com/item?id=33915690 https://news.ycombinator.com/item?id=33915690 even points out a counterexample demonstrating that it can do character level tasks just fine.
  • yunyu 4y ago
    The ByT5 paper shows significantly better resiliency to typos and noise in internet scale benchmarks, as well as better reasoning capabilities with tasks like punctuation and spacing correction: https://arxiv.org/pdf/2105.13626.pdf https://arxiv.org/pdf/2105.13626.pdf The analogy doesn’t hold because while you might not parse specific characters by default, you are trained on the character level and you can switch to that parsing mode. This simply is not possible for LLMs without hacks like inserting spaces, which then suffers from poor performance due to this being rare in pretraining. The reason byte level models aren’t as popular is because inference speed is significantly slower for a given completion length, and training for the same context window in characters is significantly more expensive I say this as someone who’s deployed ByT5 vs normal T5 in prod, having found this to be true in real world use cases. No reasonable amount of fine tuning can save GPT from performing poorly on a space insertion task (“WHOLEFDSMKT” -> “Whole Foods Market”), while ByT5 just works.
• bilsbie 4y ago
  So how does it do as good as it did on this task? It ended up getting some of the words.
  • ravi-delia 4y ago
    It sees the clusters broken up somehow, they're just not on character boundaries. So it can pick up bits of the pattern, but it's fighting uphill. Same reason it's bad at rhyming- the information isn't readily available to it in the same way
• jaybrendansmith 4y ago
  I think the type I/II thinking does make sense. I tried the classic baseball bat riddle, which is a great riddle because type I thinking fails: A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? GPT chat answered: The bat costs $1.00 + $0.10 = $<<1.00+0.10=1.10>>1.10. The ball costs $1.10 - $1.00 = $<<1.10-1.00=0.10>>0.10. Answer: \boxed{0.10}.
• pa7x1 4y ago
  This is not it or not sufficient explanation. chatGPT is able to read text with letters purposefully scrambled. Like those texts that show how humans have an innate ability to read text with letters misplaced. chatGPT can understand the text, figure out what is going on and produce text of the same type. I tried it, precisely to understand if it was working with words as its token. Interestingly, it cannot produce text with the order or characters reversed or guess that's what is going on. And if you explain it, it's able to produce code that would reverse the characters but is still not able to apply it for long sentences.
  • nverno 4y ago
    Did you try entering one of those scrambled sentences into the tokenizer? It's not tokenizing words exactly- the doc says roughly ~100 tokens => 75 words.
    • pa7x1 4y ago
      I just tried out of curiosity. The jumbled words get broken up into different tokens, the unjumbled words do not get split. https://imgur.com/a/a3zmkIv https://imgur.com/a/a3zmkIv
• ookdatnog 4y ago
  Simpler and, I'd wager, wrong. It seems to me that this stems from fundamental limitations of current machine learning tech. As far as I'm aware, not a single neural-net based AI has displayed the ability to do "System 2" style thinking well, regardless of their training set. We have AIs that are good at System 2 thinking, that is, symbolic AI. But they don't do System 1 thinking at all. We haven't managed to integrate the two meaningfully and I don't think anyone has a clue how to do it. It's not enough to just have the ML-based system make calls into a symbolic AI when it needs to do reasoning, that's like you have a child who doesn't know how to multiply, so you give them a calculator and declare "this child now knows how to multiply".
  • yunyu 4y ago
    The claim that current LLMs are not that good at deep thinking might be true in general, but it is definitely not the best or correct explanation for the rot13 task being described in the article (or other simple character manipulation tasks that can’t be represented at the subword token level, including base64 or arithmetic)