4 ms·

This is a known design flaw of LLMs. This gets posted once a month for years. It’s disheartening to see very technically adept people still find this exciting.

This is a known design flaw of LLMs. This gets posted once a month for years. It’s disheartening to see very technically adept people still find this exciting. GPT3 has been out for years, why don’t more “hackers” know about tokenizers? Basically, LLMs are “blind”. Fragments of text are converted to tokens, forming something like a big enum of possible tokens. They can’t see spellings; they cannot see the letters. So, they can’t handle things like “how many letters are in Mississippi?” reliably. Due to chat bots running with nonzero temperature, they will sometimes emit a right answer just because the dice rolled in its favor. So if you go try this and get a good answer, that’s not conclusive either. That’s the thing we’re dealing with, that’s how it works, that’s what it is.

educaysean 2y ago
The tweet in question was posted today. The point here isn't to rehash how LLMs can't distinguish letters from tokens. It's to highlight how Google's AI-generated answer will grab a blatantly false fact from the internet and use it as an authoritative source for its answers.
- thatjoeoverthr 2y ago
  No, it doesn’t show that. There is no source in the screenshot. It’s a screenshot of an LLM generated answer, looking exactly how they do. It’s true, however, that someone tweeted this again.
lainga 2y ago
I suppose if I kept seeing Ford Pintos not only driving down the highway, but replacing Ford owners' other models unannounced (for free, at midnight), and some Ford suit patiently explained to me that the gas tank was "a known design flaw of Ford Pintos", that the design of the Pinto was several years old, and how disheartened he was that "more drivers don't know about rear-end collisions", it would dampen my excitement a little.
deleted 2y ago
[deleted]
nickthegreek 2y ago
The news is the top of your Google search results is now worse.
JumpCrisscross 2y ago
> That’s the thing we’re dealing with, that’s how it works, that’s what it is That's not how they're being marketed. That's the disconnect. Gemini is being marketed as competent enough to write ad copy [1], to be trusted with your brand. Yet it will confidently tell you to go eat glue [2]. [1] https://www.thinkwithgoogle.com/marketing-strategies/automation/new-era-of-marketing/ https://www.thinkwithgoogle.com/marketing-strategies/automat... [2] https://x.com/pixelbutts/status/1793387357753999656 https://x.com/pixelbutts/status/1793387357753999656
poszlem 2y ago
Okay. Then how come other LLMs have no such issue with that question?
- thatjoeoverthr 2y ago
  I open ChatGPT, and I ask GPT-4o: “How many Rs are in the word przepraszam?” It answers: “There are three "R"s in the word "przepraszam.”” It’s interesting that you think other LLMs don’t have this issue. I assume you don’t find this technology important. That’s OK and you might be right, but then I don’t know why comment on it.
  - astrange 2y ago
    "Begins with" and "contains" are different questions and it should be better at "begins with". On the other hand, it can't do "ends with" at all because LLMs can't reason backwards without chain of thought. Which it could do, it could put every country's names into its context and then look at those, tokenization issues aside.
    - thatjoeoverthr 2y ago
      It would need a bit more than CoT; in this case probably it would actually need to write code, find a dataset and run the code. ChatGPT (the consumer product, not the model) can do part of this, but the model alone will tend to fail as it simply doesn’t have most spellings. A bigger meta problem is that the model doesn’t know if it knows something or not, so letting a model provide information from its “gut” is catastrophically unreliable. It can’t even know if it should look something up or not, except with a stereotype of itself, which will also be very sloppy. And it’s on top of Google because important people making decisions don’t understand this and think these are “bugs” that will be “fixed”. I use Kagi now
islewis 2y ago
> It’s disheartening to see very technically adept people still find this exciting. I dont think this is what people find amusing. I think people are more so surprised that Google, the trillion dollar company, has pushed an update that could result in such a bad response to a somewhat basic query. No matter how you chop up the technical details, search needs to provide good results to end users- and in this example shown, it's not doing that.
superfrank 2y ago
I don't read this as a "Look how dumb LLMs are", I read this as, "Look at Google destroying their credibility by putting a technology that has known flaws front and center in their search". For the past 3-5 years there've been a growing number of people talking about how Google search is in decline and stuff like this is just fuel on that fire.
add-sub-mul-div 2y ago
It keeps getting posted because it's funny, not because everyone in the world besides you is too stupid to know why it happens.
eagle2com 2y ago
I don't see why some tokens could not get associated with "begins with A". The models are obviously not trained enough on such examples to do it consistently, but I would not say they are "blind".
rsynnott 2y ago
> This gets posted once a month for years. Thing is, LLM advocates/enthusiasts keep saying "it'll get better". That's basically the entire premise of the current LLM bubble; clearly the _current_ stuff is, at best, marginally useful with heavy human supervision, but it's a Jam Tomorrow situation; everything is predicated on it getting better. And it does not get better. And I mean, in one sense, yeah, obviously it does not, but given the amount of money that's being pumped into this stuff, that it is still broken in the same ways that it was always broken is a concern.

educaysean 2y ago

The tweet in question was posted today. The point here isn't to rehash how LLMs can't distinguish letters from tokens. It's to highlight how Google's AI-generated answer will grab a blatantly false fact from the internet and use it as an authoritative source for its answers.

thatjoeoverthr 2y ago
No, it doesn’t show that. There is no source in the screenshot. It’s a screenshot of an LLM generated answer, looking exactly how they do. It’s true, however, that someone tweeted this again.

lainga 2y ago

I suppose if I kept seeing Ford Pintos not only driving down the highway, but replacing Ford owners' other models unannounced (for free, at midnight), and some Ford suit patiently explained to me that the gas tank was "a known design flaw of Ford Pintos", that the design of the Pinto was several years old, and how disheartened he was that "more drivers don't know about rear-end collisions", it would dampen my excitement a little.

deleted 2y ago

[deleted]

nickthegreek 2y ago

The news is the top of your Google search results is now worse.

JumpCrisscross 2y ago

> That’s the thing we’re dealing with, that’s how it works, that’s what it is That's not how they're being marketed. That's the disconnect. Gemini is being marketed as competent enough to write ad copy [1], to be trusted with your brand. Yet it will confidently tell you to go eat glue [2]. [1] https://www.thinkwithgoogle.com/marketing-strategies/automation/new-era-of-marketing/ https://www.thinkwithgoogle.com/marketing-strategies/automat... [2] https://x.com/pixelbutts/status/1793387357753999656 https://x.com/pixelbutts/status/1793387357753999656

poszlem 2y ago

Okay. Then how come other LLMs have no such issue with that question?

thatjoeoverthr 2y ago
I open ChatGPT, and I ask GPT-4o: “How many Rs are in the word przepraszam?” It answers: “There are three "R"s in the word "przepraszam.”” It’s interesting that you think other LLMs don’t have this issue. I assume you don’t find this technology important. That’s OK and you might be right, but then I don’t know why comment on it.
- astrange 2y ago
  "Begins with" and "contains" are different questions and it should be better at "begins with". On the other hand, it can't do "ends with" at all because LLMs can't reason backwards without chain of thought. Which it could do, it could put every country's names into its context and then look at those, tokenization issues aside.
  - thatjoeoverthr 2y ago
    It would need a bit more than CoT; in this case probably it would actually need to write code, find a dataset and run the code. ChatGPT (the consumer product, not the model) can do part of this, but the model alone will tend to fail as it simply doesn’t have most spellings. A bigger meta problem is that the model doesn’t know if it knows something or not, so letting a model provide information from its “gut” is catastrophically unreliable. It can’t even know if it should look something up or not, except with a stereotype of itself, which will also be very sloppy. And it’s on top of Google because important people making decisions don’t understand this and think these are “bugs” that will be “fixed”. I use Kagi now

islewis 2y ago

> It’s disheartening to see very technically adept people still find this exciting. I dont think this is what people find amusing. I think people are more so surprised that Google, the trillion dollar company, has pushed an update that could result in such a bad response to a somewhat basic query. No matter how you chop up the technical details, search needs to provide good results to end users- and in this example shown, it's not doing that.

superfrank 2y ago

I don't read this as a "Look how dumb LLMs are", I read this as, "Look at Google destroying their credibility by putting a technology that has known flaws front and center in their search". For the past 3-5 years there've been a growing number of people talking about how Google search is in decline and stuff like this is just fuel on that fire.

add-sub-mul-div 2y ago

It keeps getting posted because it's funny, not because everyone in the world besides you is too stupid to know why it happens.

eagle2com 2y ago

I don't see why some tokens could not get associated with "begins with A". The models are obviously not trained enough on such examples to do it consistently, but I would not say they are "blind".

rsynnott 2y ago

> This gets posted once a month for years. Thing is, LLM advocates/enthusiasts keep saying "it'll get better". That's basically the entire premise of the current LLM bubble; clearly the _current_ stuff is, at best, marginally useful with heavy human supervision, but it's a Jam Tomorrow situation; everything is predicated on it getting better. And it does not get better. And I mean, in one sense, yeah, obviously it does not, but given the amount of money that's being pumped into this stuff, that it is still broken in the same ways that it was always broken is a concern.