3 ms·

Why don't people here on HN understand that LLMs never see ASCII or other raw characters as input? Expecting spelling, rhyming, arithmetic or other character o

Why don't people here on HN understand that LLMs never see ASCII or other raw characters as input? Expecting spelling, rhyming, arithmetic or other character oriented responses will always yield disappointing results.

mdp2021 1y ago
And which other objectual ideas cannot they instance? Their task is to check, for all important mental activities - world simulation, "telling yourself reliable stories: that is what intelligence is" (Prof. Patrick Winston).
arduanika 1y ago
Because the damn things are marketed under the word "intelligence". That word used to mean something.
- sdenton4 1y ago
  It's an umwelt problem. Bats think we're idiots because we don't hear ultrasonic sound, and thus can't echolocate. And we call the LLMs idiots because they consume tokenized inputs, and don't have access to the raw character stream.
  - arduanika 1y ago
    If you open your mind up too far, your brain will fall out. LLMs are not intelligence. There's not some groovy sense in which we and they are both intelligent, just thinking on a different wavelength. Machines do not think. We are inundated with this anthropomorphic chatter about them, and need to constantly deprogram ourselves.
  - mdp2021 1y ago
    > we call the LLMs "Dangerous", because they lead into thinking they do the advanced of what they don't do basically.
  - nessbot 1y ago
    Do bat's know what senses humans have? Or have the concept of what a human is compared to other organisms or moving objects? What is this analogy?
    - sdenton4 1y ago
      Yeah, I wrote this in a bit too short a hand to meet the critics where they sit... There's an immense history of humans studying animal intelligence, which has tended pretty uniformly to find that animals are more intelligent than we previously thought at any given point in time. There's a very long history of badly designed experiments which surface 'false negative' results, and are eventually overturned. A common favor in these experiments is that the design assumes that animals have the same prescriptions and/or interests as humans. (For example, trying to do operant conditioning using a color cue with animals who can't perceive the colors. Or tasks that are easy of you happen to have approachable thumbs... That kind of thing.) Experiments eventually come along which better meet the animals where they are, and find true positive results, and our estimation of the intelligence of animals creeps slightly higher. In other words, humans, in testing intelligence, have a decided bias towards only acknowledging intelligence which is distinctly human, and failing to take into account umwelt. LLMs have a very different umwelt than we do. If they fail a test which doesn't take that umwelt into account, it doesn't indicate non-intelligence. It is, in fact, very hard to prove non-intelligence, because intelligence is poorly defined. And we have tended consistently to make the definition loftier whenever we're threatened with not being special anymore.
  - zeroonetwothree 1y ago
    Pretty sure bats don’t think about human intelligence at all.
- tocs3 1y ago
  What did it used to mean? I was under the impression that it has always be a little vague.
  - arduanika 1y ago
    Sure. Language is squishy, and psychometrics is hard. Nevertheless... "Intelligence" refers to a basket of different capabilities. Some of them are borderline cases that are hard to define. The stuff that GPT-5 failed to do here is not. Things like knowing what a question means, knowing what you know and don't, counting a single digit number of items, or replying with humility if you get stuck -- these are fairly central examples of what a very, very basic intelligence should entail.
asadotzler 1y ago
We do understand. We don't think that's okay. If a model cannot manage character level consideration, that's a serious flaw that's got potential to lead to an immeasurable number of failure states. "Duh, of course it can't count" is not the best look for a bot whose author tells us it's got PhD-level skill.
- zahlman 1y ago
  I do think it's "okay". After all, it's clear that fixing it would require a fundamentally different approach. I just also think it's a reason to mock people who don't try to understand those limitations and get way ahead of themselves hyping up the technology. The entire point of this exercise is to refute the claim that LLMs are a step towards AGI, even given "agency". And we should be happy that they aren't — because supposing that AGI is possible, the way that we currently treat LLMs shows that we as a species are nowhere near ready for the consequences of creating it.
- johnfn 1y ago
  Can you enumerate some of these "immeasurable number of failure states"? For me it starts and stops at "can't count letters in a word". That hardly seems catastrophic. All I have to do is turn on thinking mode and the error goes away. https://chatgpt.com/share/6897e630-77f0-800c-a9bf-30d9c0e27107 https://chatgpt.com/share/6897e630-77f0-800c-a9bf-30d9c0e271...
- mikewarot 1y ago
  So, if an AI can just spit out the cure for cancer, but spells some things wrong, it's not intelligent? You think all PhD candidates have perfect spelling? I'd wager most of them re-read their dissertation and edit it, over and over, a process that most LLMs don't have the luxury of doing. We'd have to give up all the efficiency of tokenizing, re-train a model (a much less optimum model) for at least twice as long to get anywhere near the same results for one that just spits out ASCII.
UncleMeat 1y ago
"LLMs are cool tools with clear limitations" is not the narrative being pushed by the bosses and boosters. "LLMs are literal magic that will replace large portions of the workforce and be a bigger revolution than fire" is what they are saying.
cute_boi 1y ago
The only issue is they shouldn't call it PHD level intelligence when they can't do simple task like this.

mdp2021 1y ago

And which other objectual ideas cannot they instance? Their task is to check, for all important mental activities - world simulation, "telling yourself reliable stories: that is what intelligence is" (Prof. Patrick Winston).

arduanika 1y ago

Because the damn things are marketed under the word "intelligence". That word used to mean something.

sdenton4 1y ago
It's an umwelt problem. Bats think we're idiots because we don't hear ultrasonic sound, and thus can't echolocate. And we call the LLMs idiots because they consume tokenized inputs, and don't have access to the raw character stream.
- arduanika 1y ago
  If you open your mind up too far, your brain will fall out. LLMs are not intelligence. There's not some groovy sense in which we and they are both intelligent, just thinking on a different wavelength. Machines do not think. We are inundated with this anthropomorphic chatter about them, and need to constantly deprogram ourselves.
- mdp2021 1y ago
  > we call the LLMs "Dangerous", because they lead into thinking they do the advanced of what they don't do basically.
- nessbot 1y ago
  Do bat's know what senses humans have? Or have the concept of what a human is compared to other organisms or moving objects? What is this analogy?
  - sdenton4 1y ago
    Yeah, I wrote this in a bit too short a hand to meet the critics where they sit... There's an immense history of humans studying animal intelligence, which has tended pretty uniformly to find that animals are more intelligent than we previously thought at any given point in time. There's a very long history of badly designed experiments which surface 'false negative' results, and are eventually overturned. A common favor in these experiments is that the design assumes that animals have the same prescriptions and/or interests as humans. (For example, trying to do operant conditioning using a color cue with animals who can't perceive the colors. Or tasks that are easy of you happen to have approachable thumbs... That kind of thing.) Experiments eventually come along which better meet the animals where they are, and find true positive results, and our estimation of the intelligence of animals creeps slightly higher. In other words, humans, in testing intelligence, have a decided bias towards only acknowledging intelligence which is distinctly human, and failing to take into account umwelt. LLMs have a very different umwelt than we do. If they fail a test which doesn't take that umwelt into account, it doesn't indicate non-intelligence. It is, in fact, very hard to prove non-intelligence, because intelligence is poorly defined. And we have tended consistently to make the definition loftier whenever we're threatened with not being special anymore.
- zeroonetwothree 1y ago
  Pretty sure bats don’t think about human intelligence at all.
tocs3 1y ago
What did it used to mean? I was under the impression that it has always be a little vague.
- arduanika 1y ago
  Sure. Language is squishy, and psychometrics is hard. Nevertheless... "Intelligence" refers to a basket of different capabilities. Some of them are borderline cases that are hard to define. The stuff that GPT-5 failed to do here is not. Things like knowing what a question means, knowing what you know and don't, counting a single digit number of items, or replying with humility if you get stuck -- these are fairly central examples of what a very, very basic intelligence should entail.

asadotzler 1y ago

We do understand. We don't think that's okay. If a model cannot manage character level consideration, that's a serious flaw that's got potential to lead to an immeasurable number of failure states. "Duh, of course it can't count" is not the best look for a bot whose author tells us it's got PhD-level skill.

zahlman 1y ago
I do think it's "okay". After all, it's clear that fixing it would require a fundamentally different approach. I just also think it's a reason to mock people who don't try to understand those limitations and get way ahead of themselves hyping up the technology. The entire point of this exercise is to refute the claim that LLMs are a step towards AGI, even given "agency". And we should be happy that they aren't — because supposing that AGI is possible, the way that we currently treat LLMs shows that we as a species are nowhere near ready for the consequences of creating it.
johnfn 1y ago
Can you enumerate some of these "immeasurable number of failure states"? For me it starts and stops at "can't count letters in a word". That hardly seems catastrophic. All I have to do is turn on thinking mode and the error goes away. https://chatgpt.com/share/6897e630-77f0-800c-a9bf-30d9c0e27107 https://chatgpt.com/share/6897e630-77f0-800c-a9bf-30d9c0e271...
mikewarot 1y ago
So, if an AI can just spit out the cure for cancer, but spells some things wrong, it's not intelligent? You think all PhD candidates have perfect spelling? I'd wager most of them re-read their dissertation and edit it, over and over, a process that most LLMs don't have the luxury of doing. We'd have to give up all the efficiency of tokenizing, re-train a model (a much less optimum model) for at least twice as long to get anywhere near the same results for one that just spits out ASCII.

UncleMeat 1y ago

"LLMs are cool tools with clear limitations" is not the narrative being pushed by the bosses and boosters. "LLMs are literal magic that will replace large portions of the workforce and be a bigger revolution than fire" is what they are saying.

cute_boi 1y ago

The only issue is they shouldn't call it PHD level intelligence when they can't do simple task like this.