5 ms·

So the models will no longer be thinking in plain English but some embedding space? Seems not like what you want.

Vampiero 2y ago
Seems exactly like what you want. We don't think in plain English, we _rationalize_ our thoughts into English (or whatever language comes out) but they must be more fundamental than language because language is acquired. Essentially, English is one of many possible encodings of an underlying intuitive, possibly non-symbolic representation.
- ekianjo 2y ago
  > We don't think in plain English That's debatable. Language shapes thoughts much more than you might think. Because you learn concepts from language that you could not imagine by yourself until you learned/read about them, so they are in effect very linked to language.
  - Vampiero 2y ago
    Fair, but there are many categories of languages. For example, I can think in formal logic. I've learned to do that, and surely my brain takes a step-by-step approach to it, but I've also internalized some of it and I don't think that my proficiency with English has anything to do with it. I could have learned the same concepts in any other language, but the end result would be the same. And surely there are many thoughts that can't be expressed purely with words. For example all that is related to qualia. You can think of a color but you can't describe what you see in your mind's eye with words, not in a way that would let a blind person share the same experience. Or try describing "love" without making a similitude. Is love a thought? Or a feeling? Is there a meaningful difference between the two?
  - samiskin 2y ago
    Language is definitely a significant part of thinking, but when I remember how cold it was outside yesterday to figure out if it was colder than today, I'm not bringing words to mind. I'm bringing up some other non-discrete information that I could never precisely encode into words and then factoring that in with the other non-discrete information I'm currently taking in through my senses. Its only after that processing that I encode it as a lossy "It was colder yesterday" statement.
    - deleted 2y ago
      [deleted]
  - idiotsecant 2y ago
    But the thing you learn is not the word 'purple'. You just use the word as the mental scaffolding to build a concept of purple. The word forms a linkage to a deeper embedding, which is further proven by the fact that it's actually slightly different in each mind that has understanding of the concept. This embedded concept is what is doing the work, the word was just the seed of the understanding and a method by which to convey that understanding to others.
  - phkahler 2y ago
    I can also think in images and internal visualizations. Geometric reasoning is also a thing. Musicians can also hear things in their mind - some can write it down, others can play it directly, and in my case I'm not good enough to get it out of my head! In all cases though these thoughts are kind of tied to representations from the real world. Sort of like other languages via different senses. So yeah, how abstract can our thoughts actually be?
  - klausa 2y ago
    You're basically talking about Sapir-Whorf here: https://en.wikipedia.org/wiki/Linguistic_relativity https://en.wikipedia.org/wiki/Linguistic_relativity >The hypothesis is in dispute, with many different variations throughout its history.[2] The strong hypothesis of linguistic relativity, now referred to as linguistic determinism, is that language determines thought and that linguistic categories limit and restrict cognitive categories. This was a claim by some earlier linguists pre-World War II;[3] since then it has fallen out of acceptance by contemporary linguists.
  - numpad0 2y ago
    eh, probably both. Why does it have to be a fight between two schools of thoughts? Thoughts can be across-modal. Some of it can be done in specific language or some could be visual. (universal grammar peoole hates this somehow, it's weird)
- intalentive 2y ago
  Cognitive scientists called it “mentalese”.
drdeca 2y ago
If you mean “not what we want” for safety reasons, I think I agree. If you don’t mean for safety reasons, I’m not sure why.
- miven 2y ago
  In section 2 they briefly mention studies such as [1] that point out that the token outputs of a chain of thought aren't always entirely faithful to the responses of the models I'm not sure whether it wouldn't be more reliable to let the model run on latents and try to train a separate latent-reading explainer module that has at least some approximation of what we want as an explicit optimization objective. Assuming it actually is or has the potential to be better than CoT, from what I gathered from the paper the current results are mostly just more efficient token-wise. [1] https://arxiv.org/abs/2305.04388 https://arxiv.org/abs/2305.04388
- DalasNoin 2y ago
  I was thinking abut safety reasons, but also usability. Seems like a pretty big difference to me if you don't understand the chain of thought. How faithful cot are is another question.
rsrsrs86 2y ago
They have never been thinking. This is important. Predicting the next word is not intelligence.

Seems exactly like what you want. We don't think in plain English, we _rationalize_ our thoughts into English (or whatever language comes out) but they must be more fundamental than language because language is acquired. Essentially, English is one of many possible encodings of an underlying intuitive, possibly non-symbolic representation.

ekianjo 2y ago
> We don't think in plain English That's debatable. Language shapes thoughts much more than you might think. Because you learn concepts from language that you could not imagine by yourself until you learned/read about them, so they are in effect very linked to language.
- Vampiero 2y ago
  Fair, but there are many categories of languages. For example, I can think in formal logic. I've learned to do that, and surely my brain takes a step-by-step approach to it, but I've also internalized some of it and I don't think that my proficiency with English has anything to do with it. I could have learned the same concepts in any other language, but the end result would be the same. And surely there are many thoughts that can't be expressed purely with words. For example all that is related to qualia. You can think of a color but you can't describe what you see in your mind's eye with words, not in a way that would let a blind person share the same experience. Or try describing "love" without making a similitude. Is love a thought? Or a feeling? Is there a meaningful difference between the two?
- samiskin 2y ago
  Language is definitely a significant part of thinking, but when I remember how cold it was outside yesterday to figure out if it was colder than today, I'm not bringing words to mind. I'm bringing up some other non-discrete information that I could never precisely encode into words and then factoring that in with the other non-discrete information I'm currently taking in through my senses. Its only after that processing that I encode it as a lossy "It was colder yesterday" statement.
  - deleted 2y ago
    [deleted]
- idiotsecant 2y ago
  But the thing you learn is not the word 'purple'. You just use the word as the mental scaffolding to build a concept of purple. The word forms a linkage to a deeper embedding, which is further proven by the fact that it's actually slightly different in each mind that has understanding of the concept. This embedded concept is what is doing the work, the word was just the seed of the understanding and a method by which to convey that understanding to others.
- phkahler 2y ago
  I can also think in images and internal visualizations. Geometric reasoning is also a thing. Musicians can also hear things in their mind - some can write it down, others can play it directly, and in my case I'm not good enough to get it out of my head! In all cases though these thoughts are kind of tied to representations from the real world. Sort of like other languages via different senses. So yeah, how abstract can our thoughts actually be?
- klausa 2y ago
  You're basically talking about Sapir-Whorf here: https://en.wikipedia.org/wiki/Linguistic_relativity https://en.wikipedia.org/wiki/Linguistic_relativity >The hypothesis is in dispute, with many different variations throughout its history.[2] The strong hypothesis of linguistic relativity, now referred to as linguistic determinism, is that language determines thought and that linguistic categories limit and restrict cognitive categories. This was a claim by some earlier linguists pre-World War II;[3] since then it has fallen out of acceptance by contemporary linguists.
- numpad0 2y ago
  eh, probably both. Why does it have to be a fight between two schools of thoughts? Thoughts can be across-modal. Some of it can be done in specific language or some could be visual. (universal grammar peoole hates this somehow, it's weird)
intalentive 2y ago
Cognitive scientists called it “mentalese”.

drdeca 2y ago

If you mean “not what we want” for safety reasons, I think I agree. If you don’t mean for safety reasons, I’m not sure why.

miven 2y ago
In section 2 they briefly mention studies such as [1] that point out that the token outputs of a chain of thought aren't always entirely faithful to the responses of the models I'm not sure whether it wouldn't be more reliable to let the model run on latents and try to train a separate latent-reading explainer module that has at least some approximation of what we want as an explicit optimization objective. Assuming it actually is or has the potential to be better than CoT, from what I gathered from the paper the current results are mostly just more efficient token-wise. [1] https://arxiv.org/abs/2305.04388 https://arxiv.org/abs/2305.04388
DalasNoin 2y ago
I was thinking abut safety reasons, but also usability. Seems like a pretty big difference to me if you don't understand the chain of thought. How faithful cot are is another question.

rsrsrs86 2y ago

They have never been thinking. This is important. Predicting the next word is not intelligence.