3 ms·

Sometimes it looks like the computationalists are trying to sneak back into the room while no-one is looking. There does seem to be quite a lot of independent

by Lerc 2y ago

Sometimes it looks like the computationalists are trying to sneak back into the room while no-one is looking. There does seem to be quite a lot of independent ad-hoc efforts making custom notations for C-O-T. I feel like we're in a period similar to just after the first programming languages and compilers were invented but regular expressions were yet to come. In a way that's quite exciting, its another little Cambrian explosion. I don't think it will be a panacea though. In my observations of failures of reasoning in LLMs, a lot of the problem isn't that they fail to follow logical steps but that they fail to notice the presence of implied premises completely. Chain of Thought is good for spotting the wrong reasoning, but not for spotting that the problem is not the one that it appears at first glance.

selfhoster11 2y ago
I genuinely think that to some extent, it’s an engineering problem. If you collect enough sufficiently high quality chains of thought with a structured notation that avoids the reversal curse, you can make a dent in the problem. Or it might be just tilting at windmills I guess, but I would want to try it at least.
mdp2021 2y ago
> the problem isn't that they fail to follow logical steps but that they fail to notice the presence of implied premises completely Could you please explain more clearly what you noticed? Can you find an example?
- Lerc 2y ago
  One example is the three killers problem There are three killers in a room, someone enters the room and kills one of them, how many killers are now in the room? Apart from the basic reasoning errors of small models (where they come up with all sorts of numbers), larger models that fail, do so because they fail to take into account that killing makes a killer. Some models succeed at this but it is unclear if this is because they are familiar with this specific problem. The model has to make the distinction between that and There are three fish in a room, someone enters the room and kills one of them, how many fish are now in the room? or There are three fish in a room, someone enters the room and kills one of them, how many killers are now in the room? In these examples the word kills is pushed in one direction or another, but when it serves double duty in the original problem it can notice one premise and miss the other. It's interesting task in frustration to try and get a model that gets it wrong to COT out of that particular error.
  - throwthrowuknow 2y ago
    Yes, the problem would be due to attention being limited to the context. Reasoning requires constructing a model that can be referred to and updated. This is what CoT attempts to do but it is limited when expressed in natural language and being append only.
- svnt 2y ago
  An example is where the link between the ideas is not present in the prompt. E.g. an oversimple example is: It’s going to rain tomorrow so you should take an umbrella. If someone doesn’t understand the relationship between an umbrella and keeping dry — the implied premise — they won’t understand why they should bring one and the statement would be puzzling. We talk about this often as someone or something “not knowing what an x is” where in this case “knowing” means in a spatial or logical sense, not just a linguistic sense. LLMs do not have this layer of understanding but they can fake it in short interactions.
  - deleted 2y ago
    [deleted]
  - Sakos 2y ago
    Otherwise known as the idea of qualia (https://en.wikipedia.org/wiki/Qualia https://en.wikipedia.org/wiki/Qualia). There's no indication that LLMs possess anything like it, so there is no "knowing". I'm also not sure we can ever imbue an LLM with qualia without rethinking how we build LLMs and how they interact and learn from their environment.
    - svnt 2y ago
      I think it is like qualia in the sense that it can be something known experientially, but spatial and logical reasoning are things that transcend subjectivity. No doubt LLMs do not have qualia as humans do, but I also think that the diversity of qualia present in humans is generally unqualified and underestimated at present.
      - Sakos 2y ago
        You can't have spatial and logical reasoning without qualia.
        svnt 2y ago
        I don’t know if that is true, unless your definition of qualia is so loose that it includes (or emerges from) basic data representations such as vectors comprised of weights and vertices, in which case I think LLMs could perhaps be argued to have basic qualia. There is often this attempt to draw a bright line on one feature or characteristic between humans and animals, or humans and AI, but I think that is a mistake based on an attempt to simplify the argument through reduction.

by Lerc 2y ago

selfhoster11 2y ago

I genuinely think that to some extent, it’s an engineering problem. If you collect enough sufficiently high quality chains of thought with a structured notation that avoids the reversal curse, you can make a dent in the problem. Or it might be just tilting at windmills I guess, but I would want to try it at least.

mdp2021 2y ago

> the problem isn't that they fail to follow logical steps but that they fail to notice the presence of implied premises completely Could you please explain more clearly what you noticed? Can you find an example?

Lerc 2y ago
One example is the three killers problem There are three killers in a room, someone enters the room and kills one of them, how many killers are now in the room? Apart from the basic reasoning errors of small models (where they come up with all sorts of numbers), larger models that fail, do so because they fail to take into account that killing makes a killer. Some models succeed at this but it is unclear if this is because they are familiar with this specific problem. The model has to make the distinction between that and There are three fish in a room, someone enters the room and kills one of them, how many fish are now in the room? or There are three fish in a room, someone enters the room and kills one of them, how many killers are now in the room? In these examples the word kills is pushed in one direction or another, but when it serves double duty in the original problem it can notice one premise and miss the other. It's interesting task in frustration to try and get a model that gets it wrong to COT out of that particular error.
- throwthrowuknow 2y ago
  Yes, the problem would be due to attention being limited to the context. Reasoning requires constructing a model that can be referred to and updated. This is what CoT attempts to do but it is limited when expressed in natural language and being append only.
svnt 2y ago
An example is where the link between the ideas is not present in the prompt. E.g. an oversimple example is: It’s going to rain tomorrow so you should take an umbrella. If someone doesn’t understand the relationship between an umbrella and keeping dry — the implied premise — they won’t understand why they should bring one and the statement would be puzzling. We talk about this often as someone or something “not knowing what an x is” where in this case “knowing” means in a spatial or logical sense, not just a linguistic sense. LLMs do not have this layer of understanding but they can fake it in short interactions.
- deleted 2y ago
  [deleted]
- Sakos 2y ago
  Otherwise known as the idea of qualia (https://en.wikipedia.org/wiki/Qualia https://en.wikipedia.org/wiki/Qualia). There's no indication that LLMs possess anything like it, so there is no "knowing". I'm also not sure we can ever imbue an LLM with qualia without rethinking how we build LLMs and how they interact and learn from their environment.
  - svnt 2y ago
    I think it is like qualia in the sense that it can be something known experientially, but spatial and logical reasoning are things that transcend subjectivity. No doubt LLMs do not have qualia as humans do, but I also think that the diversity of qualia present in humans is generally unqualified and underestimated at present.
    - Sakos 2y ago
      You can't have spatial and logical reasoning without qualia.
      - svnt 2y ago
        I don’t know if that is true, unless your definition of qualia is so loose that it includes (or emerges from) basic data representations such as vectors comprised of weights and vertices, in which case I think LLMs could perhaps be argued to have basic qualia. There is often this attempt to draw a bright line on one feature or characteristic between humans and animals, or humans and AI, but I think that is a mistake based on an attempt to simplify the argument through reduction.