3 ms·

This is why the difference between "truth" and "grammar" is relevant: because if they are instances of the same thing, then processing one is equivalent to proc

This is why the difference between "truth" and "grammar" is relevant: because if they are instances of the same thing, then processing one is equivalent to processing the other. Thus, if we can find a way to move from "grammar" to "truth", then we can say that ChatGPT has a truth model. (Which I still don't really know why it's needed for reasoning.) The separation into "structure" and "truth" sounds likes it makes a difference. But when I think about it, I'm unable to clearly delineate them. When I proofread something, and I find wrong grammar structure, I correct it. When I find a wrong truth structure like contradiction (internal or against what I know), I can correct it, too. In the end, I use the same tools: the document under analysis, and the set of facts making up the template against which I'm checking. This makes me think that "grammar structure" and "truth structure", again, don't differ in quality. Thus, ChatGPT's knowledge about grammar can be treated as knowledge about its world.

Jensson 4y ago
> Thus, if we can find a way to move from "grammar" to "truth", then we can say that ChatGPT has a truth model. (Which I still don't really know why it's needed for reasoning.) Yes, but you would need to find that first, we don't have that, until you find that we don't have a truth model. Maybe it is enough to train a model like ChatGPT on factful texts and run it as a fact checker, we don't know, it could be worth trying but until we have done it we don't have a truth checking model. Edit: > Which I still don't really know why it's needed for reasoning We don't know why anything is needed for reasoning. But it is pretty easy to guess that having a fact checking model instead of just a text structure evaluator would greatly improve reasoning skills in a model. Maybe a fact checker is all ChatGPT would need to replace most knowledge workers? That seems very probable to me, so at least I think that a fact checker layer is needed for higher level reasoning.
- rhn_mk1 4y ago
  > factful texts That might be the problem here. Humans can't agree on facts apart from some very basic ones, so I'd argue we don't have truly "factful texts" at all. We have our own definitions of truth, and they are all built on flawed data. If factful texts are needed to have a model of truth, then humans don't have one, either. Or at least then humans have imperfect models of truth informed by the training data, and, as far as I know, this fits ChatGPT as well. > pretty easy to guess that having a fact checking model instead of just a text structure evaluator would greatly improve reasoning I read this as "feeding the model better data will improve things" - after all, I just explained that I see fact checking and structure evaluation as effectively the same kind of a task. And with that in mind, I do agree with this guess :)
  - Jensson 4y ago
    > I just explained that I see fact checking and structure evaluation as effectively the same kind of a task. The tasks correlate, which is why ChatGPT doesn't always spout nonsense, but they aren't the same. Facts are a tiny part of the structure so even though structure correlates a bit with facts most of the structure just has to do with grammar, so it prefers grammatically nice text over correct text. Similarly fact checking can mostly ignore grammar and structure, as long as the relationships of items is clear you can just look at the logic. I think that looking at just structure gives a way too weak signal here, which is why ChatGPT is so dumb even though it has so much data and compute. Imagine if instead we trained the structural model to extract the non-structural logical parts, and then train another model on just the logic, that would lessen the logical burden of the structure model while also give us a separate and very pure logic signal that can be used as a truth model. That might be all we need for general intelligence, but I doubt we will get there without such a separation. So yeah, I get what you are saying, but as I said structural checks correlates a bit with truth checks, but not nearly enough to call them the same. A structural check would prefer a presidential speech full of lies over the messy forum comment debunking all those lies, so it isn't a truth model it is a structure model, even though those correlate a bit.
    - rhn_mk1 4y ago
      I don't mean that fact checking correlates to structure. I mean that there is no definition of "structure" I can think of that is not entirely dictated by facts (knowledge/training data). Grammatical structure is stretched on facts of grammar, for example. So I don't see the point of saying "facts are a tiny part of structure" because "facts" as you use them guide some sort of structure, too. Just different than the grammatical structure. You say "ChatGPT is dumb" meaning that it says untrue things. But that doesn't mean that it doesn't have a truth model. It does mean that its truth model (if it has one) doesn't match yours. This is not surprising. Garbage in, garbage out. It also says nothing about its reasoning capability. After all, you can't reason much good out of bad data. So while you might be truth that some internal structure changes might improve its truth output, I also see no reason why improving its data wouldn't improve its truthfulness.

Jensson 4y ago

> Thus, if we can find a way to move from "grammar" to "truth", then we can say that ChatGPT has a truth model. (Which I still don't really know why it's needed for reasoning.) Yes, but you would need to find that first, we don't have that, until you find that we don't have a truth model. Maybe it is enough to train a model like ChatGPT on factful texts and run it as a fact checker, we don't know, it could be worth trying but until we have done it we don't have a truth checking model. Edit: > Which I still don't really know why it's needed for reasoning We don't know why anything is needed for reasoning. But it is pretty easy to guess that having a fact checking model instead of just a text structure evaluator would greatly improve reasoning skills in a model. Maybe a fact checker is all ChatGPT would need to replace most knowledge workers? That seems very probable to me, so at least I think that a fact checker layer is needed for higher level reasoning.

rhn_mk1 4y ago
> factful texts That might be the problem here. Humans can't agree on facts apart from some very basic ones, so I'd argue we don't have truly "factful texts" at all. We have our own definitions of truth, and they are all built on flawed data. If factful texts are needed to have a model of truth, then humans don't have one, either. Or at least then humans have imperfect models of truth informed by the training data, and, as far as I know, this fits ChatGPT as well. > pretty easy to guess that having a fact checking model instead of just a text structure evaluator would greatly improve reasoning I read this as "feeding the model better data will improve things" - after all, I just explained that I see fact checking and structure evaluation as effectively the same kind of a task. And with that in mind, I do agree with this guess :)
- Jensson 4y ago
  > I just explained that I see fact checking and structure evaluation as effectively the same kind of a task. The tasks correlate, which is why ChatGPT doesn't always spout nonsense, but they aren't the same. Facts are a tiny part of the structure so even though structure correlates a bit with facts most of the structure just has to do with grammar, so it prefers grammatically nice text over correct text. Similarly fact checking can mostly ignore grammar and structure, as long as the relationships of items is clear you can just look at the logic. I think that looking at just structure gives a way too weak signal here, which is why ChatGPT is so dumb even though it has so much data and compute. Imagine if instead we trained the structural model to extract the non-structural logical parts, and then train another model on just the logic, that would lessen the logical burden of the structure model while also give us a separate and very pure logic signal that can be used as a truth model. That might be all we need for general intelligence, but I doubt we will get there without such a separation. So yeah, I get what you are saying, but as I said structural checks correlates a bit with truth checks, but not nearly enough to call them the same. A structural check would prefer a presidential speech full of lies over the messy forum comment debunking all those lies, so it isn't a truth model it is a structure model, even though those correlate a bit.
  - rhn_mk1 4y ago
    I don't mean that fact checking correlates to structure. I mean that there is no definition of "structure" I can think of that is not entirely dictated by facts (knowledge/training data). Grammatical structure is stretched on facts of grammar, for example. So I don't see the point of saying "facts are a tiny part of structure" because "facts" as you use them guide some sort of structure, too. Just different than the grammatical structure. You say "ChatGPT is dumb" meaning that it says untrue things. But that doesn't mean that it doesn't have a truth model. It does mean that its truth model (if it has one) doesn't match yours. This is not surprising. Garbage in, garbage out. It also says nothing about its reasoning capability. After all, you can't reason much good out of bad data. So while you might be truth that some internal structure changes might improve its truth output, I also see no reason why improving its data wouldn't improve its truthfulness.