3 ms·

Well, it's a shame, because I thought you were really on to something. No, I don't see a qualitative difference. There is a huge quantitative difference becaus

Well, it's a shame, because I thought you were really on to something. No, I don't see a qualitative difference. There is a huge quantitative difference because the facts that describe the world overwhelm the facts that make up language, but that doesn't obviously make a connection to reasoning clear.

Jensson 4y ago
Extreme quantitative differences results in qualitative difference. Anyway, just realized that you side-tracked the whole conversation by moving it into comparing them to each other. That discussion isn't relevant here. We are talking about a truth model directly, not whether it resembles a truth model in some ways. A spellchecker and a truth checker are both evaluators, but that doesn't mean that a spellchecker is a truth checker or vice versa. Therefore it is wrong to say that ChatGPT has a truth model, since it doesn't evaluate what we call truth, it evaluates the structure of the text not the truth of the text. Evaluators returns a boolean or some number, but that "true" or "false" are just programmatic terms, that isn't want humans normally consider trueness of falseness. Otherwise you would call any evaluator a "truth checker", since they check if the result is a true boolean. You could do that, but that doesn't make sense for this discussion, since we care about truth and false in the world, not in programmatic terms. Edit: As for why this is relevant for ChatGPT smartness. It is because we know how to evaluate text structure now, that is the new thing with ChatGPT. But we still don't know how to evaluate the truth of a text, ChatGPT can evaluate the structure but we have no way of turning a structure evaluator into a truth evaluator. We need another kind of model to start evaluating the truth of statements, and until then AI's like ChatGPT will continue to make up facts, since they have no idea what a fact is.
- rhn_mk1 4y ago
  This is why the difference between "truth" and "grammar" is relevant: because if they are instances of the same thing, then processing one is equivalent to processing the other. Thus, if we can find a way to move from "grammar" to "truth", then we can say that ChatGPT has a truth model. (Which I still don't really know why it's needed for reasoning.) The separation into "structure" and "truth" sounds likes it makes a difference. But when I think about it, I'm unable to clearly delineate them. When I proofread something, and I find wrong grammar structure, I correct it. When I find a wrong truth structure like contradiction (internal or against what I know), I can correct it, too. In the end, I use the same tools: the document under analysis, and the set of facts making up the template against which I'm checking. This makes me think that "grammar structure" and "truth structure", again, don't differ in quality. Thus, ChatGPT's knowledge about grammar can be treated as knowledge about its world.
  - Jensson 4y ago
    > Thus, if we can find a way to move from "grammar" to "truth", then we can say that ChatGPT has a truth model. (Which I still don't really know why it's needed for reasoning.) Yes, but you would need to find that first, we don't have that, until you find that we don't have a truth model. Maybe it is enough to train a model like ChatGPT on factful texts and run it as a fact checker, we don't know, it could be worth trying but until we have done it we don't have a truth checking model. Edit: > Which I still don't really know why it's needed for reasoning We don't know why anything is needed for reasoning. But it is pretty easy to guess that having a fact checking model instead of just a text structure evaluator would greatly improve reasoning skills in a model. Maybe a fact checker is all ChatGPT would need to replace most knowledge workers? That seems very probable to me, so at least I think that a fact checker layer is needed for higher level reasoning.
    - rhn_mk1 4y ago
      > factful texts That might be the problem here. Humans can't agree on facts apart from some very basic ones, so I'd argue we don't have truly "factful texts" at all. We have our own definitions of truth, and they are all built on flawed data. If factful texts are needed to have a model of truth, then humans don't have one, either. Or at least then humans have imperfect models of truth informed by the training data, and, as far as I know, this fits ChatGPT as well. > pretty easy to guess that having a fact checking model instead of just a text structure evaluator would greatly improve reasoning I read this as "feeding the model better data will improve things" - after all, I just explained that I see fact checking and structure evaluation as effectively the same kind of a task. And with that in mind, I do agree with this guess :)
      - Jensson 4y ago
        > I just explained that I see fact checking and structure evaluation as effectively the same kind of a task. The tasks correlate, which is why ChatGPT doesn't always spout nonsense, but they aren't the same. Facts are a tiny part of the structure so even though structure correlates a bit with facts most of the structure just has to do with grammar, so it prefers grammatically nice text over correct text. Similarly fact checking can mostly ignore grammar and structure, as long as the relationships of items is clear you can just look at the logic. I think that looking at just structure gives a way too weak signal here, which is why ChatGPT is so dumb even though it has so much data and compute. Imagine if instead we trained the structural model to extract the non-structural logical parts, and then train another model on just the logic, that would lessen the logical burden of the structure model while also give us a separate and very pure logic signal that can be used as a truth model. That might be all we need for general intelligence, but I doubt we will get there without such a separation. So yeah, I get what you are saying, but as I said structural checks correlates a bit with truth checks, but not nearly enough to call them the same. A structural check would prefer a presidential speech full of lies over the messy forum comment debunking all those lies, so it isn't a truth model it is a structure model, even though those correlate a bit.

Jensson 4y ago

Extreme quantitative differences results in qualitative difference. Anyway, just realized that you side-tracked the whole conversation by moving it into comparing them to each other. That discussion isn't relevant here. We are talking about a truth model directly, not whether it resembles a truth model in some ways. A spellchecker and a truth checker are both evaluators, but that doesn't mean that a spellchecker is a truth checker or vice versa. Therefore it is wrong to say that ChatGPT has a truth model, since it doesn't evaluate what we call truth, it evaluates the structure of the text not the truth of the text. Evaluators returns a boolean or some number, but that "true" or "false" are just programmatic terms, that isn't want humans normally consider trueness of falseness. Otherwise you would call any evaluator a "truth checker", since they check if the result is a true boolean. You could do that, but that doesn't make sense for this discussion, since we care about truth and false in the world, not in programmatic terms. Edit: As for why this is relevant for ChatGPT smartness. It is because we know how to evaluate text structure now, that is the new thing with ChatGPT. But we still don't know how to evaluate the truth of a text, ChatGPT can evaluate the structure but we have no way of turning a structure evaluator into a truth evaluator. We need another kind of model to start evaluating the truth of statements, and until then AI's like ChatGPT will continue to make up facts, since they have no idea what a fact is.

rhn_mk1 4y ago
This is why the difference between "truth" and "grammar" is relevant: because if they are instances of the same thing, then processing one is equivalent to processing the other. Thus, if we can find a way to move from "grammar" to "truth", then we can say that ChatGPT has a truth model. (Which I still don't really know why it's needed for reasoning.) The separation into "structure" and "truth" sounds likes it makes a difference. But when I think about it, I'm unable to clearly delineate them. When I proofread something, and I find wrong grammar structure, I correct it. When I find a wrong truth structure like contradiction (internal or against what I know), I can correct it, too. In the end, I use the same tools: the document under analysis, and the set of facts making up the template against which I'm checking. This makes me think that "grammar structure" and "truth structure", again, don't differ in quality. Thus, ChatGPT's knowledge about grammar can be treated as knowledge about its world.
- Jensson 4y ago
  > Thus, if we can find a way to move from "grammar" to "truth", then we can say that ChatGPT has a truth model. (Which I still don't really know why it's needed for reasoning.) Yes, but you would need to find that first, we don't have that, until you find that we don't have a truth model. Maybe it is enough to train a model like ChatGPT on factful texts and run it as a fact checker, we don't know, it could be worth trying but until we have done it we don't have a truth checking model. Edit: > Which I still don't really know why it's needed for reasoning We don't know why anything is needed for reasoning. But it is pretty easy to guess that having a fact checking model instead of just a text structure evaluator would greatly improve reasoning skills in a model. Maybe a fact checker is all ChatGPT would need to replace most knowledge workers? That seems very probable to me, so at least I think that a fact checker layer is needed for higher level reasoning.
  - rhn_mk1 4y ago
    > factful texts That might be the problem here. Humans can't agree on facts apart from some very basic ones, so I'd argue we don't have truly "factful texts" at all. We have our own definitions of truth, and they are all built on flawed data. If factful texts are needed to have a model of truth, then humans don't have one, either. Or at least then humans have imperfect models of truth informed by the training data, and, as far as I know, this fits ChatGPT as well. > pretty easy to guess that having a fact checking model instead of just a text structure evaluator would greatly improve reasoning I read this as "feeding the model better data will improve things" - after all, I just explained that I see fact checking and structure evaluation as effectively the same kind of a task. And with that in mind, I do agree with this guess :)
    - Jensson 4y ago
      > I just explained that I see fact checking and structure evaluation as effectively the same kind of a task. The tasks correlate, which is why ChatGPT doesn't always spout nonsense, but they aren't the same. Facts are a tiny part of the structure so even though structure correlates a bit with facts most of the structure just has to do with grammar, so it prefers grammatically nice text over correct text. Similarly fact checking can mostly ignore grammar and structure, as long as the relationships of items is clear you can just look at the logic. I think that looking at just structure gives a way too weak signal here, which is why ChatGPT is so dumb even though it has so much data and compute. Imagine if instead we trained the structural model to extract the non-structural logical parts, and then train another model on just the logic, that would lessen the logical burden of the structure model while also give us a separate and very pure logic signal that can be used as a truth model. That might be all we need for general intelligence, but I doubt we will get there without such a separation. So yeah, I get what you are saying, but as I said structural checks correlates a bit with truth checks, but not nearly enough to call them the same. A structural check would prefer a presidential speech full of lies over the messy forum comment debunking all those lies, so it isn't a truth model it is a structure model, even though those correlate a bit.