3 ms·
> If the program was trained on a thousand data points saying that the capital of Connecticut is Moscow, the model would encode this “truthfulness” information
by justinpombrio 2y ago
> If the program was trained on a thousand data points saying that the capital of Connecticut is Moscow, the model would encode this “truthfulness” information about that fact, despite it being false.
This isn't true.
You're conflating whether a model (that hasn't been fine tuned) would complete "the capital of Connecticut is ___" with "Moscow", and whether that model contains a bit labeling that fact as "false". (It's not actually stored as a bit, but you get the idea.)
Some sentences that a model learns could be classified as "trivia", and the model learns this category by sentences like "Who needs to know that octopuses have three hearts, that's just trivia". Other sentences a model learns could be classified as "false", and the model learns this category by sentences like "2 + 2 isn't 5". Whether a sentence is "false" isn't particularly important to the model, any more than whether it's "trivia", but it will learn those categories.
There's a pattern to "false" sentences. For example, even if there's no training data directly saying that "the capital of Connecticut is Moscow" is false, there are a lot of other sentences like "Moscow is in Russia" and "Moscow is really far from CT" and "people in Moscow speak Russian", that all together follow the statistical pattern of "false" sentences, so a model could categorize "Moscow is the capital of Connecticut" as "false" even if it's never directly told so.
- RandomLensman 2y agoThat would again be a "statistical" attempt at deciding on it being correct or false - it might or might not succeed depending on the data.
- justinpombrio 2y agoThat's correct on two fronts. First, I put "false" in quotes everywhere for a reason: I'm talking about the sort of thing that people would say is false, not what's actually false. And second, yes, I'm merely claiming that it's in theory learnable (in contrast to the OP's claim), not that it will necessarily be learned.
- RandomLensman 2y agoAm not sure the second part is always true: there might be situations where statistical approaches could be made kind of "infinitely" accurate as far as data is concerned but still represent a complete misunderstanding of the actual situation (aka truth), e.g., layering epicycles on epicycles in a geocentric model of the solar systems. Some data might support a statistical approach other might not even though it might not contain misrepresentations as such.
- dboreham 2y agoThe human feeling you have that what you're doing is not statistical, is false.
- RandomLensman 2y agoBased on what research is that universally true? (Other than base physics like statistical mechanics.)
- whimsicalism 2y agoBase physics is all we need to know it is true. Souls are unphysical and we've had reason to be pretty confident about that for at least a century.
- RandomLensman 2y agoYes, physics determins how phenomena work and aggregate but that doesn't necessarily support the specific claim (and we also don't know "all of physics").
- whimsicalism 2y agoDoesn't support the specific claim that souls don't exist? We know how atoms/waves interact. We have literally no reason to think there is some other soul-based mechanism. Of course, maybe induction is false and gravity will reverse in the next 3 seconds after writing this comment and God will reveal themself to us. We have no justified reason to think otherwise other than the general principle that things behave the way we observe them to and will continue to do so.
- RandomLensman 2y agoI see no need for a soul - you brought it up, not me.
- 2y ago
- whimsicalism 2y agoGood luck philosophically defending this dividing line between 'statistical' and not
- slt2021 2y agobut the model doesn't operate on token directly, right? all operations are happening in the embedding space, so these tokens get mapped into manifold and one of the dimensions could be representative of fact/trivia ?
- ottaborra 2y agotangent: any reason to assume it gets mapped to a manifold rather than something that is not?
- youoy 2y agoI think "manifolds" in AI are not the same as actual smooth manifolds. For starters I would not expect them to have locally the same dimension across the whole dataset.
- ottaborra 2y agoSomething to chew on for me. But what is a manifold then if not a topological space that is locally the same as R^(some dimension) ?
- youoy 2y agoWhat I meant is that I can imagine cases where some part of the dataset may look like R2 and then colapse to have a spike that looks like R1, so it is not a standard manifold where all of it has the same dimension. Appart from that, these "manifolds" have noise, so that is another difference with the standard manifolds.
- Terr_ 2y agoAnother version/interpretation in this "truth" space is whether a model is capturing multi-part correlations with truthy/falsy signals, much like capturing the "sadness" or "sarcasm" of a statement. In other words, a model might have local contextual indicators, but not be able to recognize global hard logical contradictions.