3 ms·
but the model doesn't operate on token directly, right? all operations are happening in the embedding space, so these tokens get mapped into manifold and one o
by slt2021 2y ago
but the model doesn't operate on token directly, right?
all operations are happening in the embedding space, so these tokens get mapped into manifold and one of the dimensions could be representative of fact/trivia ?
- ottaborra 2y agotangent: any reason to assume it gets mapped to a manifold rather than something that is not?
- youoy 2y agoI think "manifolds" in AI are not the same as actual smooth manifolds. For starters I would not expect them to have locally the same dimension across the whole dataset.
- ottaborra 2y agoSomething to chew on for me. But what is a manifold then if not a topological space that is locally the same as R^(some dimension) ?
- youoy 2y agoWhat I meant is that I can imagine cases where some part of the dataset may look like R2 and then colapse to have a spike that looks like R1, so it is not a standard manifold where all of it has the same dimension. Appart from that, these "manifolds" have noise, so that is another difference with the standard manifolds.