3 ms·

So I am a bit confused about the part where you go k distance out from the centroids. Since there are ~5000 dimensions, in which of those dimensions are we mov

So I am a bit confused about the part where you go k distance out from the centroids. Since there are ~5000 dimensions, in which of those dimensions are we moving k out ? Is the idea you just move, out, in all dimensions such that the final Euclidean distance is k ? Seems that’s how they get multiple samples at those distances. Either way I think it’s more interesting to go out in specific dimensions. Ideally there is a mapping between each dimension and something inherent about the token, like the part where a dimension corresponds with the first word of the token. We went through this discovery phase when we were generating images using autoencoders, same idea, some of those dimensions would correspond to certain features of the image, so moving along them would change the image output in some predictable way. Either way, I think the overall structure of those spaces says something about how the human brain works ( given we invented the language). I’m interested to see if anything neurologic can be derived from those vector embeddings.

panarky 3y ago
> Ideally there is a mapping ... The complexity and interdependence of dimensions within embeddings make it practically impossible to ascribe specific, human-understandable meanings to individual elements or dimensions. This research actually adds to that complexity. Rather than making the meaning of individual dimensions more understandable, it shows that the embedding space has a strange, layered structure. It suggests a peculiar, almost nonsensical organization of concepts at different distances from the central point of typical token embeddings, which doesn't make it easier to pinpoint what each dimension means. It emphasizes that embeddings capture information in a distributed and highly contextual manner. The fact that embeddings for "nokens" can lead to arbitrary or bizarre categorizations when taken out of the typical token zone underscores that embedding dimensions don't have straightforward, easily interpretable meanings.
- kridsdale1 3y ago
  My interpretation is that the nokens are novel (un-coined) coordinates in a categorized zone of the vector field. Future linguistics will plant their flag in noken territory as we need new words. This already happened with “noken”! Noken-space is all the undefined territory full of noise equivalent to the actual visual noise we see in Stable Diffusion when you travel along a vector away from an island of coherence. To put it poetically, concept space is a vast hyperspace sea of random garbage (like unallocated RAM) and there are tiny islands or planets of meaning and value that look like things that matter to humanity. I don’t see the categorizations as being very bizarre. Putting on my amateur anthropology hat, each one described in the paper was clearly an important topic to a “primitive” person. Concerned with survival, people would talk about group dynamics, sharp things, plants/animals, things that look like infections (small flat round yellow white), and places. These are pretty much all you need to talk about. Looking at an English LLM vector space to understand fundamental principles of language is like looking at human DNA and trying to understand the first eukaryotes. It’s been complicated by effectively infinite generations of specializations adding noise. The probing work described here identifies some common principles that survive through the generations. A commenter above wondered if this was discovering something about the nature of the brain. I think it’s the nature of culture. Etymology is the product of culture and history.
mhink 3y ago
> Since there are ~5000 dimensions, in which of those dimensions are we moving k out ? > Is the idea you just move, out, in all dimensions such that the final Euclidean distance is k ? If I understand correctly, the basic idea is that in earlier experiments, they were able to use a relatively-simple technique to come up with what they call a "probe vector" in the embedding space which represents "the property of a word starting with <letter>". For any given token, the authors established that it was 98% probable that the token's embedding vector would be closer (by cosine similarity) to the "probe vector" representing the first letter of that word than any other probe vector. This is shown in the first graph of the section "A puzzling discovery". With that in mind, the diagram below that should start making more sense: "emb" is a particular token's embedding vector and "probe" is the probe vector for the token's first letter. "emb_proj" is the projection of "emb" onto "probe". What they're doing is tweaking the network weights by subtracting multiples of `emb_proj` from `emb` (where the specific multiple is the parameter K), and then seeing how it behaves differently for different values of K. Their original observation when doing this was that it reliably caused the model to claim that the first letter of the tweaked word was not the letter in question. In this article, they're trying to figure out how far they can push the tweak and still get reasonably accurate definitions of a token. What they discovered is that when they push a token's embedding vector further and further out along its "first-letter vector" and ask the network to define that word, the definitions it provides seem to follow particular themes during different regimes of K.

panarky 3y ago

> Ideally there is a mapping ... The complexity and interdependence of dimensions within embeddings make it practically impossible to ascribe specific, human-understandable meanings to individual elements or dimensions. This research actually adds to that complexity. Rather than making the meaning of individual dimensions more understandable, it shows that the embedding space has a strange, layered structure. It suggests a peculiar, almost nonsensical organization of concepts at different distances from the central point of typical token embeddings, which doesn't make it easier to pinpoint what each dimension means. It emphasizes that embeddings capture information in a distributed and highly contextual manner. The fact that embeddings for "nokens" can lead to arbitrary or bizarre categorizations when taken out of the typical token zone underscores that embedding dimensions don't have straightforward, easily interpretable meanings.

kridsdale1 3y ago
My interpretation is that the nokens are novel (un-coined) coordinates in a categorized zone of the vector field. Future linguistics will plant their flag in noken territory as we need new words. This already happened with “noken”! Noken-space is all the undefined territory full of noise equivalent to the actual visual noise we see in Stable Diffusion when you travel along a vector away from an island of coherence. To put it poetically, concept space is a vast hyperspace sea of random garbage (like unallocated RAM) and there are tiny islands or planets of meaning and value that look like things that matter to humanity. I don’t see the categorizations as being very bizarre. Putting on my amateur anthropology hat, each one described in the paper was clearly an important topic to a “primitive” person. Concerned with survival, people would talk about group dynamics, sharp things, plants/animals, things that look like infections (small flat round yellow white), and places. These are pretty much all you need to talk about. Looking at an English LLM vector space to understand fundamental principles of language is like looking at human DNA and trying to understand the first eukaryotes. It’s been complicated by effectively infinite generations of specializations adding noise. The probing work described here identifies some common principles that survive through the generations. A commenter above wondered if this was discovering something about the nature of the brain. I think it’s the nature of culture. Etymology is the product of culture and history.

mhink 3y ago

> Since there are ~5000 dimensions, in which of those dimensions are we moving k out ? > Is the idea you just move, out, in all dimensions such that the final Euclidean distance is k ? If I understand correctly, the basic idea is that in earlier experiments, they were able to use a relatively-simple technique to come up with what they call a "probe vector" in the embedding space which represents "the property of a word starting with <letter>". For any given token, the authors established that it was 98% probable that the token's embedding vector would be closer (by cosine similarity) to the "probe vector" representing the first letter of that word than any other probe vector. This is shown in the first graph of the section "A puzzling discovery". With that in mind, the diagram below that should start making more sense: "emb" is a particular token's embedding vector and "probe" is the probe vector for the token's first letter. "emb_proj" is the projection of "emb" onto "probe". What they're doing is tweaking the network weights by subtracting multiples of `emb_proj` from `emb` (where the specific multiple is the parameter K), and then seeing how it behaves differently for different values of K. Their original observation when doing this was that it reliably caused the model to claim that the first letter of the tweaked word was not the letter in question. In this article, they're trying to figure out how far they can push the tweak and still get reasonably accurate definitions of a token. What they discovered is that when they push a token's embedding vector further and further out along its "first-letter vector" and ask the network to define that word, the definitions it provides seem to follow particular themes during different regimes of K.