3 ms·

> Ideally there is a mapping ... The complexity and interdependence of dimensions within embeddings make it practically impossible to ascribe specific, human-

> Ideally there is a mapping ... The complexity and interdependence of dimensions within embeddings make it practically impossible to ascribe specific, human-understandable meanings to individual elements or dimensions. This research actually adds to that complexity. Rather than making the meaning of individual dimensions more understandable, it shows that the embedding space has a strange, layered structure. It suggests a peculiar, almost nonsensical organization of concepts at different distances from the central point of typical token embeddings, which doesn't make it easier to pinpoint what each dimension means. It emphasizes that embeddings capture information in a distributed and highly contextual manner. The fact that embeddings for "nokens" can lead to arbitrary or bizarre categorizations when taken out of the typical token zone underscores that embedding dimensions don't have straightforward, easily interpretable meanings.

kridsdale1 3y ago
My interpretation is that the nokens are novel (un-coined) coordinates in a categorized zone of the vector field. Future linguistics will plant their flag in noken territory as we need new words. This already happened with “noken”! Noken-space is all the undefined territory full of noise equivalent to the actual visual noise we see in Stable Diffusion when you travel along a vector away from an island of coherence. To put it poetically, concept space is a vast hyperspace sea of random garbage (like unallocated RAM) and there are tiny islands or planets of meaning and value that look like things that matter to humanity. I don’t see the categorizations as being very bizarre. Putting on my amateur anthropology hat, each one described in the paper was clearly an important topic to a “primitive” person. Concerned with survival, people would talk about group dynamics, sharp things, plants/animals, things that look like infections (small flat round yellow white), and places. These are pretty much all you need to talk about. Looking at an English LLM vector space to understand fundamental principles of language is like looking at human DNA and trying to understand the first eukaryotes. It’s been complicated by effectively infinite generations of specializations adding noise. The probing work described here identifies some common principles that survive through the generations. A commenter above wondered if this was discovering something about the nature of the brain. I think it’s the nature of culture. Etymology is the product of culture and history.

kridsdale1 3y ago

My interpretation is that the nokens are novel (un-coined) coordinates in a categorized zone of the vector field. Future linguistics will plant their flag in noken territory as we need new words. This already happened with “noken”! Noken-space is all the undefined territory full of noise equivalent to the actual visual noise we see in Stable Diffusion when you travel along a vector away from an island of coherence. To put it poetically, concept space is a vast hyperspace sea of random garbage (like unallocated RAM) and there are tiny islands or planets of meaning and value that look like things that matter to humanity. I don’t see the categorizations as being very bizarre. Putting on my amateur anthropology hat, each one described in the paper was clearly an important topic to a “primitive” person. Concerned with survival, people would talk about group dynamics, sharp things, plants/animals, things that look like infections (small flat round yellow white), and places. These are pretty much all you need to talk about. Looking at an English LLM vector space to understand fundamental principles of language is like looking at human DNA and trying to understand the first eukaryotes. It’s been complicated by effectively infinite generations of specializations adding noise. The probing work described here identifies some common principles that survive through the generations. A commenter above wondered if this was discovering something about the nature of the brain. I think it’s the nature of culture. Etymology is the product of culture and history.