8 ms·
Mapping the semantic void: Strange goings-on in GPT embedding spaces
- PaulHoule 3y agoShould try asking it to define things in French. Korean, etc.
- empath-nirvana 3y agoThat is _profoundly_ interesting, and I think a serious study of that is going to reveal a lot about human (or at least english speaking) psychology.
- kridsdale1 3y agoI would love to compare the results across many languages to find the language concepts that are truly universal and fundamental to all people. I suspect they will be the “antiquated” or “child-like” concepts identified here: the things that matter to hunter-gatherer people.
- lainga 3y agosort-of related is https://en.wikipedia.org/wiki/Swadesh_list https://en.wikipedia.org/wiki/Swadesh_list
- flir 3y agoMaybe, idk. But you'll definitely get accidents of history. For example about half the world speaks a Proto-Indo-European-derived language, which included vocabulary around wheels, horses, wagons and chariots. Those words tend to have cognates across many languages, but that doesn't necessarily mean they're universal or fundamental.
- JPLeRouzic 3y agoPlease can someone explain the gist of this article to a mere human with no PhD in AI?
- JPLeRouzic 3y agoI asked ChatGPT, and it reluctantly told: "The phenomena discussed in the text are more about the inherent characteristics of the model's representation space rather than the input data being flawed." It took three attempts, the first two were merely a rephrasing/paraphrasing of the abstract.
- ta988 3y agoTokens (parts of words, words, symbols) are represented as vectors. They say there are nokens which are vectors that seem to represent something because they are organized in that space in a regular manner. They think those vectors may have something interesting about them because they seem to be somewhat organized and related to each other.
- ta988 3y agoAlso the seem to correspond to things that exist in our world and curiously like things that have existed for a long time, not too recent things.
- JPLeRouzic 3y agoPlease could you point out where these strange definitions are found in the article?
- great_psy 3y agoSearch for the string “children” in the context of children books/stories
- JPLeRouzic 3y agoThanks Great_psy.
- lainga 3y agoLindySpace?
- kgc 3y agoWould be interesting to see how changing the tokenization strategy — whole words or even phrases instead of traditional tokens — changes the results.
- nybsjytm 3y agoBased on the first three figures, it seems like the author isn't familiar at all with probability in high-dimensional spaces, where these phenomena are exactly what you'd expect. Because of that, the figure with intersecting spheres is a big misinterpretation. I stopped reading there, but I think it's good to generally ignore things you find on lesswrong!
- bryan0 3y agoCan you please explain why this distribution of embeddings is expected for high-dim spaces?
- nybsjytm 3y agoIt's an example of concentration of measure: https://en.wikipedia.org/wiki/Concentration_of_measure https://en.wikipedia.org/wiki/Concentration_of_measure It's easy to verify this for yourself in a simple example in something like python. Draw from a multivariate normal distribution (or almost anything) in a 1000-dimensional space and look at the norm of your vector. If you do this a lot of times you'll see that the answer is close to the same each time. This even holds true if you replace the vector norm by the distance to an arbitrary fixed point. edit: here's some code: import numpy import matplotlib.pyplot norms = [] dists = [] for i in range(5000): vec = numpy.random.normal(size=4096) norms.append(numpy.linalg.norm(vec)) dists.append(numpy.linalg.norm(vec-numpy.ones(4096))) matplotlib.pyplot.hist(norms) matplotlib.pyplot.show() matplotlib.pyplot.hist(dists) matplotlib.pyplot.show() (didn't render right, sorry)
- zyklu5 3y agoCan't believe you are the only one to mention this. I thought concentration of measure would be at least mentioned to ML students right about the time they learn about the curse of dimensionality
- ttul 3y agoOne way to think about high-dimensional spaces is that there is a huge amount of volume relative to the number of coordinates in the space. Yes, there are infinitely many coordinates; however, each coordinate has more dimensions to it relative to a lower-dimensional space. As you increase the dimensionality, coordinates representing meaningful data spread out, leading to a bizarre situation in which most data points in the space are as far apart as a randomly chosen pair of data points, even if the data points you are measuring the distance between should be "close" to each other. For instance, consider a unit cube in n dimensions. The volume of this cube (1 unit on each side) remains constant, but the distance from the center to a corner increases with the square root of the number of dimensions. This implies that in high dimensions, most of the volume of a hypercube is located at its corners. This is bizarre to comprehend because it's hard to stop imagining a "cube" in the 3D sense. Algorithms that rely on distance measures (like k-nearest neighbors or clustering) may perform poorly in high-dimensional spaces without dimensionality reduction techniques like PCA (Principal Component Analysis) or t-SNE (t-Distributed Stochastic Neighbor Embedding). And even with these techniques, you can get very misleading results.
- harveywi 3y agoMaybe a dumb question: Shouldn't these be called immersions instead of embeddings?
- great_psy 3y agoSo I am a bit confused about the part where you go k distance out from the centroids. Since there are ~5000 dimensions, in which of those dimensions are we moving k out ? Is the idea you just move, out, in all dimensions such that the final Euclidean distance is k ? Seems that’s how they get multiple samples at those distances. Either way I think it’s more interesting to go out in specific dimensions. Ideally there is a mapping between each dimension and something inherent about the token, like the part where a dimension corresponds with the first word of the token. We went through this discovery phase when we were generating images using autoencoders, same idea, some of those dimensions would correspond to certain features of the image, so moving along them would change the image output in some predictable way. Either way, I think the overall structure of those spaces says something about how the human brain works ( given we invented the language). I’m interested to see if anything neurologic can be derived from those vector embeddings.
- panarky 3y ago> Ideally there is a mapping ... The complexity and interdependence of dimensions within embeddings make it practically impossible to ascribe specific, human-understandable meanings to individual elements or dimensions. This research actually adds to that complexity. Rather than making the meaning of individual dimensions more understandable, it shows that the embedding space has a strange, layered structure. It suggests a peculiar, almost nonsensical organization of concepts at different distances from the central point of typical token embeddings, which doesn't make it easier to pinpoint what each dimension means. It emphasizes that embeddings capture information in a distributed and highly contextual manner. The fact that embeddings for "nokens" can lead to arbitrary or bizarre categorizations when taken out of the typical token zone underscores that embedding dimensions don't have straightforward, easily interpretable meanings.
- kridsdale1 3y agoMy interpretation is that the nokens are novel (un-coined) coordinates in a categorized zone of the vector field. Future linguistics will plant their flag in noken territory as we need new words. This already happened with “noken”! Noken-space is all the undefined territory full of noise equivalent to the actual visual noise we see in Stable Diffusion when you travel along a vector away from an island of coherence. To put it poetically, concept space is a vast hyperspace sea of random garbage (like unallocated RAM) and there are tiny islands or planets of meaning and value that look like things that matter to humanity. I don’t see the categorizations as being very bizarre. Putting on my amateur anthropology hat, each one described in the paper was clearly an important topic to a “primitive” person. Concerned with survival, people would talk about group dynamics, sharp things, plants/animals, things that look like infections (small flat round yellow white), and places. These are pretty much all you need to talk about. Looking at an English LLM vector space to understand fundamental principles of language is like looking at human DNA and trying to understand the first eukaryotes. It’s been complicated by effectively infinite generations of specializations adding noise. The probing work described here identifies some common principles that survive through the generations. A commenter above wondered if this was discovering something about the nature of the brain. I think it’s the nature of culture. Etymology is the product of culture and history.
- kridsdale1 3y agoThis is the first post in LessWrong that I found interesting and not insipid navel gazing. My interpretation of the finding is that the LLM training has revealed the primordial nature of human language, akin to the primary function of genetic code predominately being to ensure the ongoing mechanical homeostasis of the organism. Language’s evolutionary purpose is to drive group-fitness. Group management fundamentals are thus the predominant themes in the undefined latent space. Who is in our tribe? Who should we trust? Who should we kill? Who leads us? These matters likely dominated human speech for a thousand millennia.
- andrewflnr 3y agoI would at minimum want to see the analysis replicated across a variety of languages before I started drawing conclusions about "the primordial nature of human language". In particular, languages with no known relation to English; Chinese would be a good one, far from the Indo-European tree and with a big corpus. The membership-centric is certainly appealing, but really it's too appealing to let ourselves take it seriously without counterbalancing it with a high standard of evidence.
- __loam 3y ago> My interpretation of the finding is that the LLM training has revealed the primordial nature of human language, akin to the primary function of genetic code predominately being to ensure the ongoing mechanical homeostasis of the organism. We've truly reached peak hype. It's a token predictor dude.
- pests 3y agojust set a server up with ftp and ssh...
- bomewish 3y agoI mean to take the evopsych spin off it and just distil what you’re saying: “language is a tool we use to do stuff in the world.” Notwrong!
- flir 3y agoSomeone shoot me down if I'm wrong: In this model (one of many infinitely many possible models) a word is a point in 4096-space. This article is trying to tease out the structure of those points, and suggesting we might be able to conclude something about natural language from that structure - like looking at the large-scale structure of the Universe and deriving information about the Big Bang. Obvious questions: is the large-scale structure conserved across languages? What happens if we train on random tokens - a corpus of noise? Does structure still emerge? It might be interesting, it might be an artifact. I'd be curious to know what happens when you only examine complete words.
- jakedahn 3y agoHas anyone done this analysis for other llms like llama2?
- chpatrick 3y agoI think that's really interesting but one thing that I'm not sure about is that a lot of tokens are just fragments of words like "ei", not whole words. Can we really makes conclusions about those embeddings?