18 ms·
Language Models Represent Space and Time
- famouswaffles 3y agoThe Twitter thread is worth looking at. Very Fascinating https://twitter.com/wesg52/status/1709551516577902782?t=3b2F1xtFSUc1Bp-NPFon2Q&s=19 https://twitter.com/wesg52/status/1709551516577902782?t=3b2F...
- joshuahedlund 3y agoI don't understand what the colors represent and I can't read the thread. Can you summarize?
- famouswaffles 3y agoYou should be able to read the thread here https://nitter.net/wesg52/status/1709551516577902782 https://nitter.net/wesg52/status/1709551516577902782 The colors in the first tweet just represent different territories(continents)
- joshuahedlund 3y agoI probably saw this too late to get a reply but: Yes, I understand that part. I don’t understand how that connects to the underlying data. Where did the x and y coordinates for each point come from? Where did the territory value of each point come from for that matter?
- nmarinov 3y agoSeconded, it's a good summary with graphs and commentary. Also nitter to avoid the login wall: https://nitter.net/wesg52/status/1709551516577902782 https://nitter.net/wesg52/status/1709551516577902782
- kridsdale3 3y agoThank you for your service.
- civilitty 3y agoI think we have a new member to add to the group: Lies, damn lies, and statistics… and now neural activations. How exciting! This reminds me of all those fMRI studies that look at brain blood flow before the dead salmon experiment came out.
- somenameforme 3y agoHaha, I have no idea how I was not aware of the dead salmon experiment prior. This article [1] has a nice overview for anybody else in my shoes. Thanks for the reference. [1] - https://blogs.scientificamerican.com/scicurious-brain/ignobel-prize-in-neuroscience-the-dead-salmon-study/ https://blogs.scientificamerican.com/scicurious-brain/ignobe...
- gpderetta 3y agoFirst of all that's an excellent article. But also: > Some people like to use the salmon study as proof that fMRI is woo, but this isn't the case, it's actually a study to show the importance of correcting your stats. I.e. be aware of the "look elsewhere effect".
- corethree 3y agoA lot of HNers were so adamant that the LLMs understand absolutely nothing and that these models are just predicting the next most likely word. I think with this paper it becomes clear that the adamant denial was just human bias talking. LLMs crossed a certain line here. You would think people would be amazed or in awe or react in fear at technological break through a and they often are. The weird part for LLMs was the delusion. Many people were in denial that it was any sort of breakthrough.
- YeGoblynQueenne 3y agoThis is a preprint that was just uploaded to arxiv two days ago. Don't be hasty and assume that it settles any matter at all. Many such claims have been made before and many counter-claims also. There is still a lively debate on the subject and it will be some time before there is agreement. More generally, any scholarly article is a claim, and should never be read as automatically true. That's something to keep in mind.
- circuit10 3y agoI thought it was completely obvious that if it can predict things about the world as well as it can it has some form of world model, even if inaccurate. I’m surprised that anyone would argue that a language model doesn’t model things
- YeGoblynQueenne 3y agoYes, I see what you mean. It's frustrating but the word "model" is severely overused in Computer Science and AI and it can cause a lot of confusion. Briefly, a "world model" is a theory possessed by an autonomous agent that describes the entities that exist in the world and how they interact with each other and with the agent, and that the agent can use to make decisions. This is the sense in which "model" is used when people talk about "model based" approaches to AI (such as planning, for instance, which is "the model based approach to autonomous behaviour"). A "language model" on the other hand is a statistical model of the text in a corpus. A statistical model is really a set of events, and their probabilities. The "events" in a statistical language model are usually word collocations, where a collocation is a word A found near a word B, for some measure of "near". If you've heard about "word embeddings", that's a model of word collocations in text corpora. So the claim that LLMs "have a model" is about LLMs being autonomous agents with a world-model, while "Large Language Model" is about statistical modelling. There's no reason why a statistical language model should have a world model.
- benlivengood 3y agoThe next step in the research should be ablation of these time and space neurons to see how it affects accuracy on space/time completions which would help rule out the memorization of linear probes.
- SiempreViernes 3y agoIt would also be reassuring if they showed that they couldn't reach similar performance when they assigned random values to the target coordinates and used non-spatial words.
- cratermoon 3y agoNo, no they don't. People looking at the output of LLMs project their own internal models of the world, time, and space onto the output and project that model as if it were the LLM's.
- canjobear 3y agoMuch simpler models like word2vec also showed linear embeddings of geography in this way.
- semitones 3y agosource?
- uoaei 3y agoIt was something like Paris - France + Japan = Tokyo I know that's not a source but you could also just google "word2vec", as I recall most of the explainer blog posts had similar examples.
- dylan604 3y agoSo it's good at SAT type questions?
- uoaei 3y agoIt's not "good" at anything, it's a set of word embeddings. You have to apply them in some context using a feedforward model of some sort for them to be used in some task.
- wnoise 3y agoThat gets "analogies" rather than "spatial awareness" per-se.
- uoaei 3y agoYou put quotes around spatial awareness but the commenter being referenced said geography.
- alexpetralia 3y agoSpace and time (the abstractions) are lingisuric concepts, so this is perhaps not too surprising, but nevertheless interesting.
- klysm 3y agoThe space and time mapping they pull out of the LLM here is non-linguistic.
- klysm 3y agoI’m most surprised that the embedding is actually linear. I wonder if that makes extrapolation perform better
- nyrikki 3y agoFeed forward networks are pairwise linear in respect to inputs, they are effectively DAGs. In theory you could represent an entire LLM as a single 2 dimensional graph of linear line segments. It wouldn't be useful for much as the parameters are clustered in dense patches.
- kridsdale3 3y agoSo are my synapses, and by and large my cortex is 2-dimensional as well (folded and scrunched in to a super complex shape by the limitations of my roughly spherical cranium)
- nyrikki 3y agoEverything from ion channel response up is non-linear and not a deterministic Turing machine. Random cite https://www.jneurosci.org/content/13/1/334?ijkey=9f8730dd3d028900055afd202f57b98d5498b776&keytype2=tf_ipsecsha https://www.jneurosci.org/content/13/1/334?ijkey=9f8730dd3d0...
- xpe 3y agoThe papers cites earlier papers that posit a ‘linear representation hypothesis’. Does your surprise “factor in” these papers? Do you suspect something different in play for the current paper?
- klysm 3y agoDidn’t have that factored into my surprise because I’m not that informed. I just thought that the latent space representation being linear was interesting.
- toxik 3y ago
- yieldcrv 3y agoI bet if we were able to get better output from our brains we would see something similar, for those of us that have any spatial awareness capabilities But we don't have the output to analyze aside from our language and assumption of shared experience
- ilaksh 3y agoOne thing that is clouding this discussion is that most people are mixing up a lot of different characteristics that animals like humans have as if they were all the same thing. So for many people they don't really distinguish between things like "reasoning", "self-aware", "conscious", "alive", "sentient", "intelligent", "has world model". They also don't distinguish between different types or varying levels of cognitive abilities. It seems clear that high functioning LLMs must have some type of world model. But that doesn't mean it's necessarily exactly the same type of highly grounded model that a human would have, especially if it was trained on only text. It might be less rich or different but still quite useful. Another example, LLMs clearly don't have the same type of fast adaptation in a realtime 3d environment that animals have. (That's not to say that they can't mimic it in some rough ways). But if you don't really break all of this stuff down carefully in your head then it can be hard to accept that LLMs are doing anything interesting. Because in that worldview, it's all the same thing, so they have to give the LLM all of the other characteristics at the same time.
- xpe 3y agoWell said. Also, the paper’s use of “world model” (instead of simply “model”) doesn’t help clarify the situation.
- sebzim4500 3y agoI do not see how referring to a world model as simply a "model" could possibly provide clarity. LLM literally stands for Large Language Model, it has a 'model' in it by definition.
- sgt101 3y agoWould we think of it differently if we called it LLC - large language correlation?
- kelseyfrog 3y agoIs-a vs has-a relationship.
- 3y ago
- teaearlgraycold 3y agoThere was an interesting thought from a Microsoft researcher in an episode of This American Life. He had been given early access to GPT4 and found that it had gained an understanding of gravity and balancing. 3.5 would fail to describe a safe order for stacking 3 eggs, a bottle, a book and a nail. But 4 would give robust answers with logical justification added to each construction step for the tower. The researcher remarked (paraphrasing) “The models are built to predict the next token. At some point the only way to improve token prediction is to start modeling the world and building an understanding of how things work.” Since I heard this I see it everywhere with LLMs. Load up a 30B parameter model on your computer and toy around with it. When it fails I now see that it’s reached a concept it doesn’t understand and is in more-or-less Markov Chain mode. Give the same task to GPT4 and it proves it’s modeled the concept appropriately because it can pull the right words out of the ether.
- xpe 3y ago> Give the same task to GPT4 and it proves it’s modeled the concept appropriately because it can pull the right words out of the ether. I take your point, but we should be careful when using phrased like “modeled the concept appropriately” here. That implies correctness. There are many ‘concept models’ that could work well enough. Unless we can inspect the concept model directly (inside the LLM, somehow!) we are left to reason about how the concept is represented via statistics over a set of interactions. So, if, over the course of many interactions, it seems that a model is saying reasonable things we can have more confidence that it has achieved a concept model that is “good enough”. But such an assessment is always in relationship to the test protocol. Claims to generality should be done carefully. In some ways, my standard here is probably not that different than a rigorous human educational assessment. I’m not talking about standardized tests, I’m talking about adversarial challenges like one would hope to see during a Ph.D. defense.
- teaearlgraycold 3y agoYeah I’m setting the bar at the level of a child’s understanding. Because smaller models are well below that, just picking tokens at random. That’s how misunderstanding manifests with LLMs. They don’t say “I don’t know”, they just go off the rails. The quality bar for different use cases might be such that an understanding of a system on par with a 6 year old is isn’t useful. But right now we clearly have two clusters and I’m drawing the boundary where it’s most useful.
- didibus 3y agoCan someone explain to me how they get these graphs? What exactly are they measuring in the model execution?
- ImHereToVote 3y agoWait I had the "It's just a stochastic parrot" parroted at me ad nauseum. Was this FUD all along?
- nyrikki 3y agoNo, this paper doesn't change that claim at all. This paper is about learnability and not generalization of what is learned. From the papers summary, note that they are referencing learning and not some implications that it gets us closer to a universal quantifier. "We have provided evidence that LLMs learn linear representations of space and time that are unified across entity types and fairly robust to prompting, and that there exists individual neurons that are highly sensitive to these features. The corollary is that next token prediction alone is sufficient for learning a literal map of the world give sufficient model and data size."
- ImHereToVote 3y agoI mean parrots have neurons. I didn't realize the line referred to the actual birds "parrot". Parrots are wicked smart.
- nyrikki 3y agoParrots, in the context of stochastic parrots is in reference to repeating or imitation mechanically without understanding. The 'common sense' problem in AI is the important part, not the as of yet undefined abstract concept of intelligence. LLMs are still pattern matching and finding with some stochastic aspects. But artificial nurons are inspired by biological nurons, they are not equivalent in respect to computability. But artificial nural networks are useful and powerful for some problems that traditionally weren't solvable by other means. Biological nurons have to embrace non-linear stochastic processes, which is a different problem set. While not expressed in popsci this has been known for decades.
- ImHereToVote 3y agoGreat so biological neurons can perform feats of intuition, that simulated neurons simply can't. Even theoretically. Gotcha. Thank you very much for the explanation.
- tomjakubowski 3y agoLLMs have an internal temporal model and yet ChatGPT still can't explain to me what happened in Tenet
- jsty 3y agoI’ve watched Tenet and I’m not sure I can explain what happened either …
- gcanyon 3y agoTo be fair, Christopher Nolan made a whole 2+ hour movie attempting to explain the concept and couldn’t…
- kridsdale3 3y agoGPT4 explains it pretty clearly, I think: "Tenet" is a 2020 science fiction action-thriller film written and directed by Christopher Nolan. The story revolves around concepts of time manipulation and inversion, making it a complex narrative that can be challenging to understand on first viewing. Here's a basic summary: The protagonist (referred to as "The Protagonist", played by John David Washington) is a secret agent who gets involved in a mission to prevent World War III. He is introduced to a concept called "inversion," where the entropy of people or objects can be reversed, making them move backwards in time. His mission leads him to cross paths with a Russian oligarch named Andrei Sator (played by Kenneth Branagh), who is collecting pieces of an algorithm that can invert the entropy of the entire world, effectively reversing time and destroying the present in favor of the future. The Protagonist also meets Kat (played by Elizabeth Debicki), Sator's estranged wife, who becomes a crucial part of the mission. The Protagonist and his partner Neil (played by Robert Pattinson) use inversion to their advantage in several action sequences, including a car chase and a final battle at Sator's secret city, where they successfully get the algorithm and prevent the destruction of the present. The twist at the end of the film reveals that Neil was recruited by a future version of The Protagonist, and that they have been working together for much longer than the duration of the film's events. This means that the organization "Tenet", which they work for, was created by The Protagonist himself in the future. Neil's character is seen sacrificing himself to ensure the mission's success, highlighting the theme of fate and predestination in the movie. "Tenet" is a complex film that uses its time manipulation concept to construct a narrative that loops in on itself, with events and characters revealing their true significance only as the story progresses or even after the film ends. Nolan's movie plays with the concepts of time, fate, and free will, requiring viewers to actively engage with and decipher its narrative structure.
- gcanyon 3y ago“Have an internal world model,” or “give the appearance of having an internal world model”? (Disclaimer: I’ve only read the summary, not the full paper) Recent papers about advanced Go-playing AIs have made it abundantly clear that, at best, their conceptual understanding of Go is deeply flawed, and at worst, completely absent. Yet under most circumstances they give the appearance of super-human understanding based on superhuman play. Do we have any reason to think LLMs are different? That they understand anything beyond “this is the best word to go next here.” To be clear, LLM output is remarkable, all the more so if the above is true.
- galaxytachyon 3y agoI think sometimes it is good to zoom out a little bit and look at things on a higher level. The LLMs are not an organic creature. It doesn't in any shape or way relate to a human being except the knowledge it was trained on. A bacteria shares over 50% of its genetic sequences with a human and we don't see it as anything more than a glorified bionic automaton. An LLM shares 0% with us. In fact its "genetic sequence" is coded on a completely different physics altogether. It is unreasonable to expect it to understand or think the same way an organic creature do. But the outcome of its action is irrefutable. It plays and does things better better than us. That is all we need to evaluate it on. I know that the topic is about "sentience" but my point is we are evaluating a complete alien being using human's standards. Of course we will see that it is lacking. It isn't human.
- deleted 3y ago[deleted]
- lsy 3y agoThe interpretation of the LLM having a "world model" is a big stretch in terminology. Encoding a coherent or accurate set of spatial coordinates for places is not equivalent to understanding the actuality of world space and the relationship between items in that space. By this definition an accurate spreadsheet of the lat-long of every major city would also constitute a "world model". It shouldn't be surprising to anyone that a model trained on (likely many) such spreadsheets would also encode that coordinate data. What's notable is that it still takes a team of human researchers to plot these raw numbers onto a map and to interpret them as experiential differentiators in physical space. There's no doubt that an LLM can uncover whatever is structurally included in its training data, even if that encoding is implicit. What's less believable is that the LLM somehow achieves a grounded understanding of physical aspects of experience purely from streams of raw text. I think the paper engages in some equivocation here, as the abstract differentiates between "an enormous collection of superficial statistics" and "a coherent model of the data generating process", without admitting that the first should well imply the second. But that doesn't then further imply a "world model" in the sense that we understand it as sentient beings. For us, an internal model is useful, but when our model or texts don't agree with the actuality of the world, the actuality of the world takes precedence. For an LLM there is no distinguishing between its trained representations and an actual exterior place, or even any sense that an exterior space exists.
- nopinsight 3y agoCurrent LLMs clearly have much coarser models of the world than humans do. Their training data do not include as many modalities and training data with spatiotemporal dimensions are inadequate. Up and coming multimodal models are changing this.
- lsy 3y ago"Multimodal" models only promise an additional layer of indirection and interpretation from the world through sensors into the same conceptual mechanism. An image is converted to tokens, a sensor is converted to coordinates, and it is all fed back into a correlating model. But this isn't adequate either. The famous dress illusion picture has RGB values closer to gold and white, but the actual physical dress was black and blue. The difference between perception and reality is something we don't have a handle on when it comes to these models.
- xpe 3y agoI’m seeking a brief almost hands-on introduction to how probing is done: in earlier papers and here. Suggestions? (Preferably with some photographs of napkin drawings.))
- xpe 3y agoPerhaps the most important public discussion, at least among technical people, is making sure we really understand the experimental setup. (I admit I could be projecting my lack of understanding; I’m not rock-solid on the experimental setup yet.) Still to me, it matters if the linearity been described can be observed directly _in_ the model. My understanding is that it is not; rather, it is tested with an additional model. The term probing can be misunderstood; it isn’t -just- an internal probe. There are ‘derivative’ models as well. This raises questions about what we mean when we we are talking about internal representations! In my view, a derivative of an internal representation is not the same thing. (I welcome people who can educate, correct me, or even reframe the topics. The above thoughts are subject to change.)
- karelpeeters 3y agoIt's important to note that these probe models are very simple - they're "trained" by just doing a linear regression between the hidden activations and the desired output. This means that the probes can barely do any computation themselves, so if they work at all this is a strong indication that the signal they predict was already in the hidden activations. For even more proof, see "Figure 5: Space and time neurons in Llama-2 models" for single neurons in LLMs that already encode this information, without even having to use a probe model to extract it.
- xpe 3y agoThanks. After some additional research, I tend to agree. One thing I learned of note is that while a _positive_ result (i.e. a probe showing a relationship) is proof that a label is encoded somehow; a _negative_ result does _not_ prove that the network under test does _not_ encode the information _somehow_. (The information could be encoded in more complex ways that the probe does not discover).
- haltist 3y ago> coherent model of the data generating process -- a world model. I'd never seen a definition of world model but this seems deficient in several ways because it does not mention anything about abstraction and logical reasoning.
- wnoise 3y agoThose aren't part of the model though. The model is just a representation of the world. An abstract representation can be useful, but so can more concrete ones. A representation that is amenable to logical reasoning is also useful, but the reasoning isn't part of the model.
- haltist 3y agoThen why call it a world model? It's a "data" model and not actually a "world" model. World models would need to have logical relationships between the entities being modeled which would include non-trivial logical relations, R(x,y), and properties, P(x). Current language models do not have such representations and fail miserably on any task that is not explicitly included in the training set (sudoku being an obvious example).
- famouswaffles 3y ago>World models would need to have logical relationships between the entities being No they wouldn't. There is no evidence, basically none whatsoever that logical can model reality or that general "perfect logical reasoning" is a thing that actually exists in the real world in real world relationships. None. No animal we've observed does it. Humans certainly don't do it. The only realm this idea actually seems to work is Fiction. and this was not like for a lack of trying. Some of the greatest minds worked on this for decades and some people still don't seem to get it. Logic doesn't scale. It breaks at real world relationships. We were using logic for image, speech, language synthesis and just had to throw it away once something that actually worked came about. If logic was so great at modeling the world, we wouldn't be using LLMs like GPT-4 , we'd be using Cyc, the system with decades of a headstart. Logic systems are that guy in the stands yelling that he could've made the shot, while he's not even on the field. >included in the training set (sudoku being an obvious example) LLMs can play sudoku just fine. https://arxiv.org/abs/2305.08291 https://arxiv.org/abs/2305.08291