14 ms·
Do Large Language Models learn world models or just surface statistics?
- lapama 4y agoThe question reads like: are they lazy/careless?
- passion__desire 4y agoIdea from physics: simple bulk theory (depth) is equivalent to complicated surface theory.
- jes5199 4y agoLLMs are very intelligent they just need to try harder
- cyanf 4y agoDoes a submarine swim?
- deleted 4y ago[deleted]
- mattigames 4y agoPeople sometimes forget that words are just useful labels, the ship of Theseus is whatever ship we agree is the ship of Theseus, it was just very handy to make a sound for others to know what ship I'm talking about, but it is not an specific/defined set of atoms, it never was, the same thing happens with a lot (maybe all) concepts such as swimming, or even statistical inferences.
- pyinstallwoes 4y agoWell it’s not entirely arbitrary. You can say it objectively fits within a frequency spectrum. Similar to colors or musical notes.
- masswerk 4y ago"Nominalis! Eum comprehende, eum combure!" ;-) (Meaning, there has been an ongoing discussion about this for about a millennium, but I'm a nominalist, as well.)
- westurner 4y agoIf they don't search for Tensor path integrals, for example, can any NN or symbolic solution ever be universally sufficient? A generalized solution term expression for complex quantum logarithmic relations: e**(w*(I**x)*(Pi**z)) What sorts of relation expression term forms do LLMs synthesize from? Can [LLM XYZ] answer prompts like: "How far is the straight-line distance from (3red, 2blue, 5green) to (1red, 5blue, 7green)?" > - What are "Truthiness", Confidence Intervals and Error Propagation? > - What is Convergence? > - What does it mean for algorithmic outputs to converge given additional parametric noise? > - "How certain are you that that is the correct answer?" > - How does [ChatGPT] handle known-to-be or presumed-to-be unsolved math and physics problems? > - "How do we create room-temperature superconductivity?" "A solution for room temperature superconductivity using materials and energy from and on Earth" > - "How will planetary orbital trajectories change in the n-body gravity problem if another dense probably interstellar mass passes through our local system?" Where will a tracer ball be after time t in a fluid simulation ((super-)fluid NDEs Non-Differential Equations) of e.g. a vortex turbine next to a stream? How do General Relativity, Quantum Field Theory, Bernoulli's, Navier Stokes, and the Standard Model explain how to read and write to points in spacetime and how do we solve gravity?
- westurner 4y agoDid I forget to cite myself (without a URL)? Notable enough for a citation it isn't. "[Edu-sig] ChatGPT for py teaching" (2023) Editing Python Mailing List. (2023) No, LLMs do not learn a sufficient world model for answering basic physics questions that aren't answered in the training corpus; and, AGI-strength AI is necessary for ethical reasoning given the liability in that application domain. Hopefully, LLMs can at least fill in with possible terms like '4π' given other uncited training corpus data. LLMs are helpful for Evolutionary Algorithmic methods like mutation and crossover, but then straight-up ethical selection. Ask [the LLM] to return a confidence estimate when it can't know the correct answer, as with hard and thus valuable e.g. physics problems. What tone of voice did Peabody take in explaining to Sherman, and what does an LLM emulate?
- Animats 4y agoHuh. They've been able to demonstrate that the trained Othello player has a model. They did this by feeding the entire state of the trained system into a classifier. But they have no idea what the form of that model is, or how it is used to predict moves. Even for this simple, well-defined problem, it's a total black box. This level of incomprehensibility is worrisome.
- ramraj07 4y agoIsn’t our brain a black box too? Why exactly is it worrisome?
- tobr 4y agoBecause human brains have been known to intentionally lie and manipulate, and secretly plan horrible deeds.
- eternalban 4y agoWe'll need psych tests for our AI. Malicious AI will note this and adapt. An arms races ensues, where one adversary finances and invests in upgrading the other. Guess who wins?
- shadowofneptune 4y agoThe mind itself, perhaps. The person in possession of it though usually does have a conscious strategy to play a game once they are proficient, and can happily explain it to you.
- pyinstallwoes 4y agoHow many people can actually explain their state of mind and rational or seemingly rational thought process?
- space_fountain 4y agoAnd we have all sorts of social science these days implying the explanations when people do try to explain themselves they’re justifying decisions that came out of black box rather than actual describing how the decision happened
- ramraj07 4y agoAs is being confirmed by this report, these large language models are absolutely building models of the real world, though not exactly at the abstraction layer level we initially guessed. The only question is how thorough of a model it is, and what is the “waste factor”, I.e., how much more inefficient are these at learning the world compared to say the human brain. It’ll be tantalizing to see how GPT-4 performs but my best guess would be that there will be an even more amazing performance but not something that can truly match human beings (even a dumb one) without significant change to the architecture.
- mattigames 4y agoIt doesn't matter, brains took millions of years to become what they are now, it wouldn't be a surprise that we may need to take a completely different path to make machines perform equally or better than a human using our current technically (incl future iterations/versions of it), including stuff that looked from a biological perspective would be "wasteful".
- pyinstallwoes 4y agoIf we follow your logic you can say that whatever humans build is also a result of millions of years.
- mattigames 4y agoNot really, that would follow my logic if I had said sometbing like "the universe took millions of years to create the human species in including the human brain", and that's not what I said, the baseline is an organism and it's ability to evolve, nothing before that. And anyway in this context "evolution took millions of years" is just another way to say that a huge amount of lesser brains died while the ancestors of the current brain survived, AKA self-selection; so clearly is unlikely to be any similar to the process we will take to improve artificial intelligence as we will measure it with a different standard and NOT over its ability to survive in the wild.
- irrational 4y ago
- pharmakom 4y agoI can barely understand the state of systems I wrote by hand.
- ttyyzz 4y agoThen you should just wait a while. After a couple of years the Code might aswell been written by someone else.
- aaron695 4y agoIt's like the hidden constants in HN config, it'll just be a couple of simple numbers but no one will fully understand how they change the complex system that comes out in HN. Bram Cohen on BitTorrent on how he chose some of his constants, he just felt they were right.
- thom 4y agoThis doesn’t seem a surprising or particularly deep result, and perhaps we’re having the wrong arguments about LLMs. I certainly don’t doubt that they’re capturing deep knowledge, my only criticism would be that they’re doing so in a lossy and extremely static way. I’ve come to think of it this way. A simple Markov chain model trained on language would be the simplest example of something picking up a very shallow model of some text. Let’s call that a Layer One model, and we had fun with them for a while. Machine translation goes one step further - instead of learning just the text it’s given, a model sees that above words are representations of things and ideas, and structures to render those ideas in multiple languages. That feels like Layer Two, but I’m not aware of traditional machine translation models that will take a piece of poetry, for example, and render it in a new language, perhaps using local idioms to capture the essence, if not the exact translation, of some content. So it’s only really one layer deeper. Large language models are layers beyond this. They are both wider and deeper. They are showing that much more knowledge than we thought can be captured statically - patterns of patterns of patterns of patterns turn out to be enough to store and process a huge amount of what we previously thought was a privileged part of human intelligence. That’s the truly interesting result - the models still seem dumb to me, but boy, being dumb is much more impressive than we used to think. We’re focused on training LLMs on larger corpora to store more stuff and more parameters and layers to find deeper patterns. But even though they’re clearly learning world models, it’s still just a static snapshot of knowledge. ChatGPT shows over a short span that a model can integrate new information quite well. What’s missing is some mechanism for _constant_ integration of new information, and some degree of internal rumination about past, present and future knowledge. Something algorithmic, temporal and dynamic, not just a static repository of knowledge. I don’t think this actually requires a great deal of magic, just a slight change in direction (although I’ve no doubt this is being worked on and is probably in all sorts of papers already). Of course, there’s still time for us to decide not to do that, but the hour is close at hand.
- typon 4y agoI think to do this at scale and properly we'd need a replacement for backprop
- deleted 4y ago[deleted]
- jokoon 4y agoI don't think those ai are able to have reason and to understand the data they're given. They just reshape the data without understanding it. It's still artificial. Unless there is a breakthrough in analyzing those neural blackbox, I would not hope a lot of those ai.
- pmontra 4y agoI read the web page and the PDF paper quickly (https://arxiv.org/pdf/2210.13382.pdf https://arxiv.org/pdf/2210.13382.pdf) and it seems that there are no examples of the prompts they used. It would be great to see which technique they are using or is it obvious for experts of the field? It would help with reproducibility too.
- jchmrt 4y agoI'm no expert in this field, but as I understand it they trained a new GPT model from scratch to play Othello. So the prompts were simply the transcript of moves that came before.
- Al-Khwarizmi 4y agoYes, it's that. This model wasn't trained on natural language texts at all, so it can't be "prompted" with sentences like ChatGPT, etc. Lists of Othello moves is all it has seen.
- Tepix 4y agoDid Kenneth Li publish his model? As an Othello player i would like to examine it.
- golol 4y agoIt is good that someone formulated this argument, but I have to say that the conclusion should be obvious to anyone that is not totally blinded by philosophical beliefs about intelligence and algorithms, as opposed to looking at the real evidence of the capabilities of real AI. Pick any human. I claim there exists some subject that ChatGPT has a better understanding of than this human, based on a surface-level evaluation.
- quotemstr 4y agoRight. One of the most remarkable side effects of the recent advances in AI has been a swath of the tech industry bleating desperately in unison that whatever an ML model might do, it isn't "real" intelligence because humans are magic and, since tensors don't have that magic, AI can't "really" think.
- golol 4y agoIn fact, it is quite hilarious that in the last 5 years the go-to criticism of state-of-the art AI has switched from "It can only do one specfic thing very well!" to "It can't do any specific thing very well!". We have really struck a goldmine with unsupervised LLMs. While before the range of forms of intelligence we could cover only contained high performance "specific" intelligence, we now finally have an outpost in low performance "general" intelligence. All we have to do is perform a pincer movement! Any data is tokens, any models can be interfaced. I predict models will be trained on DNS-Server data streams.
- throw2500 4y ago> Pick any human. I claim there exists some subject that ChatGPT has a better understanding of than this human, based on a surface-level evaluation. Isn't that a low bar, though? For any human, you can find some subject that that human knows next to nothing about. It's kind of like saying "Google is smarter than humanity because for any human, I can find some fact that a Google search will reveal that that human doesn't know".
- golol 4y ago
- nickelpro 4y ago> We found that the trained Othello-GPT usually makes legal moves. The error rate is 0.01%; and for comparison, the untrained Othello-GPT has an error rate of 93.29%. This is much like the observation in our parable that the crow was announcing the next moves. I read the whole article, but I think this statement largely disproves the hypothesis? We only need a single counter example to show that Othello-GPT does not have a systemic understanding of the rules, only a statistical inference of them. A model that "knows" the rules will make no errors, a model that makes any errors does not "know" the rules. Simple as. And I feel this way about much of the article, they state they change intermediate activations and therefore the observed valid results prove that the layers make up a rules engine instead of a statistical engine. And I don't make that leap? Why would that necessarily be the case? Obviously a statistical engine trained on legal moves will mostly produce legal output. A rules engine will always produce legal output.
- cjg 4y agoI know the rules of chess, but my rate of attempting illegal moves is above 0%.
- nickelpro 4y agoA human mind is not a rules model for Chess, and comparisons to a human mind are worthless. We're not asking if the LLM is human, only if it is modeling the game rules or modeling statistics My point is this: * If the results of the model were perfect, not a single counter-example, that would be a strong argument that the LLM has built an internal model of Othello * If counter-examples exist, the result of the experiment is indistinguishable from a sufficiently clever statistical inference and proves nothing
- mannykannot 4y agoI'm trying to figure out whether this is a fair (and, I hope, neutral) representation of what the authors have achieved here: The result of training Othello-GPT is a program which, on being fed a string conforming to the syntax used to represent moves in an Othello game, outputs a usually-valid next move. Through their probing of this program, the authors have gained an understanding of the program's state after processing the input. This state can also be interpreted as representing the state of the board in the game represented by the input string. This understanding allows them to make predictions about what state they would expect the program to be in if the board had reached a slightly different state (in the examples given, the changes are flips of a single disc, without regard to whether this state of play could be reached through valid moves.) When the state of the program is modified to match the predicted state, it goes on to produce a move which is usually valid for the new board state.
- 29athrowaway 4y agoPeople seem to forget about word2vec
- dsabanin 4y agoI think the real question is – do we?
- deleted 4y ago[deleted]
- HarHarVeryFunny 4y agoI think we're all trying to grok what LLMs like ChatGPT are really doing, and I think the answer really is that it has developed a "world model" for want of better words. Of course an LLM is by design only trained to predict next word (based on the language statistics it has learned) so it's tempting to say that it's just using "surface statistics", but that's a bit too dismissive and ignores the emergent capabilities which indicates there's rather more going on... The thing is, to be a REALLY good "language model" you need to go well beyond grammar or short-range statistical predictions. To be a REALLY good language model you need to learn (and evidentially if based on a large enough transformer, can learn) about abstract contexts so that you can maintain context while generating something likely/appropriate given all the nuances of prompt (whether that's requesting a haiku, or python code, or a continuation of a fairy tale, etc, etc). I guess one way to regard this is that it's learnt statistical patterns on many different levels of hierarchy, and specific to many different contexts (fairy tale vs python code, etc), but of course this means that it's representing and maintaining these (deep, hierarchical) long-range contexts while generating word-by-word output, so it seems inappropriate to just call these "surface statistics", and more descriptive to refer to it as the "world model" it has learned. One indication of the level of abstraction of this world model was shown by a recent paper which proved that the model is representing in it's internal activations whether it's input is true or not (correctly predicting that it will regard the negation as false), which can only reflect that this is a concept/context is had to learn to predict well in some circumstances. For example, if generating a news story then it's going to be best to maintain a truthy context, but for a fairly tale not so much! I think how we describe, and understand, these very capable LLMs needs to go beyond their mechanics and training goals and reflect what we can deduce they've learned and what they are capable of. If the model is (literally) representing concepts as abstract as truth, then that seems to go far beyond what might be reasonably be called "surface statistics". While I think these architectures need to be elaborated to add key capabilities needed for AGI, its perhaps also worth noting that the other impressive "predictive intelligence", our brain, a wetware machine, could also be regarded as generating behavior only based on learned statistics, but at some point deep hierarchical context-dependent statistics are best called something else - a world model.
- meow_mix 4y agoBtw this post is based on a preprint - might be good to wait a lil https://arxiv.org/pdf/2210.13382.pdf https://arxiv.org/pdf/2210.13382.pdf
- NHQ 4y agoDoes a model learns a model, By Dr. Susse
- manv1 4y agoCan any of the AIs come up with the Fosbury flop? It sounds vaguely like AlphaGo can. If so, we should celebrate, panic, or do both. https://en.wikipedia.org/wiki/Fosbury_flop https://en.wikipedia.org/wiki/Fosbury_flop Most of human "progress" has been done by a few, with the rest of us finishing up the details (if that). Is a goal of AI to enhance those few, or to take over for the rest of us?
- codeulike 4y agoThis is fascinating paper looking at something really important. But I don't really understand the Probes at all. Can someone explain it simply? Like, the classifiers are trained by showing them internal states of AIs, but where do you get the training data for the classifiers from?