3 ms·

This subjects comes up in every single discussion on LLMs, and every time someone asserts with perfect confidence that LLMs couldn't possibly have a world model

This subjects comes up in every single discussion on LLMs, and every time someone asserts with perfect confidence that LLMs couldn't possibly have a world model. But that confidence is based on intuition, not on any empirical observations. All the research we've seen so far points in the other direction: sufficiently trained LLMs do have a world model that you can find in their weights, changing these weights changes their completions in a way consistent with the new world model, etc. See OthelloGPT for the most blatant example. I'm starting to wish we had something like a FAQ to point to, with a summary of all the research on the subject, because it's pretty settled by now.

happytiger 3y ago
Any network of sufficiently complex connected nodes develops characteristics of complexity that mirror intelligence. This was always the conjecture of the “ghost in the machine” and mech futurist sci-fi writers. Would you be willing to provide some reading about this? I’d love to see 1-2 things that could help me (and others) know more about what you know.
pdonis 3y ago
> LLMs couldn't possibly have a world model I'm not asserting that they "couldn't possibly". I'm asserting that they don't--as in, none of the LLMs we have today have a world model. Perhaps someone might invent a different something in the future that they call an "LLM" that does have a world model, but no such thing exists now. > that confidence is based on intuition, not on any empirical observations No, it's not based on either of those. It's based on the explicit descriptions of how existing LLMs work by the people who made them. Their descriptions make clear that LLMs are just confabulating text based on a given prompt and their training data. The LLM doesn't make any connection at all between the text and anything else. Contrast, for example, with Wolfram Alpha. If you give Wolfram Alpha the prompt "What is the distance from New York to Tokyo", it doesn't confabulate text based on that prompt and some corpus of training data. It first parses the text and infers that the correct response involves a lookup in its geographic database (note that you can skip this step by explicitly selecting the particular tool that does this); then it does the lookup; then it renders the response into text form and outputs it. That is what having a (rudimentary) world model looks like. LLMs do nothing of the sort.

happytiger 3y ago

Any network of sufficiently complex connected nodes develops characteristics of complexity that mirror intelligence. This was always the conjecture of the “ghost in the machine” and mech futurist sci-fi writers. Would you be willing to provide some reading about this? I’d love to see 1-2 things that could help me (and others) know more about what you know.

pdonis 3y ago

> LLMs couldn't possibly have a world model I'm not asserting that they "couldn't possibly". I'm asserting that they don't--as in, none of the LLMs we have today have a world model. Perhaps someone might invent a different something in the future that they call an "LLM" that does have a world model, but no such thing exists now. > that confidence is based on intuition, not on any empirical observations No, it's not based on either of those. It's based on the explicit descriptions of how existing LLMs work by the people who made them. Their descriptions make clear that LLMs are just confabulating text based on a given prompt and their training data. The LLM doesn't make any connection at all between the text and anything else. Contrast, for example, with Wolfram Alpha. If you give Wolfram Alpha the prompt "What is the distance from New York to Tokyo", it doesn't confabulate text based on that prompt and some corpus of training data. It first parses the text and infers that the correct response involves a lookup in its geographic database (note that you can skip this step by explicitly selecting the particular tool that does this); then it does the lookup; then it renders the response into text form and outputs it. That is what having a (rudimentary) world model looks like. LLMs do nothing of the sort.