4 ms·

I think a really interesting area of further research into LLMs is going to be how to further decouple "intelligence" from "knowledge". It would be amazing to

I think a really interesting area of further research into LLMs is going to be how to further decouple "intelligence" from "knowledge". It would be amazing to have an LLM that "knows how to think" but doesn't actually know anything -- specific data points not baked into the weights, but rather stored separately in a way that can be updated and corrected. Perhaps a step forward is the "toolformer" router, making the LLM recognize when it is referencing a fact and have it look that up (use a "tool") instead of trying to infer it. I realize that even semantically this is kind of a fine line, hard facts vs inferences, so there's some philosophy to ponder here. What can be 'stored' and what can be 'calculated'. What kind of knowledge is innate to intelligent reasoning and what kind acts more like a lookup table. But, exactly because we can't easily define these two categories, I have a feeling that if LLMs are trained to actually learn by themselves how to distinguish them in a "fuzzy" way, we might start getting somewhere.

Y_Y 4y ago
If you want something that "knows how to think" then I don't think you'll find it in an LLM. You can't divorce knowledge from this kind of model, it guesses what word comes next based on its training on a corpus of knowledge.
- georgelyon 4y ago
  No, but you could build something that guesses the next step in a list of steps to access knowledge from an external source.
- gamegoblin 4y ago
  One could imagine training a much smaller LLM on synthetic data that didn't contain much or any real knowledge, but just examples of in-context reasoning. E.g. the training data might look like "a fooble is to a tooble as a fabble is to a" with the answer "tabble". So you feed it tons of these kind of nonsense training data that forces it to only learn the in-context reasoning part of language, and none of the world knowledge. That said, it is unclear to me how much value such a model would be. You could imagine a middle ground where it does have basic knowledge, such as dictionary definitions and things derivable from those. E.g. "A tree has leaves. Leaves are green. Therefore a tree has a part that is green." type stuff. So you could give it some amount of world-grounding and common sense knowledge, but nothing involving history, proper nouns, etc. I imagine you could make such a model much smaller than these giant LLMs.
  - chii 4y ago
    The reason, i suspect, that this method won't work, is because the training dataset has knowledge embedded in it, regardless of whether you think it only contains in-context logic. The evidence is that there are these "glitched" tokens : https://www.youtube.com/watch?v=WO2X3oZEJOA https://www.youtube.com/watch?v=WO2X3oZEJOA if you watched to the end of that video, it explores a very plausible reason why such glitched tokens exist - it is because the training dataset has a bunch of "nonsense" (a large dump of the subreddit that just counts a number, as well as various other "data"). So my hypothesis is that if you attempted the in-context logic training dataset, it would produce large number of glitched tokens (different ones, but equally as "illogical" to the user).
    - gamegoblin 4y ago
      Tokenization is purely an implementation detail. If OpenAI had cared, they could have deleted those obviously glitched tokens from their tokenizer. They just didn't inspect it carefully and/or care. GPT4 does not suffer from the same glitched tokens as GPT3, presumably because it uses a different tokenizer. Furthermore, there are LLMs that operate on single bytes instead of multi-character tokens, totally obviating that problem.
- pyinstallwoes 4y ago
  You can separate that from which someone can recite from that which someone can perform in practice or do. To remember facts and present them in language is different than being able to craft a bow.
zamnos 4y ago
Bing Chat, powered by GPT-4, looks stuff up on the Internet. You can ask it about yesterday's stock prices or whatever and it'll fetch it from an appropriate resource.
- Tostino 4y ago
  The noise on the search index seems like it introduces quite a few issues with accuracy. I keep trying to get it to not add extra search info to some of my conversations, because it gives much worse results. Blog-spam really sucks.
telotortium 4y ago
Bing Chat is doing something like that, although it was far from perfect. Langchain can also do this to some extent. Then Steve Hsu claims to have been able to get a model working very well that can answer questions from college textbooks (https://twitter.com/hsu_steve/status/1636337165775908864 https://twitter.com/hsu_steve/status/1636337165775908864). So it seems like someone will figure it out in a few years.

Y_Y 4y ago

If you want something that "knows how to think" then I don't think you'll find it in an LLM. You can't divorce knowledge from this kind of model, it guesses what word comes next based on its training on a corpus of knowledge.

georgelyon 4y ago
No, but you could build something that guesses the next step in a list of steps to access knowledge from an external source.
gamegoblin 4y ago
One could imagine training a much smaller LLM on synthetic data that didn't contain much or any real knowledge, but just examples of in-context reasoning. E.g. the training data might look like "a fooble is to a tooble as a fabble is to a" with the answer "tabble". So you feed it tons of these kind of nonsense training data that forces it to only learn the in-context reasoning part of language, and none of the world knowledge. That said, it is unclear to me how much value such a model would be. You could imagine a middle ground where it does have basic knowledge, such as dictionary definitions and things derivable from those. E.g. "A tree has leaves. Leaves are green. Therefore a tree has a part that is green." type stuff. So you could give it some amount of world-grounding and common sense knowledge, but nothing involving history, proper nouns, etc. I imagine you could make such a model much smaller than these giant LLMs.
- chii 4y ago
  The reason, i suspect, that this method won't work, is because the training dataset has knowledge embedded in it, regardless of whether you think it only contains in-context logic. The evidence is that there are these "glitched" tokens : https://www.youtube.com/watch?v=WO2X3oZEJOA https://www.youtube.com/watch?v=WO2X3oZEJOA if you watched to the end of that video, it explores a very plausible reason why such glitched tokens exist - it is because the training dataset has a bunch of "nonsense" (a large dump of the subreddit that just counts a number, as well as various other "data"). So my hypothesis is that if you attempted the in-context logic training dataset, it would produce large number of glitched tokens (different ones, but equally as "illogical" to the user).
  - gamegoblin 4y ago
    Tokenization is purely an implementation detail. If OpenAI had cared, they could have deleted those obviously glitched tokens from their tokenizer. They just didn't inspect it carefully and/or care. GPT4 does not suffer from the same glitched tokens as GPT3, presumably because it uses a different tokenizer. Furthermore, there are LLMs that operate on single bytes instead of multi-character tokens, totally obviating that problem.
pyinstallwoes 4y ago
You can separate that from which someone can recite from that which someone can perform in practice or do. To remember facts and present them in language is different than being able to craft a bow.

zamnos 4y ago

Bing Chat, powered by GPT-4, looks stuff up on the Internet. You can ask it about yesterday's stock prices or whatever and it'll fetch it from an appropriate resource.

Tostino 4y ago
The noise on the search index seems like it introduces quite a few issues with accuracy. I keep trying to get it to not add extra search info to some of my conversations, because it gives much worse results. Blog-spam really sucks.

telotortium 4y ago

Bing Chat is doing something like that, although it was far from perfect. Langchain can also do this to some extent. Then Steve Hsu claims to have been able to get a model working very well that can answer questions from college textbooks (https://twitter.com/hsu_steve/status/1636337165775908864 https://twitter.com/hsu_steve/status/1636337165775908864). So it seems like someone will figure it out in a few years.