3 ms·

The author's quote is asserting the opposite of what you're saying here. Filling in "GPT" for x, "text" for y, and "world" for z, the author is stating "if GPT

The author's quote is asserting the opposite of what you're saying here. Filling in "GPT" for x, "text" for y, and "world" for z, the author is stating "if GPT models text, and text models the world, then GPT models the world." In particular, the author directly addresses your point that "Generalizing a bit, our 'shadow' model of reality in every single domain is far more powerful than language's approximation." > Modality, uncertainty, vagueness and other complexities enter but the isomorphism between world and language is there, even if inexact. My own personal view tends to lean towards the author's. GPT-3 , despite its many astonishing achievements, has many many limitations. Yet I don't see any indication as of yet that any of those limitations are fundamental ones inherent to the approach behind the GPT series, which is why GPT-3 both excites and frightens me.

akakww 6y ago
GPT models the world through text* There are many other things that go into modelling the world, like drawing pictures, visualising things, communicating in non-text ways essentially. GPT can only model the world in the ways that text can model the world, which is limited.
- dwohnitmok 6y ago
  Although a differentiating factor in the near-term, in the grand scheme of things that is merely an implementation detail. The rise of computing and its myriad encoders is an existence proof that the world we humans know can be, to an arbitrary degree of precision, modeled by 0s and 1s and what are 0s and 1s but the purest essence of text? Although fraught with many large implementation difficulties, a GPT trained on binary data does not represent a fundamental difficulty vs one trained on text.
  - randcraw 6y ago
    No, and not even close. Text cannot model many things. Try learning surgery using a textbook without pictures. Try driving a racecar competitively or learning a foreign language after only reading a book, even one with pictures. Models of all kinds are always poor surrogates for reality, and models that cannot employ logic or causality CLEARLY cannot model a world in which mechanisms cause all change. Statistical models can indeed describe many observations, but can never break out from the echo chamber of copy-catting patterns it has already seen. If a probabilistic engine like a deep net hasn’t been exposed to a concept during training, it will never induce its existence. Imagination requires initiative, the proposal of an unknown and unfamiliar outcome via logical inference or causal induction. Until deep nets can employ both of these skills, they will never master many skills humans use routinely to explain the world or extend our understanding of how it works.
    - FeepingCreature 6y ago
      If logic helps you predict the next word in the text, then a text based model will learn logic.
- p1esk 6y ago
  What makes you think it’s limited to text? See for example https://openai.com/blog/jukebox https://openai.com/blog/jukebox
rland 6y ago
Yes, fundamentally I agree, of course. If you could find a way to get all of natural knowledge into a sufficient enough network, you would probably have a very intelligent network. But getting text in is easy. It's the most manipulable proxy model. Getting other data in, data that represents what I'm talking about, is much much more difficult.
- p1esk 6y ago
  Getting in, say, video frames instead of words is more difficult, but possible imo. After all, they demonstrated it’s possible to do with raw audio: https://openai.com/blog/jukebox https://openai.com/blog/jukebox

akakww 6y ago

GPT models the world through text* There are many other things that go into modelling the world, like drawing pictures, visualising things, communicating in non-text ways essentially. GPT can only model the world in the ways that text can model the world, which is limited.

dwohnitmok 6y ago
Although a differentiating factor in the near-term, in the grand scheme of things that is merely an implementation detail. The rise of computing and its myriad encoders is an existence proof that the world we humans know can be, to an arbitrary degree of precision, modeled by 0s and 1s and what are 0s and 1s but the purest essence of text? Although fraught with many large implementation difficulties, a GPT trained on binary data does not represent a fundamental difficulty vs one trained on text.
- randcraw 6y ago
  No, and not even close. Text cannot model many things. Try learning surgery using a textbook without pictures. Try driving a racecar competitively or learning a foreign language after only reading a book, even one with pictures. Models of all kinds are always poor surrogates for reality, and models that cannot employ logic or causality CLEARLY cannot model a world in which mechanisms cause all change. Statistical models can indeed describe many observations, but can never break out from the echo chamber of copy-catting patterns it has already seen. If a probabilistic engine like a deep net hasn’t been exposed to a concept during training, it will never induce its existence. Imagination requires initiative, the proposal of an unknown and unfamiliar outcome via logical inference or causal induction. Until deep nets can employ both of these skills, they will never master many skills humans use routinely to explain the world or extend our understanding of how it works.
  - FeepingCreature 6y ago
    If logic helps you predict the next word in the text, then a text based model will learn logic.
p1esk 6y ago
What makes you think it’s limited to text? See for example https://openai.com/blog/jukebox https://openai.com/blog/jukebox

rland 6y ago

Yes, fundamentally I agree, of course. If you could find a way to get all of natural knowledge into a sufficient enough network, you would probably have a very intelligent network. But getting text in is easy. It's the most manipulable proxy model. Getting other data in, data that represents what I'm talking about, is much much more difficult.

p1esk 6y ago
Getting in, say, video frames instead of words is more difficult, but possible imo. After all, they demonstrated it’s possible to do with raw audio: https://openai.com/blog/jukebox https://openai.com/blog/jukebox