5 ms·

> It can't model the generative process of the humans who created those training set samples because that generative process has different inputs - sensory ones

> It can't model the generative process of the humans who created those training set samples because that generative process has different inputs - sensory ones (in addition to auto-regressive ones). I think that’s too strong a statement. I would say that it’s very constrained in its ability to model that, but not having access to the same inputs doesn’t mean you can’t model a process. For example, we model hurricanes based on measurements taken from satellites. Those aren’t the actual inputs to the hurricane itself, but abstracted correlates of those inputs. An LLM does have access to correlates of the inputs to human writing, i.e. textual descriptions of sensory inputs.

shanusmagnus 2y ago
Brilliant analogy. And we can imagine that, in a sci-fi world where some super-being could act on a scale that would allow it to perturb the world in a fashion amenable to causing hurricanes, the hurricane model could be substantially augmented, for the same reason motor babbling in an infant leads to fluid motion as a child. What has been a revelation to me is how, even peering through this dark glass, titanic amounts of data allow quite useful world models to emerge, even if they're super limited -- a type of "bitter lesson" that suggests we're only at the beginning of what's possible. I expect robotics + LLM to drive the next big breakthroughs, perhaps w/ virtual worlds [1] as an intermediate step. [1] https://minedojo.org/ https://minedojo.org/
HarHarVeryFunny 2y ago
You can model a generative process, but it's necessarily an auto-regressive generative process, not the same as the originating generative process which is based on the external world. Human language, and other actions, exist on a range from almost auto-regressive (generating a stock/practiced phrase such as "have a nice day") to highly interactive ones. An auto-regressive model is obviously going to have more success modelling an auto-regressive generative process. Weather prediction is really a good case of the limitation of auto-regressive models, as well as models that don't accurately reflect the inputs to the process you are attempting to predict. "There's a low pressure front coming in, so the weather will be X, same as last time", works some of the time. A crude physical weather model based on limited data points, such as weather balloon inputs, or satellite observation of hurricanes, also works some of the time. But of course these models are sometimes hopelessly wrong too. My real point wasn't about the lack of sensory data, even though this does force a purely auto-regressive (i.e. wrong) model, but rather about the difference between a passive model (such as weather prediction), and an interactive one.
- nerdponx 2y ago
  The whole innovation of GPT and LLMs in general is that an autoregressive model can make alarmingly good next-token predictions with the right inductive bias, a large number of parameters, a long context window, and a huge training set. It turns out that human communication is quite a lot more "autoregressive" than people assumed it was up until now. And that includes some level of reasoning capability, arising out of a kind of brute force pattern matching. It has limits, of course, but it's amazing that it works as well as it does.
  - HarHarVeryFunny 2y ago
    It is amazing, and interesting. Although I used the word myself, I'm not sure that "autoregressive" is quite the right word to describe how LLMs work, or our brains. Maybe better to just call both "predictive". In both cases the predictive inputs include the sequence itself (or selected parts of it, at varying depths of representation), but also global knowledge, both factual and procedural (HOW to represent the sequence). In the case of our brain there are also many more inputs that may be used such as sensory ones (passive observations, or action feedback), emotional state, etc. Regardless of what predictive inputs are available to LLMs vs brains, it does seem that in a lot of cases the more constrained inputs of an LLM don't prevent it from sounding very human like (not surprising at some level given the training goal), and an LLM chat window does create a "level playing field" (i.e. impoverished input setting for the human) where each side only sees the other as a stream of text. Maybe in this setting, the human, when not reasoning, really isn't bringing much more predictive machinery to the table than the LLM/transformer! Notwithstanding the predictive nature of LLMs, I can't help but also see them just as expert systems of sorts, albeit ones that have derived their own rules (much pertaining to language) rather than being given them. This view better matches their nature as fixed repositories of knowledge, brittle where rules are missing, as opposed to something more brain-like and intelligent, capable of continual learning.
slashdave 2y ago
Indeed. If you provided a talented individual with a sufficient quantity and variety of video streams of travels in a city (like New York), that person would be able to draw you a map.

shanusmagnus 2y ago

Brilliant analogy. And we can imagine that, in a sci-fi world where some super-being could act on a scale that would allow it to perturb the world in a fashion amenable to causing hurricanes, the hurricane model could be substantially augmented, for the same reason motor babbling in an infant leads to fluid motion as a child. What has been a revelation to me is how, even peering through this dark glass, titanic amounts of data allow quite useful world models to emerge, even if they're super limited -- a type of "bitter lesson" that suggests we're only at the beginning of what's possible. I expect robotics + LLM to drive the next big breakthroughs, perhaps w/ virtual worlds [1] as an intermediate step. [1] https://minedojo.org/ https://minedojo.org/

HarHarVeryFunny 2y ago

You can model a generative process, but it's necessarily an auto-regressive generative process, not the same as the originating generative process which is based on the external world. Human language, and other actions, exist on a range from almost auto-regressive (generating a stock/practiced phrase such as "have a nice day") to highly interactive ones. An auto-regressive model is obviously going to have more success modelling an auto-regressive generative process. Weather prediction is really a good case of the limitation of auto-regressive models, as well as models that don't accurately reflect the inputs to the process you are attempting to predict. "There's a low pressure front coming in, so the weather will be X, same as last time", works some of the time. A crude physical weather model based on limited data points, such as weather balloon inputs, or satellite observation of hurricanes, also works some of the time. But of course these models are sometimes hopelessly wrong too. My real point wasn't about the lack of sensory data, even though this does force a purely auto-regressive (i.e. wrong) model, but rather about the difference between a passive model (such as weather prediction), and an interactive one.

nerdponx 2y ago
The whole innovation of GPT and LLMs in general is that an autoregressive model can make alarmingly good next-token predictions with the right inductive bias, a large number of parameters, a long context window, and a huge training set. It turns out that human communication is quite a lot more "autoregressive" than people assumed it was up until now. And that includes some level of reasoning capability, arising out of a kind of brute force pattern matching. It has limits, of course, but it's amazing that it works as well as it does.
- HarHarVeryFunny 2y ago
  It is amazing, and interesting. Although I used the word myself, I'm not sure that "autoregressive" is quite the right word to describe how LLMs work, or our brains. Maybe better to just call both "predictive". In both cases the predictive inputs include the sequence itself (or selected parts of it, at varying depths of representation), but also global knowledge, both factual and procedural (HOW to represent the sequence). In the case of our brain there are also many more inputs that may be used such as sensory ones (passive observations, or action feedback), emotional state, etc. Regardless of what predictive inputs are available to LLMs vs brains, it does seem that in a lot of cases the more constrained inputs of an LLM don't prevent it from sounding very human like (not surprising at some level given the training goal), and an LLM chat window does create a "level playing field" (i.e. impoverished input setting for the human) where each side only sees the other as a stream of text. Maybe in this setting, the human, when not reasoning, really isn't bringing much more predictive machinery to the table than the LLM/transformer! Notwithstanding the predictive nature of LLMs, I can't help but also see them just as expert systems of sorts, albeit ones that have derived their own rules (much pertaining to language) rather than being given them. This view better matches their nature as fixed repositories of knowledge, brittle where rules are missing, as opposed to something more brain-like and intelligent, capable of continual learning.

slashdave 2y ago

Indeed. If you provided a talented individual with a sufficient quantity and variety of video streams of travels in a city (like New York), that person would be able to draw you a map.