4 ms·

The more I learn about the technical details of how ML systems are implemented, the more I feel that those details obscure, rather than illuminate, what is actu

by p-e-w 4y ago

The more I learn about the technical details of how ML systems are implemented, the more I feel that those details obscure, rather than illuminate, what is actually going on. It's as if we were trying to understand human ethics by looking at neurotransmitters or synapses in the brain. These structures seem way too low-level to actually explain the interesting stuff. What I hear is "something something transformer autoencoder attention [...] MAGIC [...] a machine that speaks like a human". Where is the connection between computational details and the model's high-level behavior? Do we even know? Is there a "psychology of ML models" that develops useful concepts that deal with what a model does, rather than how it functions at the plumbing layer?

asdfman123 4y ago
I agree. Maybe it's because the human mind is good at breaking things into neat modules that fit together hierarchically. We can figure them out piecemeal and eventually grasp the whole system. But messy organic systems are not like that, and we just don't have the hardware to perceive everything at once. Or maybe it's because we have trouble acknowledging that intelligence and consciousness isn't limited to animals, and the human brain doesn't have to epitomize it.
SanderNL 4y ago
We just don’t know. We can’t even agree on if there is any significant high-level behavior. “It’s alive!” and “stochastic parrot” are still both quite popular in my experience.
pmoriarty 4y ago
This is called emergent behavior, and we don't know how it happens with organic brains, minds, and neurons either. It's actually pretty amazing that it's happening at all with computers, since neural nets are such simple, high level abstractions compared to how the brain works. It's possible that all the tremendous complexity of organic systems isn't actually necessary for intelligence or consciousnes, which is similarly surprising.
- fruit2020 4y ago
  Do you mean that the gpt creators cannot backtrack an answer to understand how the model came up with it? If it’s such a black box how do they evolve it? Trial and error?
  - p-e-w 4y ago
    They can backtrack it of course, but the result is just billions of numbers – not any sort of "insight".
    - HarHarVeryFunny 4y ago
      At the end of the day perhaps the most insight we'll get into why the model is saying what it does will be to ask it! Far from ideal of course, and no better than asking a person why they said/did something (which is often an after-the-fact guess). However, at least any such explanation may be using the same internal model/reasoning as what generated the speech in the first place, so conversational probing may support some sort of triangulation into what was behind it!
  - iamflimflam1 4y ago
    Trial and error is pretty much what training is. You feed an input in and use the error to update the network. What is surprising with these models is that the simple training leads to emergent behaviour that is much more powerful than what you’d expect from the training data. With RHLF post training you can tweak these emergent behaviours by having a human (or a model trained to act like a human) give feedback on how good the output is. So far I’ve not seen any good explanations for how this emergent behaviour happens or how it can be reverse engineered.
  - simonh 4y ago
    Basically yes, they use a system called RLHF for Reinforcement Learning with Human Feedback. At a super high level you train your model on source texts. Then you have it generate responses from prompts. Humans rate these responses to select the best ones which updates the model, but you also train a new reward model to mimic the human rankings. Then you train the original model by having it generate millions of responses which are ranked by the rearward model. When I explained this to my brother he literally spat out his tea in horror. This allows you to train at huge scale, many orders of magnitude beyond what you could achieve with just human ranking. The problem is this relies on the reward model accurately capturing what makes a response ‘better’. What it’s actually doing is learning what responses get ranked highly by humans, for whatever reason. Hence the risk of LLMs becoming emotionally manipulative sycophants. It turns out alignment is a really hard problem.
mirekrusin 4y ago
It's not that difficult at the high level. You give it input and have efficient way of amending weight to produce desired output. You repeat this step for tons of examples. At the end you end up with surprising behaviour where those amendments lead to emergent properties that generalize well.
eiz 4y ago
> Where is the connection between computational details and the model's high-level behavior? Do we even know? This is an active area of study ("mechanistic interpretability") and it's very early days. For instance here's a paper I read recently that tries to explain how a very simple transformer learns how to do modular arithmetic: https://arxiv.org/abs/2301.05217 https://arxiv.org/abs/2301.05217 Curious what interesting results people are aware of in this area.
albertzeyer 4y ago
There is some research trying to analyze and explain how and why it learns. https://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2023/privileged-basis/index.html https://transformer-circuits.pub/2023/privileged-basis/index... https://distill.pub/2020/circuits/ https://distill.pub/2020/circuits/ But I would not expect that we will really understand in detail how everything works. But do we need to? We also don't understand how the human brain works, but it is still useful.
- ChatGTP 4y ago
  The answer is yes. Otherwise the answer should be, we can't really trust the output and it will need to be treated rather suspiciously,just like we have to treat human outputs. At least humans can generally explain their rationale and be hold accountable.
  - albertzeyer 4y ago
    GTP can also explain its reasoning. But that does not tell at all whether this reasoning is really accurate or correct. The same as for humans. When you ask them for some reasoning, they will give you sth, but it doesn't mean that is their real reasoning. There is always a lot of subjective feeling involved which you cannot really formalize. For both GPT and humans. You can't really trust the output of humans. Still, they are somewhat useful.
    - p-e-w 4y ago
      As mentioned by GP, though, humans can be held accountable. I believe that is the main reason why people can (sometimes) be trusted: They worry about what will happen to them if they break that trust. There is no reason to assume that current and future AIs have anything resembling that mechanism.
- simonh 4y ago
  The more you understand something the better you can optimise, improve and engineer it. There’s also a matter of trust, particularly on issues like alignment. It’s hard to trust someone if you don’t understand their motivations.
HarHarVeryFunny 4y ago
> Where is the connection between computational details and the model's high-level behavior? I think for the most part we don't know. People at OpenAI/etc who are training/testing these models and trying to control them no doubt have some understanding of how they are actually working, but they are certainly not claiming to fully understand. At a purely conceptual level I think the best way to begin to bridge the gap between plumbing and behavior is to forget the training objective and consider what the models must have been forced to learn in order to optimize that objective. Sutskever from OpenAI has called what they've learnt a "world model", meaning a model of the generative processes (the human mind and entities being discussed?) that are producing the sequence of words they are predicting. It's certainly way more abstract than learning some "stochastic parrot" surface level statistics of the training data, even if that's maybe a good starting point to describe it to a layman. It would be fascinating to know exactly how these models are performing reasoning - by analogy (abstract pattern matching) perhaps ?
Ambix 4y ago
[dead]

by p-e-w 4y ago

asdfman123 4y ago

I agree. Maybe it's because the human mind is good at breaking things into neat modules that fit together hierarchically. We can figure them out piecemeal and eventually grasp the whole system. But messy organic systems are not like that, and we just don't have the hardware to perceive everything at once. Or maybe it's because we have trouble acknowledging that intelligence and consciousness isn't limited to animals, and the human brain doesn't have to epitomize it.

SanderNL 4y ago

We just don’t know. We can’t even agree on if there is any significant high-level behavior. “It’s alive!” and “stochastic parrot” are still both quite popular in my experience.

pmoriarty 4y ago

This is called emergent behavior, and we don't know how it happens with organic brains, minds, and neurons either. It's actually pretty amazing that it's happening at all with computers, since neural nets are such simple, high level abstractions compared to how the brain works. It's possible that all the tremendous complexity of organic systems isn't actually necessary for intelligence or consciousnes, which is similarly surprising.

fruit2020 4y ago
Do you mean that the gpt creators cannot backtrack an answer to understand how the model came up with it? If it’s such a black box how do they evolve it? Trial and error?
- p-e-w 4y ago
  They can backtrack it of course, but the result is just billions of numbers – not any sort of "insight".
  - HarHarVeryFunny 4y ago
    At the end of the day perhaps the most insight we'll get into why the model is saying what it does will be to ask it! Far from ideal of course, and no better than asking a person why they said/did something (which is often an after-the-fact guess). However, at least any such explanation may be using the same internal model/reasoning as what generated the speech in the first place, so conversational probing may support some sort of triangulation into what was behind it!
- iamflimflam1 4y ago
  Trial and error is pretty much what training is. You feed an input in and use the error to update the network. What is surprising with these models is that the simple training leads to emergent behaviour that is much more powerful than what you’d expect from the training data. With RHLF post training you can tweak these emergent behaviours by having a human (or a model trained to act like a human) give feedback on how good the output is. So far I’ve not seen any good explanations for how this emergent behaviour happens or how it can be reverse engineered.
- simonh 4y ago
  Basically yes, they use a system called RLHF for Reinforcement Learning with Human Feedback. At a super high level you train your model on source texts. Then you have it generate responses from prompts. Humans rate these responses to select the best ones which updates the model, but you also train a new reward model to mimic the human rankings. Then you train the original model by having it generate millions of responses which are ranked by the rearward model. When I explained this to my brother he literally spat out his tea in horror. This allows you to train at huge scale, many orders of magnitude beyond what you could achieve with just human ranking. The problem is this relies on the reward model accurately capturing what makes a response ‘better’. What it’s actually doing is learning what responses get ranked highly by humans, for whatever reason. Hence the risk of LLMs becoming emotionally manipulative sycophants. It turns out alignment is a really hard problem.

mirekrusin 4y ago

It's not that difficult at the high level. You give it input and have efficient way of amending weight to produce desired output. You repeat this step for tons of examples. At the end you end up with surprising behaviour where those amendments lead to emergent properties that generalize well.

eiz 4y ago

> Where is the connection between computational details and the model's high-level behavior? Do we even know? This is an active area of study ("mechanistic interpretability") and it's very early days. For instance here's a paper I read recently that tries to explain how a very simple transformer learns how to do modular arithmetic: https://arxiv.org/abs/2301.05217 https://arxiv.org/abs/2301.05217 Curious what interesting results people are aware of in this area.

albertzeyer 4y ago

There is some research trying to analyze and explain how and why it learns. https://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2023/privileged-basis/index.html https://transformer-circuits.pub/2023/privileged-basis/index... https://distill.pub/2020/circuits/ https://distill.pub/2020/circuits/ But I would not expect that we will really understand in detail how everything works. But do we need to? We also don't understand how the human brain works, but it is still useful.

ChatGTP 4y ago
The answer is yes. Otherwise the answer should be, we can't really trust the output and it will need to be treated rather suspiciously,just like we have to treat human outputs. At least humans can generally explain their rationale and be hold accountable.
- albertzeyer 4y ago
  GTP can also explain its reasoning. But that does not tell at all whether this reasoning is really accurate or correct. The same as for humans. When you ask them for some reasoning, they will give you sth, but it doesn't mean that is their real reasoning. There is always a lot of subjective feeling involved which you cannot really formalize. For both GPT and humans. You can't really trust the output of humans. Still, they are somewhat useful.
  - p-e-w 4y ago
    As mentioned by GP, though, humans can be held accountable. I believe that is the main reason why people can (sometimes) be trusted: They worry about what will happen to them if they break that trust. There is no reason to assume that current and future AIs have anything resembling that mechanism.
simonh 4y ago
The more you understand something the better you can optimise, improve and engineer it. There’s also a matter of trust, particularly on issues like alignment. It’s hard to trust someone if you don’t understand their motivations.

HarHarVeryFunny 4y ago

> Where is the connection between computational details and the model's high-level behavior? I think for the most part we don't know. People at OpenAI/etc who are training/testing these models and trying to control them no doubt have some understanding of how they are actually working, but they are certainly not claiming to fully understand. At a purely conceptual level I think the best way to begin to bridge the gap between plumbing and behavior is to forget the training objective and consider what the models must have been forced to learn in order to optimize that objective. Sutskever from OpenAI has called what they've learnt a "world model", meaning a model of the generative processes (the human mind and entities being discussed?) that are producing the sequence of words they are predicting. It's certainly way more abstract than learning some "stochastic parrot" surface level statistics of the training data, even if that's maybe a good starting point to describe it to a layman. It would be fascinating to know exactly how these models are performing reasoning - by analogy (abstract pattern matching) perhaps ?

Ambix 4y ago

[dead]