3 ms·

There is some research trying to analyze and explain how and why it learns. https://transformer-circuits.pub/2021/framework/index.html https://transformer-circ

There is some research trying to analyze and explain how and why it learns. https://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2023/privileged-basis/index.html https://transformer-circuits.pub/2023/privileged-basis/index... https://distill.pub/2020/circuits/ https://distill.pub/2020/circuits/ But I would not expect that we will really understand in detail how everything works. But do we need to? We also don't understand how the human brain works, but it is still useful.

ChatGTP 4y ago
The answer is yes. Otherwise the answer should be, we can't really trust the output and it will need to be treated rather suspiciously,just like we have to treat human outputs. At least humans can generally explain their rationale and be hold accountable.
- albertzeyer 4y ago
  GTP can also explain its reasoning. But that does not tell at all whether this reasoning is really accurate or correct. The same as for humans. When you ask them for some reasoning, they will give you sth, but it doesn't mean that is their real reasoning. There is always a lot of subjective feeling involved which you cannot really formalize. For both GPT and humans. You can't really trust the output of humans. Still, they are somewhat useful.
  - p-e-w 4y ago
    As mentioned by GP, though, humans can be held accountable. I believe that is the main reason why people can (sometimes) be trusted: They worry about what will happen to them if they break that trust. There is no reason to assume that current and future AIs have anything resembling that mechanism.
simonh 4y ago
The more you understand something the better you can optimise, improve and engineer it. There’s also a matter of trust, particularly on issues like alignment. It’s hard to trust someone if you don’t understand their motivations.

ChatGTP 4y ago

The answer is yes. Otherwise the answer should be, we can't really trust the output and it will need to be treated rather suspiciously,just like we have to treat human outputs. At least humans can generally explain their rationale and be hold accountable.

albertzeyer 4y ago
GTP can also explain its reasoning. But that does not tell at all whether this reasoning is really accurate or correct. The same as for humans. When you ask them for some reasoning, they will give you sth, but it doesn't mean that is their real reasoning. There is always a lot of subjective feeling involved which you cannot really formalize. For both GPT and humans. You can't really trust the output of humans. Still, they are somewhat useful.
- p-e-w 4y ago
  As mentioned by GP, though, humans can be held accountable. I believe that is the main reason why people can (sometimes) be trusted: They worry about what will happen to them if they break that trust. There is no reason to assume that current and future AIs have anything resembling that mechanism.

simonh 4y ago

The more you understand something the better you can optimise, improve and engineer it. There’s also a matter of trust, particularly on issues like alignment. It’s hard to trust someone if you don’t understand their motivations.