4 ms·

I think this is great work. Interpretability is the worst problem in deep learning, as the lack of insight into what the model has learned prevents it from bein

I think this is great work. Interpretability is the worst problem in deep learning, as the lack of insight into what the model has learned prevents it from being useful for serious decision making.

jameshart 4y ago
How much insight into what humans have learned is necessary before you find them useful for serious decisionmaking?
- civilized 4y ago
  If I needed large numbers added together, I would trust a human who can explain their general addition algorithm to me, and I wouldn't trust an AI that can spit out some usually correct answers on small problems.
- Jensson 4y ago
  Humans as a group has proven to be able to build everything we have today. No AI has proven to do anything similar, there just isn't much data there to make us confident in their abilities thus far.
  - gateorade 4y ago
    Conversely though, how many people are killed in the world every single day simply because of human error? When (in the US) a 16 year old gets their license, we don't ask them to provide a formal proof of their driving technique that shows it's impossible for them to ever get in a crash. We say 'you've had the training, you've demonstrated that you can safely drive, be careful out there'.
    - mannykannot 4y ago
      We have a good understanding of this risk and have collectively chosen to accept it (while many countries have chosen not to accept it in the case of 16-year-olds.)
- hnews_account_1 4y ago
  The explicability of our decision making is like 95% of our progress. It is why we can identify biases evolution builds into us but we need to fight against. Or try and justify something beyond mere intuition. This is like asking how much math are humans using anyway.
- jojobas 4y ago
  You can question a human in an exam and ask to "show your work". If you can say solve an equation but can't explain why you did this or that transformation you'll be rightly failed. With current NNs you get an answer and that's it, there's no introspection.
retrac 4y ago
It's not just a practical problem; it's one of the most important philosophical problems in the area, too. Something like GPT-3 can do multi-digit arithmetic much better than chance, giving results for values it was certainly never trained on. Similarly, transfer learning, where you start training a model on some input less related to the task, and then switch to inputs closer to your task at the end, can substantially reduce total training time. The task can be radically different; to use GPT-3 as an example again, compared to starting with a completely randomized model, it reduces training by a factor of about 10x to go from PCM audio samples encoded as text patterns, or abstract art bitmaps encoded as text patterns, to English text. GPT-3 is learning something about arithmetic. It's learning something that is common to music, abstract art, and English text. It might be as simple as basic patterns from geometry and arithmetic (that's my guess). But no one could even begin to point you in the direction of what that structure it is teasing out really is.

I think this is great work. Interpretability is the worst problem in deep learning, as the lack of insight into what the model has learned prevents it from being useful for serious decision making.

jameshart 4y ago

How much insight into what humans have learned is necessary before you find them useful for serious decisionmaking?

civilized 4y ago
If I needed large numbers added together, I would trust a human who can explain their general addition algorithm to me, and I wouldn't trust an AI that can spit out some usually correct answers on small problems.
Jensson 4y ago
Humans as a group has proven to be able to build everything we have today. No AI has proven to do anything similar, there just isn't much data there to make us confident in their abilities thus far.
- gateorade 4y ago
  Conversely though, how many people are killed in the world every single day simply because of human error? When (in the US) a 16 year old gets their license, we don't ask them to provide a formal proof of their driving technique that shows it's impossible for them to ever get in a crash. We say 'you've had the training, you've demonstrated that you can safely drive, be careful out there'.
  - mannykannot 4y ago
    We have a good understanding of this risk and have collectively chosen to accept it (while many countries have chosen not to accept it in the case of 16-year-olds.)
hnews_account_1 4y ago
The explicability of our decision making is like 95% of our progress. It is why we can identify biases evolution builds into us but we need to fight against. Or try and justify something beyond mere intuition. This is like asking how much math are humans using anyway.
jojobas 4y ago
You can question a human in an exam and ask to "show your work". If you can say solve an equation but can't explain why you did this or that transformation you'll be rightly failed. With current NNs you get an answer and that's it, there's no introspection.

retrac 4y ago

It's not just a practical problem; it's one of the most important philosophical problems in the area, too. Something like GPT-3 can do multi-digit arithmetic much better than chance, giving results for values it was certainly never trained on. Similarly, transfer learning, where you start training a model on some input less related to the task, and then switch to inputs closer to your task at the end, can substantially reduce total training time. The task can be radically different; to use GPT-3 as an example again, compared to starting with a completely randomized model, it reduces training by a factor of about 10x to go from PCM audio samples encoded as text patterns, or abstract art bitmaps encoded as text patterns, to English text. GPT-3 is learning something about arithmetic. It's learning something that is common to music, abstract art, and English text. It might be as simple as basic patterns from geometry and arithmetic (that's my guess). But no one could even begin to point you in the direction of what that structure it is teasing out really is.