4 ms·

Obviously debatable, but this seems wrong. It could easily be that we can't figure out how to do something but when presented with a solution understand it. Man

Obviously debatable, but this seems wrong. It could easily be that we can't figure out how to do something but when presented with a solution understand it. Many people understand public private key cryptography but could never in a million years come up with it. Same for much of physics, a lot of math etc. As for the sheer complexity of millions/billions of parameters - people can explain and understand molecular dynamics simulations perfectly well. They can't use a small number of words to explain a specific configuration, but it's explainable.

janalsncm 4y ago
I think there’s a major difference though. You can understand public key crypto at varying levels of complexity because it is a topic created from first principles. Any sub-topic within the umbrella of public key crypto can be explored further. In this sense the knowledge is like a large program. Neural networks aren’t like that. The whole thing works together simultaneously to derive a result. So I’m skeptical that there is a high level explanation other than “it’s doing it’s best to approximate the data it was trained on” which is not that useful.
- low_tech_love 4y ago
  There must be some kind of pattern in the high-dimensional space of NN decision making that can be used to explain its decision, at least broadly (given some well-defined query). I’m not saying it’s easy, and I’m not saying the end user will have access to it, but surely someone in the pipeline needs to be able to acquire that. Or else there will never be a serious certification process before general and widespread use.
  - janalsncm 4y ago
    I guess this raises the question of levels of explanation. What kind of explanation would you like to be able to get? It reminds me of something the physicist Tim Maudlin described on a recent podcast (I will give a poor summary): Suppose you have the spinning color wheel on your MacBook. You give it to a computer scientist who says that the program is in an infinite loop, and the wheel will never stop spinning. Then you give the computer to a superhuman physicist who studies all of the components including the state of the transistors and internal components and concludes that the wheel will spin for 9 years before the screen gives out. [1] The point is that both are correct, but we’d really prefer the first explanation, since it gives insight into what the program is doing. The problem is that neural networks aren’t like programs. They don’t have reducible internal components unless you reduce down to the weights of the network itself. [2] [1] https://m.youtube.com/watch?v=k0CQ_D7dwA0 https://m.youtube.com/watch?v=k0CQ_D7dwA0 [2] https://karpathy.medium.com/software-2-0-a64152b37c35 https://karpathy.medium.com/software-2-0-a64152b37c35
    - low_tech_love 4y ago
      Yes, I totally agree, and thanks for the Karpathy link, it’s really good. And yes I don’t know at which level is the sweet spot. But the fact that it’s software 2.0 means we need debugging 2.0 too.

janalsncm 4y ago

I think there’s a major difference though. You can understand public key crypto at varying levels of complexity because it is a topic created from first principles. Any sub-topic within the umbrella of public key crypto can be explored further. In this sense the knowledge is like a large program. Neural networks aren’t like that. The whole thing works together simultaneously to derive a result. So I’m skeptical that there is a high level explanation other than “it’s doing it’s best to approximate the data it was trained on” which is not that useful.

low_tech_love 4y ago
There must be some kind of pattern in the high-dimensional space of NN decision making that can be used to explain its decision, at least broadly (given some well-defined query). I’m not saying it’s easy, and I’m not saying the end user will have access to it, but surely someone in the pipeline needs to be able to acquire that. Or else there will never be a serious certification process before general and widespread use.
- janalsncm 4y ago
  I guess this raises the question of levels of explanation. What kind of explanation would you like to be able to get? It reminds me of something the physicist Tim Maudlin described on a recent podcast (I will give a poor summary): Suppose you have the spinning color wheel on your MacBook. You give it to a computer scientist who says that the program is in an infinite loop, and the wheel will never stop spinning. Then you give the computer to a superhuman physicist who studies all of the components including the state of the transistors and internal components and concludes that the wheel will spin for 9 years before the screen gives out. [1] The point is that both are correct, but we’d really prefer the first explanation, since it gives insight into what the program is doing. The problem is that neural networks aren’t like programs. They don’t have reducible internal components unless you reduce down to the weights of the network itself. [2] [1] https://m.youtube.com/watch?v=k0CQ_D7dwA0 https://m.youtube.com/watch?v=k0CQ_D7dwA0 [2] https://karpathy.medium.com/software-2-0-a64152b37c35 https://karpathy.medium.com/software-2-0-a64152b37c35
  - low_tech_love 4y ago
    Yes, I totally agree, and thanks for the Karpathy link, it’s really good. And yes I don’t know at which level is the sweet spot. But the fact that it’s software 2.0 means we need debugging 2.0 too.