4 ms·

We're not deep learning systems. So reasoning by analogy there isn't correct. (Aside: There have been experiments that show that many explanation for [everyda

by opless 8y ago

We're not deep learning systems. So reasoning by analogy there isn't correct. (Aside: There have been experiments that show that many explanation for [everyday] actions are, in fact made up after the decision to execute an action. Which gives some insight to the whole nature of free will - but that's another discussion entirely. However, indeed we're able to show our workings for higher level workings, like mathematical proofs. ) Deep learning systems, are examples of supervised learning. Once trained, they cannot adapt to new input. You have to retrain to accommodate new features that you want to capture. Deep learning systems, are in essence a very long equation that hides all meaning over the entire network - good luck in finding out why it came to a decision. You might get lucky and see a definitive triggering path through the network, but that's an exception to the rule, as far as I understand. (I'm willing to be proven wrong here.) Symbolic systems, can trivially be made to trace the code execution paths they made for any action they might take. (also - hi Ben! I recognise your blog!)

FakeComments 8y ago
I don’t think we’re able to show our workings for higher level things like proofs. For instance, I can’t tell you (accurately) how I composed the proof conceptually, ie what strategy it takes, or even determined the appropriate steps, ie how I determined to use a particular tactic. I can tell you the premises lead to the conclusion via a chain of reasoning, but that’s the artifact of my thought process, not anything about how it works. The output of proving is that chain of steps. So I think here is somewhere DL systems may have an advantage: they actually can freely introspect their thought process as part of their thought process.
- red75prime 8y ago
  > they actually can freely introspect their thought process as part of their thought process. Do you mean that they can in principle? I don't know existing systems, which can inspect their own weights and output a vector of confidence scores that they can recognize classes A, B, C, ...
  - deleted 8y ago
    [deleted]
  - p1esk 8y ago
    output a vector of confidence scores that they can recognize classes A, B, C, ... This is literally what neural networks do when classifying patterns.
    - red75prime 8y ago
      They classify patterns, not their own ability to classify patterns.
      - p1esk 8y ago
        Again, they literally produce confidence scores - probabilities that each prediction is correct. For example, say there are 3 classes, and the network is shown two different examples of class 2. Say it outputs a class probability distribution {0.02, 0.45, 0.43} for the first example, and {0.02, 0.9, 0.08} for the second. Even though in both cases it correctly identifies class 2, it's a lot more confident in its prediction in the second case.
        red75prime 8y ago
        I just say that there's no introspection in existing networks. There's no part, which can take the weights of a convolutional part of a network and images of a certain class, and then output confidence score of the convolutional part classification ability for that class. I don't say that it is possible or useful. I said that I don't know of any deep learning systems, which "[...] can freely introspect their thought process as part of their thought process."
- mindgam3 8y ago
  "I don’t think we’re able to show our workings for higher level things like proofs." I can't speak for proofs, but this is false in the case of higher level things like chess. I'm not a grandmaster, but I was a chess master at age 10 and I am ranked ~2400 in bullet chess. To the average person my ability to play chess is "magic". But to me it's not magic at all. I can explain my thought process at any time. It's all based on symbolic manipulation at progressively higher levels, i.e. clustering pieces into "chunks", connecting these chunks into higher level patterns like weak pawn structure or forks, and ultimately deciding on the best course of action by weighing all of the different high level patterns. Every step is rule-based logic which I can readily explain to anyone, even a chess novice. The part that appears "magic" is the ability to do all of these calculations in the blink of an eye. But that too is simply due to having trained so many of these patterns extensively at a young age. Anyone who can speak a language is doing the same thing, manipulating complex symbolic objects in real time at progressively higher levels (i.e. letters, words, sentences, paragraphs).
  - FakeComments 8y ago
    There’s chess theory, which is rule based and what you start off describing. But then you admit you don’t work directly with chess theory when selecting moves, there’s a trained black box evaluator that selects candidate moves, which you then select from via chess theory. That’s how you’re finding chess moves in the blink of an eye: you run a fuzzy approximation, then refine the results using higher level reasoning. But you don’t have access to the network doing the evaluation and can’t describe exactly how it operates, just that it was trained on chess theory. It’s that fuzzy reasoning to speed up the process of actually finding solutions that I was calling out as the source of the unknowns in our processing — and at least from my exposure to board games (and their players), it’s often the source of things like innovative moves.
    - mindgam3 8y ago
      "there’s a trained black box evaluator that selects candidate moves, which you then select from via chess theory" You're missing my point. There is no "trained black box evaluator." There is indeed a trained evaluator, but it is not a black box. It is fully understandable. If I gave you private chess lessons, I could teach you my heuristics. And eventually you would understand them enough to be able to teach them to others. This would not be possible if it were truly a black box.
ben_w 8y ago
> We're not deep learning systems. So reasoning by analogy there isn't correct. That’s why I’m asking which definition you’re using. It’s broad enough that some models are biologically plausible even though there is criticism that many others are not: https://arxiv.org/abs/1502.04156 https://arxiv.org/abs/1502.04156 > Deep learning systems, are examples of supervised learning. Once trained, they cannot adapt to new input. You have to retrain to accommodate new features that you want to capture. Supervised, unsupervised, and reinforcement; and there is work on both incremental learning (https://arxiv.org/abs/1712.02719 https://arxiv.org/abs/1712.02719) and avoiding catastrophic forgetting (https://arxiv.org/abs/1812.02464 https://arxiv.org/abs/1812.02464) > (also - hi Ben! I recognise your blog!) Oh no, I’m becoming famous. (Infamous?) :)
- opless 8y ago
  ooh interesting links -thanks! I'll read them later. Famous-ish :) We probably have similar views on brexit.

by opless 8y ago

FakeComments 8y ago

I don’t think we’re able to show our workings for higher level things like proofs. For instance, I can’t tell you (accurately) how I composed the proof conceptually, ie what strategy it takes, or even determined the appropriate steps, ie how I determined to use a particular tactic. I can tell you the premises lead to the conclusion via a chain of reasoning, but that’s the artifact of my thought process, not anything about how it works. The output of proving is that chain of steps. So I think here is somewhere DL systems may have an advantage: they actually can freely introspect their thought process as part of their thought process.

red75prime 8y ago
> they actually can freely introspect their thought process as part of their thought process. Do you mean that they can in principle? I don't know existing systems, which can inspect their own weights and output a vector of confidence scores that they can recognize classes A, B, C, ...
- deleted 8y ago
  [deleted]
- p1esk 8y ago
  output a vector of confidence scores that they can recognize classes A, B, C, ... This is literally what neural networks do when classifying patterns.
  - red75prime 8y ago
    They classify patterns, not their own ability to classify patterns.
    - p1esk 8y ago
      Again, they literally produce confidence scores - probabilities that each prediction is correct. For example, say there are 3 classes, and the network is shown two different examples of class 2. Say it outputs a class probability distribution {0.02, 0.45, 0.43} for the first example, and {0.02, 0.9, 0.08} for the second. Even though in both cases it correctly identifies class 2, it's a lot more confident in its prediction in the second case.
      - red75prime 8y ago
        I just say that there's no introspection in existing networks. There's no part, which can take the weights of a convolutional part of a network and images of a certain class, and then output confidence score of the convolutional part classification ability for that class. I don't say that it is possible or useful. I said that I don't know of any deep learning systems, which "[...] can freely introspect their thought process as part of their thought process."
mindgam3 8y ago
"I don’t think we’re able to show our workings for higher level things like proofs." I can't speak for proofs, but this is false in the case of higher level things like chess. I'm not a grandmaster, but I was a chess master at age 10 and I am ranked ~2400 in bullet chess. To the average person my ability to play chess is "magic". But to me it's not magic at all. I can explain my thought process at any time. It's all based on symbolic manipulation at progressively higher levels, i.e. clustering pieces into "chunks", connecting these chunks into higher level patterns like weak pawn structure or forks, and ultimately deciding on the best course of action by weighing all of the different high level patterns. Every step is rule-based logic which I can readily explain to anyone, even a chess novice. The part that appears "magic" is the ability to do all of these calculations in the blink of an eye. But that too is simply due to having trained so many of these patterns extensively at a young age. Anyone who can speak a language is doing the same thing, manipulating complex symbolic objects in real time at progressively higher levels (i.e. letters, words, sentences, paragraphs).
- FakeComments 8y ago
  There’s chess theory, which is rule based and what you start off describing. But then you admit you don’t work directly with chess theory when selecting moves, there’s a trained black box evaluator that selects candidate moves, which you then select from via chess theory. That’s how you’re finding chess moves in the blink of an eye: you run a fuzzy approximation, then refine the results using higher level reasoning. But you don’t have access to the network doing the evaluation and can’t describe exactly how it operates, just that it was trained on chess theory. It’s that fuzzy reasoning to speed up the process of actually finding solutions that I was calling out as the source of the unknowns in our processing — and at least from my exposure to board games (and their players), it’s often the source of things like innovative moves.
  - mindgam3 8y ago
    "there’s a trained black box evaluator that selects candidate moves, which you then select from via chess theory" You're missing my point. There is no "trained black box evaluator." There is indeed a trained evaluator, but it is not a black box. It is fully understandable. If I gave you private chess lessons, I could teach you my heuristics. And eventually you would understand them enough to be able to teach them to others. This would not be possible if it were truly a black box.

ben_w 8y ago

> We're not deep learning systems. So reasoning by analogy there isn't correct. That’s why I’m asking which definition you’re using. It’s broad enough that some models are biologically plausible even though there is criticism that many others are not: https://arxiv.org/abs/1502.04156 https://arxiv.org/abs/1502.04156 > Deep learning systems, are examples of supervised learning. Once trained, they cannot adapt to new input. You have to retrain to accommodate new features that you want to capture. Supervised, unsupervised, and reinforcement; and there is work on both incremental learning (https://arxiv.org/abs/1712.02719 https://arxiv.org/abs/1712.02719) and avoiding catastrophic forgetting (https://arxiv.org/abs/1812.02464 https://arxiv.org/abs/1812.02464) > (also - hi Ben! I recognise your blog!) Oh no, I’m becoming famous. (Infamous?) :)

opless 8y ago
ooh interesting links -thanks! I'll read them later. Famous-ish :) We probably have similar views on brexit.