6 ms·

This work uses conditional computation to allow for "Outrageously large" networks which may work better in practice but which will be even harder to understand.

This work uses conditional computation to allow for "Outrageously large" networks which may work better in practice but which will be even harder to understand. I'm very interested in working on using this same sort of conditional computation to make reasonably sized neural networks easier to understand. Has anyone seen papers on this sort of work?

hadsed 7y ago
Can you elaborate on what you think might make it easier?
- sgillen 7y ago
  I’m imagining that by having conditional computation you’re basically partitioning the state space up, with each part of the space getting its own smaller network. this might allow you the generality of a larger network, but with the ability to hone in and analyze the smaller networks for certain parts of your state space.
rq1 7y ago
https://arxiv.org/pdf/1807.06699 https://arxiv.org/pdf/1807.06699
nl 7y ago
Neural networks are NOT hard to understand. Tools like LIME[1] have been around for a few years, and work really well for inspecting and understanding the decisions made by NNs. It's true that complex decisions are hard to explain, but LIME does a better job than (say) anyway of trying to explain a SVN+Human coded feature. LIME has been out since 2016. It and similar techniques are widely used in production systems. It's time this "NNs are hard to understand" idea died. [1] https://github.com/marcotcr/lime https://github.com/marcotcr/lime
- p1esk 7y ago
  How does it explain adversarial examples where changing a single pixel leads to class prediction change?
  - nl 7y ago
    https://openreview.net/pdf?id=BkpiPMbA- https://openreview.net/pdf?id=BkpiPMbA- Code: https://github.com/sunblaze-ucb/decision-boundaries https://github.com/sunblaze-ucb/decision-boundaries
    - p1esk 7y ago
      The problem is we have very little understanding of how decision boundaries are formed, and very little intuition about how the input manifold “looks like”. That’s why it’s so difficult to make sure the boundaries have large enough margin (i.e. robust for all reasonable samples). After reading papers like this it feels like while we make some progress towards understanding some aspects of NN operation, we uncover deeper aspects that we don’t understand. This process itself looks like gradient descent (saddles everywhere!), and we are still pretty far from the global minima. :)
      - nl 7y ago
        Sure. But have you ever tried explaining a SVN decision to an actual human before? I'm not convinced that a NN is any worse than that.
plutonorm 7y ago
What is it with needing to understand NNs? Its the same thing as not trusting a human to drive a car because you cannot understand every stage Of computation happening in their brain. If a neural network learns a task, test it well enough to know that it performs well enough in target domain before using it. Don't expect to be able to understand how it works and from there claim to know that it will work well and have more confidence in it. This approach barely works in standard software, let alone a neural network. Stop worrying and learn to love the NN, after all it is a mirror of your own ineffable nature.
- varjag 7y ago
  "Those are scary things, those gels. You know one suffocated a bunch of people in London a while back?" Yes, Joel's about to say, but Jarvis is back in spew mode. "No shit. It was running the subway system over there, perfect operational record, and then one day it just forgets to crank up the ventilators when it's supposed to. Train slides into station fifteen meters underground, everybody gets out, no air, boom." "These things teach themselves from experience, right?," Jarvis continues. "So everyone just assumed it had learned to cue the ventilators on something obvious. Body heat, motion, CO2 levels, you know. Turns out instead it was watching a clock on the wall. Train arrival correlated with a predictable subset of patterns on the digital display, so it started the fans whenever it saw one of those patterns." "Yeah. That's right." Joel shakes his head. "And vandals had smashed the clock, or something."
  - sgt101 7y ago
    To summarise: if a system is not understood there exists the possibility of sudden, unexpected harm. The system is unsafe. This is (probably) ok if the system is putting icing on donuts (you might get a bad batch) but is definitely not ok if the system is deciding on dosing levels for drugs or controlling machines that could suddenly smash into queues of school children.
    - varjag 7y ago
      Moreover, if (as customary in all technology) you build systems upon these systems, even the low malfunction probabilities will multiply into nearly assured failures. With a system you understand you can find the cause and fix the issue for all: this what allows us to build ever more complex systems over decades of engineering R&D. But for a system you don't understand you are at whim of cascading patterns of errors in underlying behavior.

hadsed 7y ago

Can you elaborate on what you think might make it easier?

sgillen 7y ago
I’m imagining that by having conditional computation you’re basically partitioning the state space up, with each part of the space getting its own smaller network. this might allow you the generality of a larger network, but with the ability to hone in and analyze the smaller networks for certain parts of your state space.

rq1 7y ago

https://arxiv.org/pdf/1807.06699 https://arxiv.org/pdf/1807.06699

nl 7y ago

Neural networks are NOT hard to understand. Tools like LIME[1] have been around for a few years, and work really well for inspecting and understanding the decisions made by NNs. It's true that complex decisions are hard to explain, but LIME does a better job than (say) anyway of trying to explain a SVN+Human coded feature. LIME has been out since 2016. It and similar techniques are widely used in production systems. It's time this "NNs are hard to understand" idea died. [1] https://github.com/marcotcr/lime https://github.com/marcotcr/lime

p1esk 7y ago
How does it explain adversarial examples where changing a single pixel leads to class prediction change?
- nl 7y ago
  https://openreview.net/pdf?id=BkpiPMbA- https://openreview.net/pdf?id=BkpiPMbA- Code: https://github.com/sunblaze-ucb/decision-boundaries https://github.com/sunblaze-ucb/decision-boundaries
  - p1esk 7y ago
    The problem is we have very little understanding of how decision boundaries are formed, and very little intuition about how the input manifold “looks like”. That’s why it’s so difficult to make sure the boundaries have large enough margin (i.e. robust for all reasonable samples). After reading papers like this it feels like while we make some progress towards understanding some aspects of NN operation, we uncover deeper aspects that we don’t understand. This process itself looks like gradient descent (saddles everywhere!), and we are still pretty far from the global minima. :)
    - nl 7y ago
      Sure. But have you ever tried explaining a SVN decision to an actual human before? I'm not convinced that a NN is any worse than that.

plutonorm 7y ago

What is it with needing to understand NNs? Its the same thing as not trusting a human to drive a car because you cannot understand every stage Of computation happening in their brain. If a neural network learns a task, test it well enough to know that it performs well enough in target domain before using it. Don't expect to be able to understand how it works and from there claim to know that it will work well and have more confidence in it. This approach barely works in standard software, let alone a neural network. Stop worrying and learn to love the NN, after all it is a mirror of your own ineffable nature.

varjag 7y ago
"Those are scary things, those gels. You know one suffocated a bunch of people in London a while back?" Yes, Joel's about to say, but Jarvis is back in spew mode. "No shit. It was running the subway system over there, perfect operational record, and then one day it just forgets to crank up the ventilators when it's supposed to. Train slides into station fifteen meters underground, everybody gets out, no air, boom." "These things teach themselves from experience, right?," Jarvis continues. "So everyone just assumed it had learned to cue the ventilators on something obvious. Body heat, motion, CO2 levels, you know. Turns out instead it was watching a clock on the wall. Train arrival correlated with a predictable subset of patterns on the digital display, so it started the fans whenever it saw one of those patterns." "Yeah. That's right." Joel shakes his head. "And vandals had smashed the clock, or something."
- sgt101 7y ago
  To summarise: if a system is not understood there exists the possibility of sudden, unexpected harm. The system is unsafe. This is (probably) ok if the system is putting icing on donuts (you might get a bad batch) but is definitely not ok if the system is deciding on dosing levels for drugs or controlling machines that could suddenly smash into queues of school children.
  - varjag 7y ago
    Moreover, if (as customary in all technology) you build systems upon these systems, even the low malfunction probabilities will multiply into nearly assured failures. With a system you understand you can find the cause and fix the issue for all: this what allows us to build ever more complex systems over decades of engineering R&D. But for a system you don't understand you are at whim of cascading patterns of errors in underlying behavior.