4 ms·

I wonder if there are certain patterns that take a lot of parameters to express in an NN, that could be more efficiently represented using some other kind of lo

by Aron 9y ago

I wonder if there are certain patterns that take a lot of parameters to express in an NN, that could be more efficiently represented using some other kind of logic, and that could be automatically discovered by some variety of algorithm such that subsection of the NN is replaced with this alternate form. I mean a simple case is that I'm sure that an NN trained to do multiplication is less efficient than just running a multiply op in the hardware. I'm talking about the complicated scenario where some subset of the NN is performing a replaceable and inefficient function.

visarga 9y ago
Yes, this is a real technique. There are NNs that are mixed with regular programming. As data propagates through the code, a graph is created and gradients flow automatically backwards training the various neural net bits. All this is fully mixable with functions, loops, if's and math expressions, the only condition is that any instruction used has to allow for gradients to flow - so it needs to be able to assign blame correctly from outputs to inputs. A second technique is to use deep learning to learn from stack traces. Any old software could be stack-traced by inserting a few prints here and there. Then a NN could learn recursive algorithms just by trying to recreate the whole stack trace, not just the actual outputs. It's a way to distill plain old programming into NNs, by incorporating side information that is cheap to get. This would be useful to quickly teach a NN some algorithm while making it less brittle than symbolic approaches. Imagine how many algorithms could be extracted from conventional software.
- Aron 9y ago
  Thanks for that. Very interesting. On the first one, I assume we are still talking hand-generation\coordination of the procedural bits. I was waving my hands at possibly learning the topology of those bits, possibly even from recognizing them as being reproduced [inefficiently] in a trained NN. I don't think I've ever pondered that second technique before and it's very intriguing. Is there a canonical best-of-class in that category? Offhand, it sounds brutally hard to do. Also, I think DeepMind might have published something on an NN that learned to write a procedural program that wrote a sort algorithm. Is that related?
  - visarga 9y ago
    After some considerable digging I came out with these papers: Differentiable Programs with Neural Libraries - https://www.microsoft.com/en-us/research/wp-content/uploads/2017/03/main2.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... Making Neural Programming Architectures Generalize via Recursion - https://openreview.net/forum?id=BkbY4psgg https://openreview.net/forum?id=BkbY4psgg Neural Programmer Interpreters - https://arxiv.org/abs/1511.06279 https://arxiv.org/abs/1511.06279
    - Aron 9y ago
      Your first paper references the one I was thinking of: the Turing NN. Thanks. I hope you learned something useful while digging.

by Aron 9y ago

visarga 9y ago

Yes, this is a real technique. There are NNs that are mixed with regular programming. As data propagates through the code, a graph is created and gradients flow automatically backwards training the various neural net bits. All this is fully mixable with functions, loops, if's and math expressions, the only condition is that any instruction used has to allow for gradients to flow - so it needs to be able to assign blame correctly from outputs to inputs. A second technique is to use deep learning to learn from stack traces. Any old software could be stack-traced by inserting a few prints here and there. Then a NN could learn recursive algorithms just by trying to recreate the whole stack trace, not just the actual outputs. It's a way to distill plain old programming into NNs, by incorporating side information that is cheap to get. This would be useful to quickly teach a NN some algorithm while making it less brittle than symbolic approaches. Imagine how many algorithms could be extracted from conventional software.

Aron 9y ago
Thanks for that. Very interesting. On the first one, I assume we are still talking hand-generation\coordination of the procedural bits. I was waving my hands at possibly learning the topology of those bits, possibly even from recognizing them as being reproduced [inefficiently] in a trained NN. I don't think I've ever pondered that second technique before and it's very intriguing. Is there a canonical best-of-class in that category? Offhand, it sounds brutally hard to do. Also, I think DeepMind might have published something on an NN that learned to write a procedural program that wrote a sort algorithm. Is that related?
- visarga 9y ago
  After some considerable digging I came out with these papers: Differentiable Programs with Neural Libraries - https://www.microsoft.com/en-us/research/wp-content/uploads/2017/03/main2.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... Making Neural Programming Architectures Generalize via Recursion - https://openreview.net/forum?id=BkbY4psgg https://openreview.net/forum?id=BkbY4psgg Neural Programmer Interpreters - https://arxiv.org/abs/1511.06279 https://arxiv.org/abs/1511.06279
  - Aron 9y ago
    Your first paper references the one I was thinking of: the Turing NN. Thanks. I hope you learned something useful while digging.