3 ms·

I have thought about that as well. I think it should be pretty straightforward. You need an algorithm that can divide your graph (weights and activations) into

by woah 6y ago

I have thought about that as well. I think it should be pretty straightforward. You need an algorithm that can divide your graph (weights and activations) into batches where no member of a batch depends on values that have not yet been calculated. I think, but am not sure, that this will result in alternating batches of weights and activations. Then you can calculate the results of applying the weights in the first batch to every example in a minibatch in parallel. Then you move on to calculate the results of applying the activations in the second batch to the results of the first batch, and so on until you have finished all the batches. The hard part is making the batches, although it feels like the kind of classical algorithms 101 problem that has some great solutions out there already. Maybe it is an application of graph coloring or something. I have not really pursued this because I'm not sure if there is a benefit to breaking the neurons out of their orderly layers. It would be interesting to just try training a few randomly generated networks, but I haven't had the time so far. Email me at hi at jehan dot email if you want to talk about this.

cs702 6y ago
I'm not so sure. Say we want to build a CNN that accepts batches of images of different size without having to resize and/or pad them to the same size. Or say we want to build a convolutional layer in which kernels need not be rectangular and of the same shape and size but can be of different shapes and sizes (e.g., circular kernels that vary in size as we move away from the center of attention). Or say we want to remove connections with low weights (as proposed in this paper) and randomly add new connections between arbitrary individual neurons in different layers between backprop updates. These are just a few examples off the top of my head of things that in principle would be easy to code in any modern high-level language (e.g., Python, Swift) but look very hard to get to run efficiently using today's software + hardware infrastructure (PyTorch/TensorFlow + GPUs/TPUs).
- woah 6y ago
  Just need to find a way to put together batches of operations that can happen in parallel, right?
  - cs702 6y ago
    No. Unfortunately, the problem is more complex than that: https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=true https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=...
buildbot 6y ago
I’m not super familiar with the technique, but something like NEAT or hyperNEAT might be wgat you are looking for?

by woah 6y ago

cs702 6y ago

I'm not so sure. Say we want to build a CNN that accepts batches of images of different size without having to resize and/or pad them to the same size. Or say we want to build a convolutional layer in which kernels need not be rectangular and of the same shape and size but can be of different shapes and sizes (e.g., circular kernels that vary in size as we move away from the center of attention). Or say we want to remove connections with low weights (as proposed in this paper) and randomly add new connections between arbitrary individual neurons in different layers between backprop updates. These are just a few examples off the top of my head of things that in principle would be easy to code in any modern high-level language (e.g., Python, Swift) but look very hard to get to run efficiently using today's software + hardware infrastructure (PyTorch/TensorFlow + GPUs/TPUs).

woah 6y ago
Just need to find a way to put together batches of operations that can happen in parallel, right?
- cs702 6y ago
  No. Unfortunately, the problem is more complex than that: https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=true https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=...

buildbot 6y ago

I’m not super familiar with the technique, but something like NEAT or hyperNEAT might be wgat you are looking for?