5 ms·

This technique figures out how to find a subset of neural network edge weights that can replicate most of the full model's performance, and is indeed quite simp

by brilee 6y ago

This technique figures out how to find a subset of neural network edge weights that can replicate most of the full model's performance, and is indeed quite simple. The catch is that this sparse subset of NN edge weights has no structure, so it doesn't get efficient execution on a GPU. If you're on a CPU with no specialized matrix math, this does in fact cut down on execution costs, but if you have an embedded ML chip, this doesn't really help. Distillation is another shrinkage technique that is also pretty foolproof, but allows you to pick an arbitrary architecture - this way, you can make full use of whatever hardware you're deploying too.

walrus 6y ago
I'm just speculating (and haven't read the paper yet), but it may be possible to achieve similar speedups on GPUs by pruning the smallest 20% of blocks of size ≥K×K to produce block-sparse weights[0], rather than pruning the smallest 20% of weights. [0] https://openai.com/blog/block-sparse-gpu-kernels/ https://openai.com/blog/block-sparse-gpu-kernels/
lonelappde 6y ago
Does a GPU run a pruned network slower than unpruned, or just not as faster as the modern shrinkages would suggest?
- fxtentacle 6y ago
  A naive GPU implementation on a non-block sparse network will be just as slow as the full un-pruned network.
  - visarga 6y ago
    Another approach would be to lower precision. By going from 32bits to 16 or 8 you can get massive speedups in memory operations and computation.
    - fxtentacle 6y ago
      Yes, that'll work. But it's orthogonal to this paper.
      - yalok 6y ago
        The trick is to combine these 2 approaches. Quantizing to bytes helps a lot, but forces you to use much larger pruning group size to see any speed up from pruning.
    - sk0g 6y ago
      Nvidia's apex.amp does just this - attempts half-precision training where possible, and adjusts scaling factors during training. I achieved a very significant speedup using this on my RTX 2080 Super, though it doesn't support Windows that well.
cs702 6y ago
> The catch is that this sparse subset of NN edge weights has no structure, so it doesn't get efficient execution... I would love to be able to model each artificial neuron as a separate object in code, making it trivial to group/ungroup neurons into irregularly shaped and/or overlapping groups and easily shrink/expand/alter such groups on the fly without incurring a performance penalty. Easy pruning would be just one application. As you point out, right now, those things not practical with frameworks like PyTorch and TensorFlow if we want decent performance. Currently we have little choice but to pigeonhole all our designs explicitly into layers that operate on grid-like structures of fixed size and resort to things like masking so we can have fast multiply-sum operations. By coincidence I was just talking about this on an unrelated thread: https://news.ycombinator.com/item?id=23118348 https://news.ycombinator.com/item?id=23118348
- j-pb 6y ago
  > separate object in code I hope you don't mean object in the OOP sense, but in a more general way, because that would be the slowest thing ever, no matter how smart your compiler/framework. You'd probably be better off to just have each group as a copy of the original one. > not practical with frameworks like PyTorch and TensorFlow The problem is not the frameworks, the problem is hardware, and physics. We've hit the wall with clock speeds, we've hit the wall with pipeline depth, we've hit the wall with branch prediction. The only thing we have left is specialised hardware, parallel hardware, and bigger caches. But all of these only work with SIMD, because it makes data fetches predictable, it makes the cores simpler and it makes adding more compute cheaper (no separate fetch decode silicon required). For the memory hierarchy to work you want stuff packed like in ECS systems for games, or Matrices for ML. Even with specialised hardware, changing your neural architecture on the fly would be ridiculously expensive because of all the communication overhead.
  - cs702 6y ago
    The problem is more complex than "just" hardware. If I may paraphrase a recent paper out of Google Brain: the current systems (hardware and software) we use for for numerical computing are "stuck in a local basin of performance and programmability."[a] [a] https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=true https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=...
jb775 6y ago
> The catch is that this sparse subset of NN edge weights has no structure, so it doesn't get efficient execution on a GPU Could you elaborate a little on this aspect?
- jointpdf 6y ago
  I could be off-base here, but I think they’re saying that a sparse matrix (for holding the edge weights) is ultimately something like a list-of-lists format, rather than a dense n x m array (for which GPUs have optimized routines). Except I that there are libraries doing sparse matrix operations on GPU (https://developer.nvidia.com/cusparse https://developer.nvidia.com/cusparse).
ur-whale 6y ago
> The catch is that this sparse subset of NN edge weights has no structure, so it doesn't get efficient execution I you parallelize your computation along the training data dimension, the problem is embarrassingly parallel and the structure of your neural network does not matter on bit.
Eridrus 6y ago
Having seen a bunch of people fail to make distillation work in the sense of training a better model than by training the smaller model from scratch, I don't think it's really fool proof...
enchiridion 6y ago
How optimized are current neural networks to specific GPU architectures?
iddiid 6y ago
A month or so ago a paper about a deep learning model using a cpu beating the best (4000$ or so) nvidia gpu was on hn, the trick was a quick hashing technique similar to nearest neighbour, so the algorithm keep only the most active neurons. I think the gpu price was about 4000$ and the computer was 60% of that based on some ball park estimates from the comments in HN. Edited: This is the paper:https://news.ycombinator.com/item?id=22502131 https://news.ycombinator.com/item?id=22502131 Perhaps both techniques can be used together, since the paper take advantage of sparcity to beat gpu when computation are on a sparse subset.

by brilee 6y ago

walrus 6y ago

I'm just speculating (and haven't read the paper yet), but it may be possible to achieve similar speedups on GPUs by pruning the smallest 20% of blocks of size ≥K×K to produce block-sparse weights[0], rather than pruning the smallest 20% of weights. [0] https://openai.com/blog/block-sparse-gpu-kernels/ https://openai.com/blog/block-sparse-gpu-kernels/

lonelappde 6y ago

Does a GPU run a pruned network slower than unpruned, or just not as faster as the modern shrinkages would suggest?

fxtentacle 6y ago
A naive GPU implementation on a non-block sparse network will be just as slow as the full un-pruned network.
- visarga 6y ago
  Another approach would be to lower precision. By going from 32bits to 16 or 8 you can get massive speedups in memory operations and computation.
  - fxtentacle 6y ago
    Yes, that'll work. But it's orthogonal to this paper.
    - yalok 6y ago
      The trick is to combine these 2 approaches. Quantizing to bytes helps a lot, but forces you to use much larger pruning group size to see any speed up from pruning.
  - sk0g 6y ago
    Nvidia's apex.amp does just this - attempts half-precision training where possible, and adjusts scaling factors during training. I achieved a very significant speedup using this on my RTX 2080 Super, though it doesn't support Windows that well.

cs702 6y ago

> The catch is that this sparse subset of NN edge weights has no structure, so it doesn't get efficient execution... I would love to be able to model each artificial neuron as a separate object in code, making it trivial to group/ungroup neurons into irregularly shaped and/or overlapping groups and easily shrink/expand/alter such groups on the fly without incurring a performance penalty. Easy pruning would be just one application. As you point out, right now, those things not practical with frameworks like PyTorch and TensorFlow if we want decent performance. Currently we have little choice but to pigeonhole all our designs explicitly into layers that operate on grid-like structures of fixed size and resort to things like masking so we can have fast multiply-sum operations. By coincidence I was just talking about this on an unrelated thread: https://news.ycombinator.com/item?id=23118348 https://news.ycombinator.com/item?id=23118348

j-pb 6y ago
> separate object in code I hope you don't mean object in the OOP sense, but in a more general way, because that would be the slowest thing ever, no matter how smart your compiler/framework. You'd probably be better off to just have each group as a copy of the original one. > not practical with frameworks like PyTorch and TensorFlow The problem is not the frameworks, the problem is hardware, and physics. We've hit the wall with clock speeds, we've hit the wall with pipeline depth, we've hit the wall with branch prediction. The only thing we have left is specialised hardware, parallel hardware, and bigger caches. But all of these only work with SIMD, because it makes data fetches predictable, it makes the cores simpler and it makes adding more compute cheaper (no separate fetch decode silicon required). For the memory hierarchy to work you want stuff packed like in ECS systems for games, or Matrices for ML. Even with specialised hardware, changing your neural architecture on the fly would be ridiculously expensive because of all the communication overhead.
- cs702 6y ago
  The problem is more complex than "just" hardware. If I may paraphrase a recent paper out of Google Brain: the current systems (hardware and software) we use for for numerical computing are "stuck in a local basin of performance and programmability."[a] [a] https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=true https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=...

jb775 6y ago

> The catch is that this sparse subset of NN edge weights has no structure, so it doesn't get efficient execution on a GPU Could you elaborate a little on this aspect?

jointpdf 6y ago
I could be off-base here, but I think they’re saying that a sparse matrix (for holding the edge weights) is ultimately something like a list-of-lists format, rather than a dense n x m array (for which GPUs have optimized routines). Except I that there are libraries doing sparse matrix operations on GPU (https://developer.nvidia.com/cusparse https://developer.nvidia.com/cusparse).

ur-whale 6y ago

> The catch is that this sparse subset of NN edge weights has no structure, so it doesn't get efficient execution I you parallelize your computation along the training data dimension, the problem is embarrassingly parallel and the structure of your neural network does not matter on bit.

Eridrus 6y ago

Having seen a bunch of people fail to make distillation work in the sense of training a better model than by training the smaller model from scratch, I don't think it's really fool proof...

enchiridion 6y ago

How optimized are current neural networks to specific GPU architectures?

iddiid 6y ago

A month or so ago a paper about a deep learning model using a cpu beating the best (4000$ or so) nvidia gpu was on hn, the trick was a quick hashing technique similar to nearest neighbour, so the algorithm keep only the most active neurons. I think the gpu price was about 4000$ and the computer was 60% of that based on some ball park estimates from the comments in HN. Edited: This is the paper:https://news.ycombinator.com/item?id=22502131 https://news.ycombinator.com/item?id=22502131 Perhaps both techniques can be used together, since the paper take advantage of sparcity to beat gpu when computation are on a sparse subset.