4 ms·

Two sparsities are better than one: Performance of sparse-sparse networks

51 points by Anon84 5y ago

PaulHoule 5y ago
Sparsity could really be a good thing but so many times I've tried to use it and walked away disappointed in terms of accuracy.
- sailingparrot 5y ago
  It is already a good thing, but it currently requires a lot of engineering effort to actually get it to work with acceptable quality. It's not something that works out of the box like half precision, or for some models, int8. And to your point, for many production scenarios the ratio of engineering work vs performance gains is maybe not worth it. But for models that are going to handle massive load in inference it is worth it in my experience. I expect that this will be made much easier in future with better hardware support and smarter sparsification libraries.
  - synthos 5y ago
    It's the power savings that are the real goal. If you can create and (re)train a network for incredible sparsity you can fit inference in some pretty low power envelopes. I think to that end the work required to get to adequate perform is justified
Chinjut 5y ago
An abundance of sparsities.
md2020 5y ago
Always cool to see research from Numenta. They don’t get as much love as DeepMind, Google Brain, and OpenAI because their results aren’t as flashy, but I do feel like they’ve got a principled approach to engineering intelligent systems distinct from that of the big players.
yalok 5y ago
Sparse operations still don’t have good support in any of the generic SIMD instructions (avx2/512, Neon), or in GPU… And memory access/cache should also be optimized for them to have even more power saving and better speed. For ASICs/FPGAs implementations, however, there’s no such limit and perf gains are higher.
- rajnathani 5y ago
  > Sparse operations still don’t have good support in any of the generic SIMD instructions (avx2/512, Neon), or in GPU… This is incorrect. For GPUs: Nvidia A100 (launched in 2020) has sparse matrix multiplication support for inference. For CPUs: Check out SLIDE from 2019: https://arxiv.org/pdf/1903.03129.pdf https://arxiv.org/pdf/1903.03129.pdf

51 points by Anon84 5y ago

Sparsity could really be a good thing but so many times I've tried to use it and walked away disappointed in terms of accuracy.

sailingparrot 5y ago
It is already a good thing, but it currently requires a lot of engineering effort to actually get it to work with acceptable quality. It's not something that works out of the box like half precision, or for some models, int8. And to your point, for many production scenarios the ratio of engineering work vs performance gains is maybe not worth it. But for models that are going to handle massive load in inference it is worth it in my experience. I expect that this will be made much easier in future with better hardware support and smarter sparsification libraries.
- synthos 5y ago
  It's the power savings that are the real goal. If you can create and (re)train a network for incredible sparsity you can fit inference in some pretty low power envelopes. I think to that end the work required to get to adequate perform is justified

Chinjut 5y ago

An abundance of sparsities.

md2020 5y ago

Always cool to see research from Numenta. They don’t get as much love as DeepMind, Google Brain, and OpenAI because their results aren’t as flashy, but I do feel like they’ve got a principled approach to engineering intelligent systems distinct from that of the big players.

yalok 5y ago

Sparse operations still don’t have good support in any of the generic SIMD instructions (avx2/512, Neon), or in GPU… And memory access/cache should also be optimized for them to have even more power saving and better speed. For ASICs/FPGAs implementations, however, there’s no such limit and perf gains are higher.

rajnathani 5y ago
> Sparse operations still don’t have good support in any of the generic SIMD instructions (avx2/512, Neon), or in GPU… This is incorrect. For GPUs: Nvidia A100 (launched in 2020) has sparse matrix multiplication support for inference. For CPUs: Check out SLIDE from 2019: https://arxiv.org/pdf/1903.03129.pdf https://arxiv.org/pdf/1903.03129.pdf