4 ms·

Wow I'm a total novice but as far as I know most of the optimization methods for models (Pruning, Weight Sharing, Quantization, Low Rank Approximation, Binary /

Wow I'm a total novice but as far as I know most of the optimization methods for models (Pruning, Weight Sharing, Quantization, Low Rank Approximation, Binary / Ternary Net, Winograd Transformation) aren't really implemented in most ML frameworks. Would be fun to try it out one of these weekends

edhu2017 8y ago
Pruning, Weight Sharing and Quantization are commonly used techniques and can be done in tf.
- gandreani 8y ago
  Are you talking about these? https://www.tensorflow.org/api_docs/python/tf/contrib/model_pruning https://www.tensorflow.org/api_docs/python/tf/contrib/model_...
twtw 8y ago
Winograd transform probably wouldn't be exposed at the framework layer. I wouldn't be surprised if cuDNN/TensorRT/TPU driver already use it internally.
- liuliu 8y ago
  It would actually be interesting to quantize Winograd transformed weights directly and then see whether we can squeeze some accuracy out (since Winograd is more accurate numerically than direct convolution). Maybe just too much work for potentially marginal improvement and you will get better result when incorporating quantization into your training process anyway.
  - borramakot 8y ago
    Is there a way to do nonlinearity (ReLU) in the Winograd domain?
    - twtw 8y ago
      Not an answer, but I've never heard it called the "Winograd domain." The Winograd transform is an algorithm for computing small discrete Fourier transforms, so its the same Fourier domain.
    - liuliu 8y ago
      Assuming you mean Fourier domain. It is not very useful to keep the activation in that domain. Winograd transform operates at 4x4 tiles (it is effective because 16 FMA of that 4x4 tiles can generate 2x2 activations (roughly speaking), while direct convolution requires 3x3 FMA to generate 1 activation). If you keep the activation in that domain, you are looking at keeping 4x4 / 2x2 thus 4x more information than it is needed, which is not useful or desirable. Thus, the most optimizations are just keep the transformed weights (you are still looking at 4x4 / 3x3, 1.9x more data), but that can be somewhat justified by the computation cost. The transformed activations are not useful.
- jph00 8y ago
  cuDNN certainly does, if you use the CUDNN_CONVOLUTION_BWD_DATA_ALGO_WINOGRAD option (and you can ask it to automatically pick the best approach if you like).

edhu2017 8y ago

Pruning, Weight Sharing and Quantization are commonly used techniques and can be done in tf.

gandreani 8y ago
Are you talking about these? https://www.tensorflow.org/api_docs/python/tf/contrib/model_pruning https://www.tensorflow.org/api_docs/python/tf/contrib/model_...

twtw 8y ago

Winograd transform probably wouldn't be exposed at the framework layer. I wouldn't be surprised if cuDNN/TensorRT/TPU driver already use it internally.

liuliu 8y ago
It would actually be interesting to quantize Winograd transformed weights directly and then see whether we can squeeze some accuracy out (since Winograd is more accurate numerically than direct convolution). Maybe just too much work for potentially marginal improvement and you will get better result when incorporating quantization into your training process anyway.
- borramakot 8y ago
  Is there a way to do nonlinearity (ReLU) in the Winograd domain?
  - twtw 8y ago
    Not an answer, but I've never heard it called the "Winograd domain." The Winograd transform is an algorithm for computing small discrete Fourier transforms, so its the same Fourier domain.
  - liuliu 8y ago
    Assuming you mean Fourier domain. It is not very useful to keep the activation in that domain. Winograd transform operates at 4x4 tiles (it is effective because 16 FMA of that 4x4 tiles can generate 2x2 activations (roughly speaking), while direct convolution requires 3x3 FMA to generate 1 activation). If you keep the activation in that domain, you are looking at keeping 4x4 / 2x2 thus 4x more information than it is needed, which is not useful or desirable. Thus, the most optimizations are just keep the transformed weights (you are still looking at 4x4 / 3x3, 1.9x more data), but that can be somewhat justified by the computation cost. The transformed activations are not useful.
jph00 8y ago
cuDNN certainly does, if you use the CUDNN_CONVOLUTION_BWD_DATA_ALGO_WINOGRAD option (and you can ask it to automatically pick the best approach if you like).