3 ms·

Kornia: Differential Computer Vision

69 points by ArtWomb 4y ago

radarsat1 4y ago
I'm really interested in using this library, but I have to admit I'm struggling to think of use cases. I can see differentiability being useful in feature extraction and maybe some uses of augmentation like colour correction, or spatial tranformations -- for example in the image registration example they give -- but overall the kinds of operations offered by the library tend to be used for preprocessing, and not really needed in the trainable pipeline. For instance edge detectors and even most feature detectors are basically hard-coded kernels, are they not? And you wouldn't apply them on intermediate representations, they are only really valid operations for the input layer, and not needed during back-propagation. I find the examples interesting, but mostly are things that could be done using OpenCV and don't exploit the differentiability in any meaningful way. But maybe I am missing something.
skainswoo 4y ago
BTW title should be s/Differential/Differentiable
- a3w 4y ago
  As a non-native-english speaker foreign to ML and CV, what does that mean?
  - rsfern 4y ago
    This library is super cool because it implements a ton of important classical computer vision methods in a way that is compatible with automatic differentiation. So you can combine modern deep learning methods with classical methods
  - andersource 4y ago
    In case you're asking about the parent comment (s/Differential/Differentiable), it means the title should be "differentiable CV" (instead of "differential CV"). "Differentiable" describes some computation for which derivatives can be computed. "Differential" is a more general term which means that something has to do with differences, e.g. differential equations deal with equations that specify how things change together. With the recent surge in deep learning came significant improvements in optimization techniques and hardware, making it feasible to formulate some computations in a differentiable manner. Doing that allows one to optimize the computation process relatively efficiently, at least in theory. Some other examples: differentiable programming[0] (other differentiable techniques are a subset of this), differentiable rendering[1], differentiable signal processing[2]. [0] https://en.wikipedia.org/wiki/Differentiable_programming https://en.wikipedia.org/wiki/Differentiable_programming [1] https://arxiv.org/abs/2006.12057 https://arxiv.org/abs/2006.12057 [2] https://github.com/magenta/ddsp https://github.com/magenta/ddsp

69 points by ArtWomb 4y ago

I'm really interested in using this library, but I have to admit I'm struggling to think of use cases. I can see differentiability being useful in feature extraction and maybe some uses of augmentation like colour correction, or spatial tranformations -- for example in the image registration example they give -- but overall the kinds of operations offered by the library tend to be used for preprocessing, and not really needed in the trainable pipeline. For instance edge detectors and even most feature detectors are basically hard-coded kernels, are they not? And you wouldn't apply them on intermediate representations, they are only really valid operations for the input layer, and not needed during back-propagation. I find the examples interesting, but mostly are things that could be done using OpenCV and don't exploit the differentiability in any meaningful way. But maybe I am missing something.

skainswoo 4y ago

BTW title should be s/Differential/Differentiable

a3w 4y ago
As a non-native-english speaker foreign to ML and CV, what does that mean?
- rsfern 4y ago
  This library is super cool because it implements a ton of important classical computer vision methods in a way that is compatible with automatic differentiation. So you can combine modern deep learning methods with classical methods
- andersource 4y ago
  In case you're asking about the parent comment (s/Differential/Differentiable), it means the title should be "differentiable CV" (instead of "differential CV"). "Differentiable" describes some computation for which derivatives can be computed. "Differential" is a more general term which means that something has to do with differences, e.g. differential equations deal with equations that specify how things change together. With the recent surge in deep learning came significant improvements in optimization techniques and hardware, making it feasible to formulate some computations in a differentiable manner. Doing that allows one to optimize the computation process relatively efficiently, at least in theory. Some other examples: differentiable programming[0] (other differentiable techniques are a subset of this), differentiable rendering[1], differentiable signal processing[2]. [0] https://en.wikipedia.org/wiki/Differentiable_programming https://en.wikipedia.org/wiki/Differentiable_programming [1] https://arxiv.org/abs/2006.12057 https://arxiv.org/abs/2006.12057 [2] https://github.com/magenta/ddsp https://github.com/magenta/ddsp