4 ms·
If we're allowed to do that, why can't my activation function be sin(x)?
by whatshisface 5y ago
If we're allowed to do that, why can't my activation function be sin(x)?
- deleted 5y ago[deleted]
- blackcat201 5y agoYou can use sin as activation function, but that would require careful initialization to avoid gradient explosion as you would ended up with a lot of points where gradient is simply zero. You can refer to Implicit Neural Representations with Periodic Activation Functions for more details.
- zarzavat 5y agoIt works perfectly if you don’t have any parameters.
- yobbo 5y agoIt can, but sin(x) has infinite number of extremes, and the gradients will vanish at those points. Activations will get stuck at 1 and -1 (x=π/2, 3π/2, ...). They set x+(1/a)*sin²(x) to be monotonic, which fixes this. Or you need to optimize without using gradients.
- PartiallyTyped 5y agoYou can actually do that. https://www.vincentsitzmann.com/siren/ https://www.vincentsitzmann.com/siren/
- sicp-enjoyer 5y agoI'm surprised to see almost no discussion of fourier series in that paper, considering fourier series is all about representing signals as linear combinations of sinusoidal functions.
- PartiallyTyped 5y agoYou may be interested in [1] where they go to a great extend to show that the convolution operation that we consider in DL is the dual of fourier series [2]. [1] https://geometricdeeplearning.com https://geometricdeeplearning.com [2] https://arxiv.org/pdf/2104.13478.pdf https://arxiv.org/pdf/2104.13478.pdf page 27 (23 if you count book pages).
- amelius 5y agoIs convolution in DL not implemented with the FFT as the underlying workhorse?
- PartiallyTyped 5y agoProbably no since FFT is slower and less parallelizable than products.
- anoy8888 5y agoI like your article and went to your home page to find more good articles and I like what I saw. Thank you for sharing. The only thing is that I am reading from my phone and the site is not very mobile friendly