4 ms·
It already is.. can’t find the paper, but some image related deep learning model used sin(x) type of inputs to their model. This was a core improvement. Not un
by srcreigh 4y ago
It already is.. can’t find the paper, but some image related deep learning model used sin(x) type of inputs to their model. This was a core improvement.
Not unlike how you can linear regress on x1,x1^2,x1^3,x2,x2^2. Encoding prior knowledge about input data as additional inputs helps with machine learning. 1980 called it wants its accomplishments back.
- syrrim 4y agoGpt does this iirc
- srcreigh 4y agoI doubt you’ll find any decent audio or image related models which don’t use sine inputs. It’s ubiquitous. It’s why jpgs and mp3s work.
- leereeves 4y agoI think most transformer models use sin and cos for positional encoding. They were recommended in the original transformer paper, Attention is All You Need. But in that case it's not quite Fourier analysis because those functions aren't applied to the input; they are just constants that are added to the input because transformers are permutation invariant (without some positional encoding). Was there a model that applied sin and cos to the input?
- gessha 4y agoIn a recent-ish paper, there was an analysis of positional encodings using Neural Tangent Kernels which showed positional encodings are very similar to Fourier transformations of the inputs: Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, https://arxiv.org/abs/2006.10739 https://arxiv.org/abs/2006.10739
- gessha 4y agoSin(x)-style encoding is called positional encoding and it’s used in neural radiance fields (NeRFs) and also in transformer-based models.
- f_devd 4y agoCould also be Sirens[1] which was all about sin as activation, while sinusoidal positional encoding are (iirc) just from the original transformer paper. [1]: https://arxiv.org/abs/2006.09661 https://arxiv.org/abs/2006.09661