3 ms·
Could also be Sirens[1] which was all about sin as activation, while sinusoidal positional encoding are (iirc) just from the original transformer paper. [1]: h
by f_devd 4y ago
Could also be Sirens[1] which was all about sin as activation, while sinusoidal positional encoding are (iirc) just from the original transformer paper.
[1]: https://arxiv.org/abs/2006.09661 https://arxiv.org/abs/2006.09661