3 ms·
You'd think that boosters for a technology whose very foundations rely on the sigmoid and tanh functions used as neuron activation functions would intuitively g
by Earw0rm 1y ago
You'd think that boosters for a technology whose very foundations rely on the sigmoid and tanh functions used as neuron activation functions would intuitively get this...
- andy99 1y agoIt's all relu these days
- seanhunter 1y agoWhen people want a smooth function so they can do calculus they often use something like gelu or the swish function rather than relu. And the swish function involves a sigmoid. https://en.wikipedia.org/wiki/Swish_function https://en.wikipedia.org/wiki/Swish_function
- tripplyons 1y agoThe gated variants of these functions have been dominant for a few years.
- tripplyons 1y agoMost LLMs use GeGLU or SwiGLU.