4 ms·
It's all relu these days
by andy99 1y ago
It's all relu these days
- seanhunter 1y agoWhen people want a smooth function so they can do calculus they often use something like gelu or the swish function rather than relu. And the swish function involves a sigmoid. https://en.wikipedia.org/wiki/Swish_function https://en.wikipedia.org/wiki/Swish_function
- tripplyons 1y agoThe gated variants of these functions have been dominant for a few years.
- tripplyons 1y agoMost LLMs use GeGLU or SwiGLU.