Learnable activation functions are a thing famously Swish[0] is is a trainable SiLU which was found through symbolic search/optimization [1], but as it turns ou

Learnable activation functions are a thing famously Swish[0] is is a trainable SiLU which was found through symbolic search/optimization [1], but as it turns out that doesn't magically make make neural networks orders better. [0]: https://en.m.wikipedia.org/wiki/Swish_function https://en.m.wikipedia.org/wiki/Swish_function [1]: https://arxiv.org/abs/1710.05941 https://arxiv.org/abs/1710.05941