When people want a smooth function so they can do calculus they often use something like gelu or the swish function rather than relu. And the swish function inv

When people want a smooth function so they can do calculus they often use something like gelu or the swish function rather than relu. And the swish function involves a sigmoid. https://en.wikipedia.org/wiki/Swish_function https://en.wikipedia.org/wiki/Swish_function