The wisdom of that curly activation can be debated, since in every extreme-point (horizontal) the gradients are zero, and all activations will eventually be tra

The wisdom of that curly activation can be debated, since in every extreme-point (horizontal) the gradients are zero, and all activations will eventually be trapped or centered there. Histogram plots can be revealing to understand activations. Very cool graphics.