4 ms·also there are 'leaking' ReLUs f(x) = a if x<0 else b usually 0 < a << bby singularity2001 9y agoalso there are 'leaking' ReLUs f(x) = a if x<0 else b usually 0 < a << bchillee 9y agoI'm aware. He's using ReLU in this case though.rvarma 9y agoI actually think the idea of using leaky ReLUs is interesting, because it'll still provide a small gradient when x < 0, which perhaps may slightly alleviate the vanishing gradients issue