I'm going to reply to your question below here since HN is preventing a reply (anti-flaming/long threads I guess). Be careful with jumping to conclusions: I ne

I'm going to reply to your question below here since HN is preventing a reply (anti-flaming/long threads I guess). Be careful with jumping to conclusions: I never even cited ReLUs or Sigmoids in my post! I don't have any opinion on which non-linearity is better, I only know both are dramatic non-linearities. My claims were about linear circuits. You should use whatever nonlinear element works best in your Neural Network, of course (and I've heard ReLUs have good advantages).