2 ms·
Question about "We give a polynomial-time algorithm for learning neural networks with one hidden layer of sigmoids feeding into any smooth, monotone activation
by _0ffh 9y ago
Question about "We give a polynomial-time algorithm for learning neural networks with one hidden layer of sigmoids feeding into any smooth, monotone activation function (e.g., sigmoid or ReLU)":
ReLU is piecewise linear, I always thought that was non-smooth. Did I get that wrong?
- isoprophlex 9y agoI think the discontinuity in the first derivative is smoothed or interpolated somehow, and that this also happens in existing nn approaches where a gradient must be computed... I could be wrong though