2 ms·
The paper only gives a representation result, so no method is known yet for constructing the component functions `g` for the discontinuous case. See the last pa
by ubj 3y ago
The paper only gives a representation result, so no method is known yet for constructing the component functions `g` for the discontinuous case. See the last paragraph of the Conclusion.
It may turn out that the correct representation can be constructed using an alternate method than backpropagation. But this is still an open question.
- magicalhippo 3y agoThis isn't my field at all, so this might be silly. I was thinking about some weird activation function like say 0 for x < 0.5 0.5 for x in [0.5, 1] 0.1x + 1 for x > 1 While it is piece-wise differentiable, similar to ReLU, I'm guessing regular backpropagation would have would struggle with it. My probably silly thought was, what if you used a smoothed version for computing the differentials for the backpropagation step, while keeping the discontinuous function for actual evaluation? My thought was this would make backprop sensitive to the step changes in the function, while allowing for discontinuous activation. Of course this example function is just something random without any further thought, so probably not a useful one in actual usage.