3 ms·
Love to see more papers like this. I remember a recent paper that showed decent model performance when the model was allowed to pick its own activation function
by jdonaldson 8y ago
Love to see more papers like this. I remember a recent paper that showed decent model performance when the model was allowed to pick its own activation functions. It was picking wacky things like sine waves. You'd like to learn something about the model by the way it configures itself, but right now we can only understand simple model features.
- ralusek 8y agoThat's so weird, I can't help but feel like that would only ever result in it coming up with something that initially got it a better result and the rest of the network adjusted around it and is able to carry the weight of that weirdness. What IS interesting is that it shows just how resilient the networks that come out of this really are that they can support such weirdo activations. But there are other elements of introspection that should absolutely be handled by neural nets, things like choosing how deep a given network needs to be should be accomplished by other neural nets specifically looking for trends while training.
- jdonaldson 8y agoYeah, I agree about smarter parameter search and training. That was a big theme at ICLR this year. Fwiw, here's the paper on RNN architecture generation referenced earlier. https://arxiv.org/abs/1712.07316 https://arxiv.org/abs/1712.07316