4 ms·
You do something similar in Udacity's SDC nanodegree but you use their simulator rather than a new car. Interestingly through trial and error on my project, ELU
by syntaxing 7y ago
You do something similar in Udacity's SDC nanodegree but you use their simulator rather than a new car. Interestingly through trial and error on my project, ELU activation functions was the only thing that prevent vanishing gradient problems. You use the same activation function and I'm curious why you selected that one. I always wonder why it was the only function that worked for me.
- littlemtman 7y agoBecause you can still find the best gradient with ELU. Say you use sigmoid and you have big weights then your gradient can get stuck
- syntaxing 7y agoThanks for the answer! I also used Relu with no luck either. I figured Relu would work better than ELU but it didn't work for this application. Would you happen to know why ELU is superior to ReLU?
- littlemtman 7y agoRELU is really good for mostly positive weights. ELU is good for both
- czr 7y agoproblem is most likely that your initialization are bad (see e.g. https://openreview.net/pdf?id=H1gsz30cKX https://openreview.net/pdf?id=H1gsz30cKX for explanation). make sure to use variance scaling, taking activation into account (relu cuts variance by half). probably you need to multiply all initial kernel weights by 2. make sure that initial prediction of model before training is same order of magnitude as typical target, not saturated to zero or other extreme value. batchnorm and skip connections can also ease problems of bad initialization, so worth trying.