3 ms·
> >> RMSE for NN approx: 0.00011776185865537907 Error is approximately 0.0001 because `tol`, the parameter that tells optimization to finish, is 0.0001.
by Veedrac 5y ago
> >> RMSE for NN approx: 0.00011776185865537907
Error is approximately 0.0001 because `tol`, the parameter that tells optimization to finish, is 0.0001.
Set tol=0, and then beta_2=1-1e-15, epsilon=1e-30 to maximize stability from the optimizer, and I got RMSE for the neural network to go below 5e-7.
This is all very academic because stochastic gradient descent is a horrific tool to be using for this purpose. You aren't wrong about that.
- credit_guy 5y agoFair enough, I wrote the snippet in 5 min and didn't check the tolerance parameter. But with your improved choice of parameters, the NN is still about 1000 times worse than the cubic spline, despite having 50 times as many nodes.
- Veedrac 5y agoI don't think these numbers are meaningful. It's not far off from a degree-2 interpolation already, and I got a hidden layer size of 20 to an error of 7e-5 by just letting it optimise for longer and picking a seed that worked well, which is basically the same error as a degree-1 interpolation that gets 5e-5. Like sure the spline is doing better, but that's not why we care, it's not like there's a general sense in which spline interpolations are going to be better than the optimal fit from a larger neural network, they're just a simpler, faster, more numerically stable way of solving simpler problems. An optimiser designed for 1D interpolation of small neural networks, for all I know, might get extremely accurate results.