4 ms·
My main question is whether this is still helpful in an SGD context? I.e. if you have $f = \sum_i^n f_i$ and with every batch you make $f_S = \sum_{j\in S} f_j
by abeppu 4y ago
My main question is whether this is still helpful in an SGD context?
I.e. if you have $f = \sum_i^n f_i$ and with every batch you make $f_S = \sum_{j\in S} f_j$ for that batch ... being able to take a step which definitely causes $f_S$ to decrease doesn't necessarily cause $f$ to decrease / the window for $f_S$ may be different than the window for $f$?
If it only really makes sense for cases where you can evaluate the whole function of interest at every step, which tend to be lower-dimensional anyways, then I think an interesting comparison would be to quasi-newton methods like LBGFS. I.e. if you're involving the taylor-series machinery, when should you use it to pick the direction to step (normal QN) vs when should you use it to pick only the step size (this work)?
- nyrikki 4y agoThe newton method has the nasty Wada property arise, which is often glossed over as being the same as scale invariant noise in class but which is better thought of as a topological feature rather than scale invariant noise. It is indecomposable continua and not the imperfect knowledge of initial conditions like chaotic systems often invoke. Boundary conditions becoming indeterminate is a problem with 3 or more attractors or exit basins.
- bollu 4y agoCould you provide a link to some more reading about this? i do not know what a wada property is, nor why it is topological
- nyrikki 4y agoThe Wada property is when you have 3+ disjoint sets that share the same boundary set, at least for the complete Wada property. In Newton's fractal, any circle you draw will either have a single root or all roots, no matter how small you draw that circle. If your initial conditions are near that boundary the exit basin you take is indeterminate because that point wise boundary point is the boundary of multiple exit basins and isn't simply connected. This differs from other scale invariant features like self similar fractal scattering which will have similar 'noise'at different scales but is somewhat deterministic if treated like noise as an example. The Wada property arises in several places like Hamilton systems, delayed partials and predator prey with food and refuge. If you look for papers on light exiting binary black holes there are some good papers. This page may help, but the Wada property is counter intuitive. https://users.math.yale.edu/public_html/People/frame/Fractals/MandelSet/RealNewton/NMWP/NMWP.html https://users.math.yale.edu/public_html/People/frame/Fractal...
- deleted 4y ago[deleted]