3 ms·
IIRC the original author of the Lottery Ticket Hypothesis now disavows that idea. One intuitive way of looking at it is like so - let's say that you have a gau
by Scene_Cast2 3mo ago
IIRC the original author of the Lottery Ticket Hypothesis now disavows that idea.
One intuitive way of looking at it is like so - let's say that you have a gaussian-looking plot. You want to fit a gaussian. You have a stupid simple model where you can slide your gaussian left and right.
If your initial starting point happens to be roughly within range, great, your optimizer will take care of it for you and slide it into the correct place. If you're too far, too bad, no meaningful gradient.
Instead, neural nets give you the option to spawn a gaussian anywhere you please. In this case, no sliding is necessary, but it comes at a heavy parametrization cost.
- WithinReason 3mo agoHow is this view inconsistent with the lottery ticket hypothesis?
- getnormality 3mo agoA while ago a lot of the discussion about overparameterization was about explaining "double descent", the observation that test error doesn't descend monotonically and actually hits a local maximum around the point where the model has just enough parameters to interpolate the data. My favorite article about double descent looks at this in terms of splines [1]. If I can try to summarize that article: when you are designing a parametrized model to fit to data, you have a choice. You can either: 1. Avoid overparameterization by design. Manually create or choose a space of functions that has limited degrees of freedom by construction. 2. Accept overparameterization and regularize. The latter tends to be more robust, because of the bitter lesson. It's not practical to manually design an ideal, on-demand, just-right limited-parameter model for every dataset we are presented with. The best way to approach that ideal, it turns out, is really to just let the computer figure it out via regularized optimization over an overparameterized space. Statisticians started moving in favor of overparameterization long before deep learning got off the ground. This trend dates back at least to the machine learning bible, Elements of Statistical Learning (2001). [1] https://mlu-explain.github.io/double-descent/ https://mlu-explain.github.io/double-descent/
- porridgeraisin 3mo agoHi, I work on RL, or as it is known today, "classical" RL. I'm interested in knowing the latest work that explains double descent and in general optimisation behaviour of overparameterized neural networks. Do you have a survey paper or blog post or anything else to recommend?
- schmuhblaster 3mo ago> This trend dates back at least to the machine learning bible, Elements of Statistical Learning (2001). Could you elaborate on this?
- disgruntledphd2 3mo agoI was also confused at first. I think what OP meant is that ESL talks a lot about regularization but doesn't mention over parametrization much. However, most of their example datasets are small, and they use complicated models, so it's sort of implicit. They definitely don't mention double descent though.
- skybrian 3mo ago> It's not practical to manually design an ideal, on-demand, just-right limited-parameter model for every dataset we are presented with. What if that could be automated?
- rbanffy 3mo agoThe process would need to have some knowledge of the desired outcome, much like a human expert would have a hunch of the design decisions to make.
- Scene_Cast2 3mo agoNot necessarily. I'm a proponent of (admittedly not very popular) methodology of "train, do interpretability analysis, adjust model architecture". It's not more popular for a few reasons: 1) you first need to train a full general model anyhow 2) interpretability is nontrivial and not guaranteed 3) once you make the architectural changes, you can't commit to that architecture as you might miss out in the future with more advancements 4) with modern transformers, there is limited amount of architectural "play" happening.