3 ms·

What would be a better method for machine learning folks to take? As a sincere curiosity / desire to learn, not meant as a rhetorical implication that I disagre

What would be a better method for machine learning folks to take? As a sincere curiosity / desire to learn, not meant as a rhetorical implication that I disagree. I interpret your criticism to mean that ML folks tend to re-use a test set multiple times without worrying that doing so reduces the meaning of the results. If that's what you mean, then I do agree. Informally, some researchers are aware of this and aim to use a separate validation data set for all parameter tuning, and would like to use a held out test set as few times as possible — ideally just once. But it gets more complicated than that because, for example, different subsets of the data may not really be independent samples from the run-time distribution (example: data points = medical data about patients who lived or died, but only from three hospitals; the model can learn about different success rates per hospital successfully but it would not generalize to other hospitals). In other words, there are a lot of subtle ways in which a held out test set can result in overconfidence, and I always like to learn of better ways to resist that overconfidence.

igorkraw 4y ago
Ben Recht actually has a line of work showing that we aren't over fitting the validation/test set for now (amazingly...). What I mean is, by chasing higher and higher SotA with more and more money and compute, whole fields can go "improving" only for papers like https://arxiv.org/abs/2003.08505 https://arxiv.org/abs/2003.08505 or "Implementation matters in deep RL" to come out and show that what's going on is different from the literature consensus. The standards for showing improvement are low, while standards for négative résultats are high (I'm a bit biased because I have a rejected paper trying to show empirically some deep RL work didn't add marginal value but I think the case still holds). Everyone involved is trying their best to do good science but unless someone like me asks for it, there simply isn't a value add for your career to do exhaustive checking. A concrete improvement would be only being allowed to change 1 thing at a time per paper, and measure the impact of changing that one thing. But then you couldn't realistically publish anything outside of megacorps. Another solution might be banning corporate papers, or at least making a separate track...from reviewing papers, it seems like single authors or small teams in academia need to compete with Google where multiple teams might share aspects of a project, one doing the architecture, the other a new training algorithm etc...which won't be disclosed, you'll just read a paper where for some reason a novel architecture is introduced using a baseline which is a bit exotic but also used in another paper that came out close to this one, and a regulariser which was introduced just before that ... If you limit the pools, you can put much higher standards on experiments on corporate where you have the budget, while giving academia more points for novelty and creativity
- tylerneylon 4y ago
  Thanks! I hadn't previously thought about the internal boost that large well-sponsored (aka corporate) teams get this way. It seems worth being aware of. I'm certainly in favor of encouraging & including researchers who work in smaller teams or with less funding.

igorkraw 4y ago

Ben Recht actually has a line of work showing that we aren't over fitting the validation/test set for now (amazingly...). What I mean is, by chasing higher and higher SotA with more and more money and compute, whole fields can go "improving" only for papers like https://arxiv.org/abs/2003.08505 https://arxiv.org/abs/2003.08505 or "Implementation matters in deep RL" to come out and show that what's going on is different from the literature consensus. The standards for showing improvement are low, while standards for négative résultats are high (I'm a bit biased because I have a rejected paper trying to show empirically some deep RL work didn't add marginal value but I think the case still holds). Everyone involved is trying their best to do good science but unless someone like me asks for it, there simply isn't a value add for your career to do exhaustive checking. A concrete improvement would be only being allowed to change 1 thing at a time per paper, and measure the impact of changing that one thing. But then you couldn't realistically publish anything outside of megacorps. Another solution might be banning corporate papers, or at least making a separate track...from reviewing papers, it seems like single authors or small teams in academia need to compete with Google where multiple teams might share aspects of a project, one doing the architecture, the other a new training algorithm etc...which won't be disclosed, you'll just read a paper where for some reason a novel architecture is introduced using a baseline which is a bit exotic but also used in another paper that came out close to this one, and a regulariser which was introduced just before that ... If you limit the pools, you can put much higher standards on experiments on corporate where you have the budget, while giving academia more points for novelty and creativity

tylerneylon 4y ago
Thanks! I hadn't previously thought about the internal boost that large well-sponsored (aka corporate) teams get this way. It seems worth being aware of. I'm certainly in favor of encouraging & including researchers who work in smaller teams or with less funding.