5 ms·

>> The paper shows that their model first overfitted the data. By overfitting I mean 100% train dataset accuracy and ~0% validation dataset accuracy. The model

>> The paper shows that their model first overfitted the data. By overfitting I mean 100% train dataset accuracy and ~0% validation dataset accuracy. The model never gets any feedback from the validation dataset trough the training procedure. Yes, but the researchers get plenty of feedback from the validation set and there's nothing easier for them than to tweak their system to perform well on the validation set. That's overfitting on the validation set by proxy. It's absolutely inevitable when the validation set is visible to the researchers and it's very difficult to guard against because of course a team who has spent maybe a month or two working on a system with a publication deadline looming are not going to just give up on their work once they figure it it doesn't work very well. They're going to tweak it and tweak it and tweak it, until it does what they want it to. They're going to converge -they are going to converge- on some ideal set of hyperparameters that optimises their system's performance on its validation set (or the test set, it doesn't matter what it's called, it matters that it is visible to the authors). They will even find a region of the weight space where it's best to initialise their system to get it to perform well on the validation set. And, of course, if they can't find a way to get good performance out of their system, you and I will never hear about it because nobody ever publishes negative results. So there are very strong confirmation and survivorship biases at play and it's not surprising to see, like you say, that the system keeps doing better. And that suffices to explain its performance, without the need for any mysterious post-overfitting grokking ability. But maybe I haven't read the paper that carefully and they do guard against this sort of overfitting-by-proxy? Have you found something like that in the paper? If so, sorry for missing it myself.

Isinlor 5y ago
> And that suffices to explain its performance, without the need for any mysterious post-overfitting grokking ability. It actually still does not suffice. It is just not expected no matter what the authors would be doing. Just the fact that they managed to get that effect is interesting. Granted, the phenomenon may be limited in scope. For example, on ImageNet it may require ridiculously long time scales. But maybe there is some underlying reason we can exploit to get to grokking faster. It's basically all in fig 2.: - they use 3 random seeds per result - they show results for 12 different simple algorithmic datasets - they evaluate 12 different combinations of hyperparameters - for each hyperparameters combination they use 10+ different ratios of train to validation splits So they do some 10*12*3*2 = 720 runs. They conclude that hyperparameters are important. Seems like weight decay is especially important for the grokking phenomenon to happen when model has access to low ratio of training data. Also, at least 2 other people managed to replicate that results: https://twitter.com/sea_snell/status/1461344037504380931 https://twitter.com/sea_snell/status/1461344037504380931 https://twitter.com/lieberum_t/status/1480779426535288834 https://twitter.com/lieberum_t/status/1480779426535288834 One hypothesis may be that models are just biased to randomly stumble upon wide, flat local minima. And wide, flat local minima generalize well.
- YeGoblynQueenne 5y ago
  I don't agree. Once a researcher has full control of the data, they can use it to prove anything at all. This is especially so in work like the one we discuss, where the experiments are performed on artificial data and the researchers have even more control on it than usual. As they say themselves, such effects are much harder to obtain on real-world data. This hints to the fact that the effect depends on the structure of the dataset and so it's unlikely to, well, generalise to data that cannot be strictly controlled. You are impressed by the fact that one particular, counter-intuitive result was obtained, but of course there is an incentive to publish something that stands out, rather than something less notable. There is a well-known paper by John Ioannidis on cognitive biases in medical research: Why most published research findings are false https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... It's not about machine learning per sé, but its observations can be applied to any field where empirical studies are common, like machine learning. Especially in the field of deep learning where scholarly work tends to be primarily empirical and where understanding the behaviour of systems is impeded by the black-box nature of deep learning models, observing something mysterious and unexpected must be cause for suspicion and scrutiny of methodology, rather than accepted unconditionally as an actual observation. In particular, any hypothesis that tends towards magick, for example suggesting that a change in quantities (data, compute, training time) yields qualitative improvements (prediction transmogrifying into understanding, overfitting transforming into generalisation), should be discarded with extreme prejudice.
  - Isinlor 5y ago
    > In particular, any hypothesis that tends towards magick, for example suggesting that a change in quantities (data, compute, training time) yields qualitative improvements (prediction transmogrifying into understanding, overfitting transforming into generalisation), should be discarded with extreme prejudice. It does not tend towards magic. It does happen and people can replicate it. Melanie Mitchell recently brought back the point by Drew McDermott that AI people tend to use wishful mnemonic. Words like understanding or generalisation can easily be just a wishful mnemonic. I fully agree with that. But the fact remains. A model that has ~100% training accuracy and ~0% validation accuracy on simple but non-trivial dataset is able to reach ~100% training and ~100% validation accuracy. > This hints to the fact that the effect depends on the structure of the dataset and so it's unlikely to, well, generalise to data that cannot be strictly controlled. Indeed, but it is still interesting. It may be that it manifests itself because there is a very simple rule underlying the dataset and the dataset is finite. But it also seems to work under some degree of noise and that's encouraging. For example, the fact that it may help study connection of wide, flat local-minima and generalization is encouraging. > You are impressed by the fact that one particular, counter-intuitive result was obtained I'm impressed by double descent phenomenon as well. And this one shows up all over the place. > There is a well-known paper by John Ioannidis on cognitive biases in medical research: Why most published research findings are false I know about John Ioannidis. I was writing and thinking a lot about replication crisis in science in general. BTW - it's quite a pity that Ioannidis himself started selecting data towards his thesis with regard to COVID-19. > It's not about machine learning per sé, but its observations can be applied to any field where empirical studies are common, like machine learning. Unfortunately, it applies to theoretical findings too. For example, universal approximation theorem, no free lunch theorem or incompleteness theorems, are widely misunderstood. There are also countless less known theoretical results that are similarly misunderstood.

Isinlor 5y ago

> And that suffices to explain its performance, without the need for any mysterious post-overfitting grokking ability. It actually still does not suffice. It is just not expected no matter what the authors would be doing. Just the fact that they managed to get that effect is interesting. Granted, the phenomenon may be limited in scope. For example, on ImageNet it may require ridiculously long time scales. But maybe there is some underlying reason we can exploit to get to grokking faster. It's basically all in fig 2.: - they use 3 random seeds per result - they show results for 12 different simple algorithmic datasets - they evaluate 12 different combinations of hyperparameters - for each hyperparameters combination they use 10+ different ratios of train to validation splits So they do some 10*12*3*2 = 720 runs. They conclude that hyperparameters are important. Seems like weight decay is especially important for the grokking phenomenon to happen when model has access to low ratio of training data. Also, at least 2 other people managed to replicate that results: https://twitter.com/sea_snell/status/1461344037504380931 https://twitter.com/sea_snell/status/1461344037504380931 https://twitter.com/lieberum_t/status/1480779426535288834 https://twitter.com/lieberum_t/status/1480779426535288834 One hypothesis may be that models are just biased to randomly stumble upon wide, flat local minima. And wide, flat local minima generalize well.

YeGoblynQueenne 5y ago
I don't agree. Once a researcher has full control of the data, they can use it to prove anything at all. This is especially so in work like the one we discuss, where the experiments are performed on artificial data and the researchers have even more control on it than usual. As they say themselves, such effects are much harder to obtain on real-world data. This hints to the fact that the effect depends on the structure of the dataset and so it's unlikely to, well, generalise to data that cannot be strictly controlled. You are impressed by the fact that one particular, counter-intuitive result was obtained, but of course there is an incentive to publish something that stands out, rather than something less notable. There is a well-known paper by John Ioannidis on cognitive biases in medical research: Why most published research findings are false https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... It's not about machine learning per sé, but its observations can be applied to any field where empirical studies are common, like machine learning. Especially in the field of deep learning where scholarly work tends to be primarily empirical and where understanding the behaviour of systems is impeded by the black-box nature of deep learning models, observing something mysterious and unexpected must be cause for suspicion and scrutiny of methodology, rather than accepted unconditionally as an actual observation. In particular, any hypothesis that tends towards magick, for example suggesting that a change in quantities (data, compute, training time) yields qualitative improvements (prediction transmogrifying into understanding, overfitting transforming into generalisation), should be discarded with extreme prejudice.
- Isinlor 5y ago
  > In particular, any hypothesis that tends towards magick, for example suggesting that a change in quantities (data, compute, training time) yields qualitative improvements (prediction transmogrifying into understanding, overfitting transforming into generalisation), should be discarded with extreme prejudice. It does not tend towards magic. It does happen and people can replicate it. Melanie Mitchell recently brought back the point by Drew McDermott that AI people tend to use wishful mnemonic. Words like understanding or generalisation can easily be just a wishful mnemonic. I fully agree with that. But the fact remains. A model that has ~100% training accuracy and ~0% validation accuracy on simple but non-trivial dataset is able to reach ~100% training and ~100% validation accuracy. > This hints to the fact that the effect depends on the structure of the dataset and so it's unlikely to, well, generalise to data that cannot be strictly controlled. Indeed, but it is still interesting. It may be that it manifests itself because there is a very simple rule underlying the dataset and the dataset is finite. But it also seems to work under some degree of noise and that's encouraging. For example, the fact that it may help study connection of wide, flat local-minima and generalization is encouraging. > You are impressed by the fact that one particular, counter-intuitive result was obtained I'm impressed by double descent phenomenon as well. And this one shows up all over the place. > There is a well-known paper by John Ioannidis on cognitive biases in medical research: Why most published research findings are false I know about John Ioannidis. I was writing and thinking a lot about replication crisis in science in general. BTW - it's quite a pity that Ioannidis himself started selecting data towards his thesis with regard to COVID-19. > It's not about machine learning per sé, but its observations can be applied to any field where empirical studies are common, like machine learning. Unfortunately, it applies to theoretical findings too. For example, universal approximation theorem, no free lunch theorem or incompleteness theorems, are widely misunderstood. There are also countless less known theoretical results that are similarly misunderstood.