8 ms·

Myths in Machine Learning Research

princeofwands 8y ago
To use the test set explicitly for evaluation is a deadly sin. When found out, you'd face serious damage to your reputation (Like Baidu did a few years ago). [1] What the decreasing performance results on the remade CIFAR-10 test set shows, is probably more akin to a subtle form of overfitting (Due to these datasets being around for a long time, the good results get published, and the bad results discarded, leading to a feedback loop). [2] It is also possible the original test set was closer in distribution to the train set, than the remade one. The ranks stay too consistent for test set evaluation cheating. I also think the "do not trust saliency maps" is too strongly worded. The authors of that paper used adversarial techniques to change the saliency maps. Not just random noise or slight variation, but carefully crafted noise to attack saliency feature importance maps. > For example, while it would be nice to have a CNN identify a spot on an MRI image as a malignant cancer-causing tumor, these results should not be trusted if they are based on fragile interpretation methods. Interpretation methods are as fragile as the deep learning model itself, which is susceptible to adversarial images too. If you allow for scenario's with adversarial images, not only should you not trust the interpretation methods, but also the predictions themselves, destroying any pragmatic value left. It is hard to imagine a realistic threat scenario where MRI's are altered by an adversary, _before_ they are fed into a CNN. When such a scenario is realistic, all bets are off. It is much like blaming Google Chrome exposing passwords during an evil maid attack (when someone has access to your computer, they can do all sorts of nasty stuff, it is nearly impossible to guard against this). [3] [1] https://www.technologyreview.com/s/538111/why-and-how-baidu-cheated-an-artificial-intelligence-test/ https://www.technologyreview.com/s/538111/why-and-how-baidu-... [2] http://hunch.net/?p=22 http://hunch.net/?p=22 [3] https://www.theguardian.com/technology/2013/aug/07/google-chrome-password-security-flaw https://www.theguardian.com/technology/2013/aug/07/google-ch... EDIT: meta(I liked the article. I do not want to argue it is wrong. It is difficult for me to start a thread without finding the one or two things to nitpick at, or to expand upon a point, but this article was already very resourceful)
- p1esk 8y ago
  To use the test set explicitly for evaluation is a deadly sin I’ve seen tons of papers doing that and getting published, especially on cifar10. Not saying it’s a good practice, just that it’s fairly common.
- abidlabs 8y ago
  As one of the authors of the "Interpretation of Neural Networks is Fragile" paper, I would agree with you. To a certain extent, saliency maps can be perturbed even with random noise, but the more dramatic attacks (and certainly the targeted attacks, in which we move the saliency map from one region of the image to a specified another region of the image) require carefully-crafted adversarial perturbations.
- sgt101 8y ago
  >"It is hard to imagine a realistic threat scenario where MRI's are altered by an adversary, _before_ they are fed into a CNN." what about when people in the hospital who have a patient that they suspect has cancer use the best machine to create that patients scans and tend to push patients that they think are ok to the older less good instrument? Or if they choose to utilise time on the best instrument for children? What about when the MRI's done at night are done by one technician who uses a slightly different process from the technicians who created the MRI data set? At the very least there is a significant risk of systematic error being introduced by these kind of bias, and as you say, it's really hard to guard against this, but if a classifier that I produce is used where this happens and people die... Well, whatever I feel I would be responsible.
max_likelihood 8y ago
Even though I don't know what a Tensor is, I had a suspicion that TensorFlow was really just "MatrixFlow". I felt validated after reading myth 1, but I'm still trying to wrap my head around the difference between Tensors & Matrices. I have a feeling that I missing out on something beautiful, like Fourier Transforms, and when I finally get it a deep smile will spread across my face.
- deleted 8y ago
  [deleted]
- stilley2 8y ago
  This is probably wrong, but I always think of tensor's as n-dimensional generalizations of matrices with units attached. Edit: After some wikipediaing, "bases" might be a better word than "units."
  - ummonk 8y ago
    Well intuitively an n-dimensional generalization of matrices would just be a big multi-dimensional table. But a tensor is different in that you have some number of dimensions which are covariant and some number which are contravariant. Additionally, you've sort of got it backwards. A matrix with units (and a set of basis vectors) attached is one representation of a rank (1, 1) tensor. But it's not really a unique representation of the tensor - you could choose a different set of basis vectors and come up with a different matrix representation of the exact same tensor. The tensor is an entity, while the matrix is a representation of an entity within a given coordinate system.
- artwr 8y ago
  Tensors are in effect a generalization of matrices in higher dimensions. A tensor of dimension 3 is a similar step up to the step from a linear array to a matrix. They arise in all sorts of places though fluids is where I met them first. Operations on tensors are also a bit more challenging than operations on matrices.
- mr_toad 8y ago
  https://math.stackexchange.com/questions/412423/what-are-the-differences-between-a-matrix-and-a-tensor https://math.stackexchange.com/questions/412423/what-are-the...
edmack 8y ago
Nice refreshing list :) I didn't think "Attention > Convolution" was a prevalent myth, given how integral convolutions are to SOTA image classifiers and GANs (if anything, I believe attention is unde-utilised here and due to grow in usage a lot)
- sdenton4 8y ago
  Ugh... This list seems like a mishmash of actual bad practices and active areas of discussion labeled 'myths...' which seems pretty harmful to me.
howlin 8y ago
Myth 8: Machine learning == visual object classification
- khaledh 8y ago
  I was expecting a list of general ML myths, but this list is laser focused on neural networks, and in particular their use in object recognition in images.
adamnemecek 8y ago
What’s up with convolution and attention? Like in general, what’s their relationship.
simonster 8y ago
#3 is actually wrong. The results of Recht et al. do not show that people are performing validation on the test set. If this were true, one would expect a poor correlation between accuracy on the original CIFAR-10 test set and the new test set, whereas the authors observe an extremely high correlation. The results actually indicate that attempting to follow the same dataset collection procedures as the creators of CIFAR-10 results in a dataset that is slightly harder than the original dataset (at least for models trained on the original dataset). The follow-up paper (http://people.csail.mit.edu/ludwigs/papers/imagenet.pdf http://people.csail.mit.edu/ludwigs/papers/imagenet.pdf) makes this point explicitly. The fact that the relative ordering of models is preserved on the new dataset suggests that the creators of the models didn't cheat, or at least didn't cheat enough to invalidate CIFAR-10 test set performance as an evaluation metric.
- Majromax 8y ago
  > The results of Recht et al. do not show that people are performing validation on the test set. As I read #3, the hypothesis doesn't depend on individual researchers acting unethically by validating against the test set. Instead, I read it as an analogy to significance bias in other sciences: machine learning models that don't perform as well on the validation set simply aren't published, so the field as a whole over-fits, as if validation is performed on the test set. In the paper you link, the authors themselves do explicitly note some test-set shenanigans: > But this assumption [that the models are independent of the test set] is undermined by the common practice of tuning model hyperparameters directly on the test set, which introduces dependencies between the model $\ˆf$ and the test set S. In the extreme case, this can be seen as training directly on the test set
  - jph00 8y ago
    The authors made various claims or implications not backed up by their experiment. However, it absolutely does not answer the question that is asked in the title of the paper, and the process they use is incapable of answering that question. If you go back and read the original CIFAR10 paper, you'll see that the process they carefully went through meant that they curated the most suitable images for each category. By definition, what's left over (which is what the Recht et al paper chose from) is less good images, which are of course therefore harder to classify. All the experiment measures is how good they are at matching the distribution of the original dataset. The answer, they discovered, is: not very.
andreilys 8y ago
Why is using a test set for evaluation a deadly sin? As I understand it, you fit() with training, then do parameter tuning with validation and the best parameter tuned model is used on test. Now I'm still a little confused as to why we don't just fit() then do hyperparameter tuning with the test set (best-tuned model wins, no need for test). Why would calling predict() on a model cause it to update its weights and overfit?
- kalenx 8y ago
  It's a deadly sin considering that many (most?) researchers do not treat it as a test set, but as a "validation set #2". Basically, you tune your hyperparameters (up to the random seed!) to fare better on the test set. So, as shown in the cited paper, the results are not generalization results anymore.
- EchoAce 8y ago
  I think the idea is that "calling predict and tuning on a model with the test set" is the "overfitting". It's not actual overfitting like we know in ML; it's as if the researcher is performing "descent" to get the best hyper-parameters. Problem is, if we use the test set to find these hyper-parameters, we'll have no idea how well it does in the real-world/in general. We'd need another set to figure that out - and we're back where we started.
- princeofwands 8y ago
  It is not the ML model that is updated with information, but the predictive modeler herself is updated. She now finds parameters that make the model perform well on that specific test set. This gives you overly optimistic estimates of generalization performance (thus unsound science, and, in business, it is better to report too low performance, than too high, because a policy build on a model that is overfit like this can ruin a company or a life). For smarter approaches to this problem, see the research on reusable holdout sets.
- xamuel 8y ago
  You could easily achieve perfect accuracy on the test set by just hardcoding the entire test set into your "model" and the entire model is "1. See image, 2. Look up image in test set, 3. Read off answer". It would be interesting if someone would see whether they could sneakily (Sokal-style) publish a paper like the following: "We took (popular model X) and augmented it with an additional lexicon of specific lookup data, and the result blows away all the competition. This is deeply profound and implies that built-in lexicons could be the key to true general intelligence!" (When in fact all they did was hard-code the test set or part of the test set into their model.) Then see how many popular presses churn out sensational articles.
AndrewKemendo 8y ago
The author doesn't even try to make a good case that these are "commonly believed to be true." I don't know any serious researcher who would claim any of those and #3 isn't really a myth it's just poor practices. Would have been better titled, "Things to avoid when doing ML research"
dkislyuk 8y ago
I appreciate the authors calling out #2. ImageNet and CIFAR are both "solved" benchmarks, to the extent that state of the art algorithms by now are likely overfitting to the specific dataset details. In particular, objects in ImageNet are nearly always in canonical pose, with no occlusion, few confounding objects, illuminated, and in a semantically-obvious configuration. For some applications this is acceptable but as an industry benchmark ImageNet is not informative (user generated photo distributions are never this clean). Even more damning is the recent BagNet paper (nice summary here: https://blog.evjang.com/2019/02/bagnet.html https://blog.evjang.com/2019/02/bagnet.html), which indicates that ImageNet can likely be solved with no global features (i.e. model doesn't have to learn anything truly abstract, just configurations of textures, shapes, colors). I thought the author of that blog post put it nicely: "As someone who is deeply interested in AGI, I find ImageNet much less interesting now, precisely because it can be solved with models that have little global understanding of images."
- joshvm 8y ago
  This also happened (and is happening?) in stereo imaging research. For years, Middlebury was what you tested on, and for years that's what got you published. Nowadays Middlebury is viewed as solved by the top algorithms. If you try those algorithms on your own data, good luck getting similar performance; at least I've not seen any kind of advantage in using anything other than SGM (outside of specific research contexts like my PhD). I'm more concerned that everyone is using KITTI as a (often the only) benchmark for deep-learning based stereo matchers, since those are all images of roads. At least with classical stereo you have some idea what* your cost function is. The other one people are increasingly using is Scene Flow, which is (entirely?) synthetic. Not a great situation. * KITTI is a widely used dataset of driving imagery
tanilama 8y ago
> Myth 3: Machine Learning researchers do not use the test set for validation Damn this hits the nails. People are essentially using 'test' set as validation set, the validation set as the early stopping helper set.
mcilai 8y ago
Number 6 is not a myth!!!
- acdha 8y ago
  Can you explain why?
gok 8y ago
4-7 of these aren't so much "myths" are "ideas that one or two recent papers has cast some doubt upon." Not to say that these won't turn out to be more thoroughly debunked in the future, but it hasn't happened yet. Browsing through papers from a few years ago some of these myths in 2011 might have been: 1. You need to pre-train large networks so that they converge 2. You need GPUs to train deep networks efficiently #1 did indeed turn out to be false; pre-training has pretty convincingly been debunked. But if anything #2 turned out very right; there's no serious training happening today on CPUs. A bit of conjecture here but I suspect word embeddings are going to turn to be the next big thing that turns out not to be all that useful.
- screye 8y ago
  Has #1 really been proven to be false ? Transfer learning using the first few layers pertained on imagenet or a related task have consistently given 1-2% improvements in scores... As recently as mid 2018. This is especially for complex tasks like VQA
  - gok 8y ago
    Transfer learning (or student-teacher training) isn't really the same thing as pre-training as it was being talked about 10 years ago. And in those days the claim wasn't that it helped a bit, but that it made training many deeper networks possible at all.
deleted 8y ago
[deleted]
banachtarski 8y ago
People reading point #3 as saying that researchers are intentionally cheating and using test data for validation need to increase their reading comprehension skills.
kevintb 8y ago
Excellent post! I found #3 pretty informative (and funny, in a sad way) in particular.
dcbadacd 8y ago
Seeing one of those saliency maps one question arose, why do researchers not rotate and wiggle (crop a tiny amount but thus change the pixel that's at the center of the image) source images?
singularity2001 8y ago
> These tricks make lightweight and dynamic convolutions several orders of magnitude more efficient that standard non-separable convolutions. Is that backed up by any data?

To use the test set explicitly for evaluation is a deadly sin. When found out, you'd face serious damage to your reputation (Like Baidu did a few years ago). [1] What the decreasing performance results on the remade CIFAR-10 test set shows, is probably more akin to a subtle form of overfitting (Due to these datasets being around for a long time, the good results get published, and the bad results discarded, leading to a feedback loop). [2] It is also possible the original test set was closer in distribution to the train set, than the remade one. The ranks stay too consistent for test set evaluation cheating. I also think the "do not trust saliency maps" is too strongly worded. The authors of that paper used adversarial techniques to change the saliency maps. Not just random noise or slight variation, but carefully crafted noise to attack saliency feature importance maps. > For example, while it would be nice to have a CNN identify a spot on an MRI image as a malignant cancer-causing tumor, these results should not be trusted if they are based on fragile interpretation methods. Interpretation methods are as fragile as the deep learning model itself, which is susceptible to adversarial images too. If you allow for scenario's with adversarial images, not only should you not trust the interpretation methods, but also the predictions themselves, destroying any pragmatic value left. It is hard to imagine a realistic threat scenario where MRI's are altered by an adversary, _before_ they are fed into a CNN. When such a scenario is realistic, all bets are off. It is much like blaming Google Chrome exposing passwords during an evil maid attack (when someone has access to your computer, they can do all sorts of nasty stuff, it is nearly impossible to guard against this). [3] [1] https://www.technologyreview.com/s/538111/why-and-how-baidu-cheated-an-artificial-intelligence-test/ https://www.technologyreview.com/s/538111/why-and-how-baidu-... [2] http://hunch.net/?p=22 http://hunch.net/?p=22 [3] https://www.theguardian.com/technology/2013/aug/07/google-chrome-password-security-flaw https://www.theguardian.com/technology/2013/aug/07/google-ch... EDIT: meta(I liked the article. I do not want to argue it is wrong. It is difficult for me to start a thread without finding the one or two things to nitpick at, or to expand upon a point, but this article was already very resourceful)

p1esk 8y ago
To use the test set explicitly for evaluation is a deadly sin I’ve seen tons of papers doing that and getting published, especially on cifar10. Not saying it’s a good practice, just that it’s fairly common.
abidlabs 8y ago
As one of the authors of the "Interpretation of Neural Networks is Fragile" paper, I would agree with you. To a certain extent, saliency maps can be perturbed even with random noise, but the more dramatic attacks (and certainly the targeted attacks, in which we move the saliency map from one region of the image to a specified another region of the image) require carefully-crafted adversarial perturbations.
sgt101 8y ago
>"It is hard to imagine a realistic threat scenario where MRI's are altered by an adversary, _before_ they are fed into a CNN." what about when people in the hospital who have a patient that they suspect has cancer use the best machine to create that patients scans and tend to push patients that they think are ok to the older less good instrument? Or if they choose to utilise time on the best instrument for children? What about when the MRI's done at night are done by one technician who uses a slightly different process from the technicians who created the MRI data set? At the very least there is a significant risk of systematic error being introduced by these kind of bias, and as you say, it's really hard to guard against this, but if a classifier that I produce is used where this happens and people die... Well, whatever I feel I would be responsible.

max_likelihood 8y ago

Even though I don't know what a Tensor is, I had a suspicion that TensorFlow was really just "MatrixFlow". I felt validated after reading myth 1, but I'm still trying to wrap my head around the difference between Tensors & Matrices. I have a feeling that I missing out on something beautiful, like Fourier Transforms, and when I finally get it a deep smile will spread across my face.

deleted 8y ago
[deleted]
stilley2 8y ago
This is probably wrong, but I always think of tensor's as n-dimensional generalizations of matrices with units attached. Edit: After some wikipediaing, "bases" might be a better word than "units."
- ummonk 8y ago
  Well intuitively an n-dimensional generalization of matrices would just be a big multi-dimensional table. But a tensor is different in that you have some number of dimensions which are covariant and some number which are contravariant. Additionally, you've sort of got it backwards. A matrix with units (and a set of basis vectors) attached is one representation of a rank (1, 1) tensor. But it's not really a unique representation of the tensor - you could choose a different set of basis vectors and come up with a different matrix representation of the exact same tensor. The tensor is an entity, while the matrix is a representation of an entity within a given coordinate system.
artwr 8y ago
Tensors are in effect a generalization of matrices in higher dimensions. A tensor of dimension 3 is a similar step up to the step from a linear array to a matrix. They arise in all sorts of places though fluids is where I met them first. Operations on tensors are also a bit more challenging than operations on matrices.
mr_toad 8y ago
https://math.stackexchange.com/questions/412423/what-are-the-differences-between-a-matrix-and-a-tensor https://math.stackexchange.com/questions/412423/what-are-the...

edmack 8y ago

Nice refreshing list :) I didn't think "Attention > Convolution" was a prevalent myth, given how integral convolutions are to SOTA image classifiers and GANs (if anything, I believe attention is unde-utilised here and due to grow in usage a lot)

sdenton4 8y ago
Ugh... This list seems like a mishmash of actual bad practices and active areas of discussion labeled 'myths...' which seems pretty harmful to me.

howlin 8y ago

Myth 8: Machine learning == visual object classification

khaledh 8y ago
I was expecting a list of general ML myths, but this list is laser focused on neural networks, and in particular their use in object recognition in images.

adamnemecek 8y ago

What’s up with convolution and attention? Like in general, what’s their relationship.

simonster 8y ago

#3 is actually wrong. The results of Recht et al. do not show that people are performing validation on the test set. If this were true, one would expect a poor correlation between accuracy on the original CIFAR-10 test set and the new test set, whereas the authors observe an extremely high correlation. The results actually indicate that attempting to follow the same dataset collection procedures as the creators of CIFAR-10 results in a dataset that is slightly harder than the original dataset (at least for models trained on the original dataset). The follow-up paper (http://people.csail.mit.edu/ludwigs/papers/imagenet.pdf http://people.csail.mit.edu/ludwigs/papers/imagenet.pdf) makes this point explicitly. The fact that the relative ordering of models is preserved on the new dataset suggests that the creators of the models didn't cheat, or at least didn't cheat enough to invalidate CIFAR-10 test set performance as an evaluation metric.

Majromax 8y ago
> The results of Recht et al. do not show that people are performing validation on the test set. As I read #3, the hypothesis doesn't depend on individual researchers acting unethically by validating against the test set. Instead, I read it as an analogy to significance bias in other sciences: machine learning models that don't perform as well on the validation set simply aren't published, so the field as a whole over-fits, as if validation is performed on the test set. In the paper you link, the authors themselves do explicitly note some test-set shenanigans: > But this assumption [that the models are independent of the test set] is undermined by the common practice of tuning model hyperparameters directly on the test set, which introduces dependencies between the model $\ˆf$ and the test set S. In the extreme case, this can be seen as training directly on the test set
- jph00 8y ago
  The authors made various claims or implications not backed up by their experiment. However, it absolutely does not answer the question that is asked in the title of the paper, and the process they use is incapable of answering that question. If you go back and read the original CIFAR10 paper, you'll see that the process they carefully went through meant that they curated the most suitable images for each category. By definition, what's left over (which is what the Recht et al paper chose from) is less good images, which are of course therefore harder to classify. All the experiment measures is how good they are at matching the distribution of the original dataset. The answer, they discovered, is: not very.

andreilys 8y ago

Why is using a test set for evaluation a deadly sin? As I understand it, you fit() with training, then do parameter tuning with validation and the best parameter tuned model is used on test. Now I'm still a little confused as to why we don't just fit() then do hyperparameter tuning with the test set (best-tuned model wins, no need for test). Why would calling predict() on a model cause it to update its weights and overfit?

kalenx 8y ago
It's a deadly sin considering that many (most?) researchers do not treat it as a test set, but as a "validation set #2". Basically, you tune your hyperparameters (up to the random seed!) to fare better on the test set. So, as shown in the cited paper, the results are not generalization results anymore.
EchoAce 8y ago
I think the idea is that "calling predict and tuning on a model with the test set" is the "overfitting". It's not actual overfitting like we know in ML; it's as if the researcher is performing "descent" to get the best hyper-parameters. Problem is, if we use the test set to find these hyper-parameters, we'll have no idea how well it does in the real-world/in general. We'd need another set to figure that out - and we're back where we started.
princeofwands 8y ago
It is not the ML model that is updated with information, but the predictive modeler herself is updated. She now finds parameters that make the model perform well on that specific test set. This gives you overly optimistic estimates of generalization performance (thus unsound science, and, in business, it is better to report too low performance, than too high, because a policy build on a model that is overfit like this can ruin a company or a life). For smarter approaches to this problem, see the research on reusable holdout sets.
xamuel 8y ago
You could easily achieve perfect accuracy on the test set by just hardcoding the entire test set into your "model" and the entire model is "1. See image, 2. Look up image in test set, 3. Read off answer". It would be interesting if someone would see whether they could sneakily (Sokal-style) publish a paper like the following: "We took (popular model X) and augmented it with an additional lexicon of specific lookup data, and the result blows away all the competition. This is deeply profound and implies that built-in lexicons could be the key to true general intelligence!" (When in fact all they did was hard-code the test set or part of the test set into their model.) Then see how many popular presses churn out sensational articles.

AndrewKemendo 8y ago

The author doesn't even try to make a good case that these are "commonly believed to be true." I don't know any serious researcher who would claim any of those and #3 isn't really a myth it's just poor practices. Would have been better titled, "Things to avoid when doing ML research"

dkislyuk 8y ago

I appreciate the authors calling out #2. ImageNet and CIFAR are both "solved" benchmarks, to the extent that state of the art algorithms by now are likely overfitting to the specific dataset details. In particular, objects in ImageNet are nearly always in canonical pose, with no occlusion, few confounding objects, illuminated, and in a semantically-obvious configuration. For some applications this is acceptable but as an industry benchmark ImageNet is not informative (user generated photo distributions are never this clean). Even more damning is the recent BagNet paper (nice summary here: https://blog.evjang.com/2019/02/bagnet.html https://blog.evjang.com/2019/02/bagnet.html), which indicates that ImageNet can likely be solved with no global features (i.e. model doesn't have to learn anything truly abstract, just configurations of textures, shapes, colors). I thought the author of that blog post put it nicely: "As someone who is deeply interested in AGI, I find ImageNet much less interesting now, precisely because it can be solved with models that have little global understanding of images."

joshvm 8y ago
This also happened (and is happening?) in stereo imaging research. For years, Middlebury was what you tested on, and for years that's what got you published. Nowadays Middlebury is viewed as solved by the top algorithms. If you try those algorithms on your own data, good luck getting similar performance; at least I've not seen any kind of advantage in using anything other than SGM (outside of specific research contexts like my PhD). I'm more concerned that everyone is using KITTI as a (often the only) benchmark for deep-learning based stereo matchers, since those are all images of roads. At least with classical stereo you have some idea what* your cost function is. The other one people are increasingly using is Scene Flow, which is (entirely?) synthetic. Not a great situation. * KITTI is a widely used dataset of driving imagery

tanilama 8y ago

> Myth 3: Machine Learning researchers do not use the test set for validation Damn this hits the nails. People are essentially using 'test' set as validation set, the validation set as the early stopping helper set.

mcilai 8y ago

Number 6 is not a myth!!!

acdha 8y ago
Can you explain why?

gok 8y ago

4-7 of these aren't so much "myths" are "ideas that one or two recent papers has cast some doubt upon." Not to say that these won't turn out to be more thoroughly debunked in the future, but it hasn't happened yet. Browsing through papers from a few years ago some of these myths in 2011 might have been: 1. You need to pre-train large networks so that they converge 2. You need GPUs to train deep networks efficiently #1 did indeed turn out to be false; pre-training has pretty convincingly been debunked. But if anything #2 turned out very right; there's no serious training happening today on CPUs. A bit of conjecture here but I suspect word embeddings are going to turn to be the next big thing that turns out not to be all that useful.

screye 8y ago
Has #1 really been proven to be false ? Transfer learning using the first few layers pertained on imagenet or a related task have consistently given 1-2% improvements in scores... As recently as mid 2018. This is especially for complex tasks like VQA
- gok 8y ago
  Transfer learning (or student-teacher training) isn't really the same thing as pre-training as it was being talked about 10 years ago. And in those days the claim wasn't that it helped a bit, but that it made training many deeper networks possible at all.

deleted 8y ago

[deleted]

banachtarski 8y ago

People reading point #3 as saying that researchers are intentionally cheating and using test data for validation need to increase their reading comprehension skills.

kevintb 8y ago

Excellent post! I found #3 pretty informative (and funny, in a sad way) in particular.

dcbadacd 8y ago

Seeing one of those saliency maps one question arose, why do researchers not rotate and wiggle (crop a tiny amount but thus change the pixel that's at the center of the image) source images?

singularity2001 8y ago

> These tricks make lightweight and dynamic convolutions several orders of magnitude more efficient that standard non-separable convolutions. Is that backed up by any data?