6 ms·

Generate a scribble. For a positive example, take two random crops of the scribble. For a negative example, take crops of two different scribbles. Ask the netwo

Generate a scribble. For a positive example, take two random crops of the scribble. For a negative example, take crops of two different scribbles. Ask the network if they both came from the same scribble. The meat of the paper is that the scribbles can be remarkably shitty and still get you decent pretraining. Compared to alternatives: imagenet: exhaustively annotated, disk-space-consuming, nonfreeforcommercialuse real world photos - huge and has licensing issues rotnet: take random images, rotate them and ask the network whether they're right side up or not - easy, but still takes up disk space 3d renders: render realistic 3d scenes on the fly and use those as input - nvidia has enough of my money already, thx fractaldb: generate fractal images and pretrain by predicting the fractal's parameters. - why bother with fractals if you don't need them

momenti 5y ago
The title is click-bait then because it is not only unit/pixel-level noise, but also "noise" at the gestalt-level (e.g. large geometric shapes). It is basically an experiment how the mid-to-low-level features in NNs generalize from random abstract generated images to real images. This is not a very surprising result, because it is known NNs generalize fairly well and at these levels you mostly only have blobs and edges that do not look much different from artificial ones.
- ninjin 5y ago
  In complete agreement, not only the title but the entire article is cluttered with click-bait/hype wording. Simply refer to the paper instead [1], which is well written and should be accessible to even those with a modest understanding of the area if you stay away from the technical details. From my own scientific viewpoint, I think the finding is very neat and at least moderately surprising. So I did enjoy my quick read of the work and even wonder if the finding will be transferable to my own research area. [1]: https://proceedings.neurips.cc/paper/2021/hash/14f2ebeab937ca128186e7ba876faef9-Abstract.html https://proceedings.neurips.cc/paper/2021/hash/14f2ebeab937c... However, this is now the second time this week I have spotted shoddy writing coming from unite.ai – I am referring to the conspiracy-esque spin from [2] that we saw just a few days ago. Apparently they come from the same author even, that seems to put a great deal of emphasis on their articles reaching the Hacker News front page [3]. I am not sure how I feel about this, I would like to believe that “we” as a community are better than to fall for hype and click bait; I am also very uncomfortable with the idea of there being professional prestige in getting onto the front page. [2]: https://www.unite.ai/a-cartel-of-influential-datasets-are-dominating-machine-learning-research-new-study-suggests https://www.unite.ai/a-cartel-of-influential-datasets-are-do... [3]: https://martinanderson.ai https://martinanderson.ai
- evouga 5y ago
  Thanks, this is the perfect explanation that was missing from the original article.
- YeGoblynQueenne 5y ago
  >> This is not a very surprising result, because it is known NNs generalize fairly well and at these levels you mostly only have blobs and edges that do not look much different from artificial ones. ANNs generalise atrociously badly, hence the need to train with ever bigger data to cover every nook and cranny of the instance space of each class of interest. Incidentally, when people say that ANNs "generalise" they mean many different things, for example that they "genealise on the test set" which is usually only observed when the tests set is known in advance (and so it has been used in tuning hyperparameters and the like) or even, incredibly that they "generalise on the training set" (i.e. the validation partitions in cross-validation). Conversely, there is a glut of novel terminology like "out-of-sample" or "out-of-distribution" to describe generalisation beyond the test set, but this kind of generalisation is typically held up as a weakness of ANN, because they're genreally really bad at it. In any case, strong evidence of robust generalisation on out-of-sample data from few examples and with no or little pre-training, in ANNs, would be a surprise, indeed.
  - urthor 5y ago
    Exactly this. The difference between interpolation and extrapolation is almost the most important concept in all of machine learning practice. It's infinitesimally rare (from what I've seen so far) that a practical machine learning model can perform high quality extrapolation, for many different metrics of quality. There's almost always far, far too many confounding variable.
    - Isinlor 5y ago
      Depends on how you define interpolation and extrapolation. In high dimensional spaces basically everything is extrapolation including in pixel space and embedding space. > The notion of interpolation and extrapolation is fundamental in various fields from deep learning to function approximation. Interpolation occurs for a sample x whenever this sample falls inside or on the boundary of the given dataset's convex hull. Extrapolation occurs when x falls outside of that convex hull. One fundamental (mis)conception is that state-of-the-art algorithms work so well because of their ability to correctly interpolate training data. A second (mis)conception is that interpolation happens throughout tasks and datasets, in fact, many intuitions and theories rely on that assumption. We empirically and theoretically argue against those two points and demonstrate that on any high-dimensional (>100) dataset, interpolation almost surely never happens. Those results challenge the validity of our current interpolation/extrapolation definition as an indicator of generalization performances. https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485 Also: > The location of decision boundaries inside the convex hull of training set can be investigated in relation to the training samples. However, our analysis shows that in standard image classification datasets, all testing images are considerably outside that convex hull, in the pixel space, in the wavelet space, and in the internal representations learned by deep networks. Therefore, the performance of a trained model partially depends on how its decision boundaries are extended outside the convex hull of its training data. https://arxiv.org/abs/2101.09849 https://arxiv.org/abs/2101.09849
- wwilim 5y ago
  The second paragraph of your comment should have been at the top of the linked article. Thank you for the explanation!
fennecfoxen 5y ago
> nvidia has enough of my money already, thx Wednesday was RSU vesting day. Your purchase is appreciated. <3

momenti 5y ago

The title is click-bait then because it is not only unit/pixel-level noise, but also "noise" at the gestalt-level (e.g. large geometric shapes). It is basically an experiment how the mid-to-low-level features in NNs generalize from random abstract generated images to real images. This is not a very surprising result, because it is known NNs generalize fairly well and at these levels you mostly only have blobs and edges that do not look much different from artificial ones.

ninjin 5y ago
In complete agreement, not only the title but the entire article is cluttered with click-bait/hype wording. Simply refer to the paper instead [1], which is well written and should be accessible to even those with a modest understanding of the area if you stay away from the technical details. From my own scientific viewpoint, I think the finding is very neat and at least moderately surprising. So I did enjoy my quick read of the work and even wonder if the finding will be transferable to my own research area. [1]: https://proceedings.neurips.cc/paper/2021/hash/14f2ebeab937ca128186e7ba876faef9-Abstract.html https://proceedings.neurips.cc/paper/2021/hash/14f2ebeab937c... However, this is now the second time this week I have spotted shoddy writing coming from unite.ai – I am referring to the conspiracy-esque spin from [2] that we saw just a few days ago. Apparently they come from the same author even, that seems to put a great deal of emphasis on their articles reaching the Hacker News front page [3]. I am not sure how I feel about this, I would like to believe that “we” as a community are better than to fall for hype and click bait; I am also very uncomfortable with the idea of there being professional prestige in getting onto the front page. [2]: https://www.unite.ai/a-cartel-of-influential-datasets-are-dominating-machine-learning-research-new-study-suggests https://www.unite.ai/a-cartel-of-influential-datasets-are-do... [3]: https://martinanderson.ai https://martinanderson.ai
evouga 5y ago
Thanks, this is the perfect explanation that was missing from the original article.
YeGoblynQueenne 5y ago
>> This is not a very surprising result, because it is known NNs generalize fairly well and at these levels you mostly only have blobs and edges that do not look much different from artificial ones. ANNs generalise atrociously badly, hence the need to train with ever bigger data to cover every nook and cranny of the instance space of each class of interest. Incidentally, when people say that ANNs "generalise" they mean many different things, for example that they "genealise on the test set" which is usually only observed when the tests set is known in advance (and so it has been used in tuning hyperparameters and the like) or even, incredibly that they "generalise on the training set" (i.e. the validation partitions in cross-validation). Conversely, there is a glut of novel terminology like "out-of-sample" or "out-of-distribution" to describe generalisation beyond the test set, but this kind of generalisation is typically held up as a weakness of ANN, because they're genreally really bad at it. In any case, strong evidence of robust generalisation on out-of-sample data from few examples and with no or little pre-training, in ANNs, would be a surprise, indeed.
- urthor 5y ago
  Exactly this. The difference between interpolation and extrapolation is almost the most important concept in all of machine learning practice. It's infinitesimally rare (from what I've seen so far) that a practical machine learning model can perform high quality extrapolation, for many different metrics of quality. There's almost always far, far too many confounding variable.
  - Isinlor 5y ago
    Depends on how you define interpolation and extrapolation. In high dimensional spaces basically everything is extrapolation including in pixel space and embedding space. > The notion of interpolation and extrapolation is fundamental in various fields from deep learning to function approximation. Interpolation occurs for a sample x whenever this sample falls inside or on the boundary of the given dataset's convex hull. Extrapolation occurs when x falls outside of that convex hull. One fundamental (mis)conception is that state-of-the-art algorithms work so well because of their ability to correctly interpolate training data. A second (mis)conception is that interpolation happens throughout tasks and datasets, in fact, many intuitions and theories rely on that assumption. We empirically and theoretically argue against those two points and demonstrate that on any high-dimensional (>100) dataset, interpolation almost surely never happens. Those results challenge the validity of our current interpolation/extrapolation definition as an indicator of generalization performances. https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485 Also: > The location of decision boundaries inside the convex hull of training set can be investigated in relation to the training samples. However, our analysis shows that in standard image classification datasets, all testing images are considerably outside that convex hull, in the pixel space, in the wavelet space, and in the internal representations learned by deep networks. Therefore, the performance of a trained model partially depends on how its decision boundaries are extended outside the convex hull of its training data. https://arxiv.org/abs/2101.09849 https://arxiv.org/abs/2101.09849
wwilim 5y ago
The second paragraph of your comment should have been at the top of the linked article. Thank you for the explanation!

fennecfoxen 5y ago

> nvidia has enough of my money already, thx Wednesday was RSU vesting day. Your purchase is appreciated. <3