5 ms·

I'm confused. Are they just saying if you throw noise-like images at your hot dog classier and label them 'not hot dog' it improves performance?

TheDudeMan 5y ago
I'm also confused. They seem to be missing a paragraph near the start of the article.
fennecfoxen 5y ago
If I'm reading it right, it seems to be saying that if you throw random noise at your image classifier, it's able to do produce an intermediate artifact that you can train more effectively when you throw hot dogs at it. It won't be quite as good as you get with the best models, but it's strikingly good for random noise. The paper puts it like so: "Although GANs need real images to learn the network parameters, we show in this paper that they introduce a structural prior useful to encode image properties without requiring any training." https://openreview.net/pdf?id=RQUl8gZnN7O https://openreview.net/pdf?id=RQUl8gZnN7O
- samkater 5y ago
  I’m not an expert, but my light reading of the paper makes it seem like it isn’t “random noise” in the colloquial sense, but more like “there are properties to the randomness even though the images look like noise.” The abstract seems to say that, but again, I could be misinterpreting: “Our findings show that it is important for the noise to capture certain structural properties of real data but that good performance can be achieved even with processes that are far from realistic.”
  - jswulff 5y ago
    That is correct -- by "noise", we don't meet pixel noise, but instead a stochastic process (in fact, different processes different properties which we compare in the paper) from which we can sample large amounts of varied training images.
    - Retric 5y ago
      Except your not actually training on noise, your training on procedurally generated images which include random modifications. It’s just the summery is really bad at describing the process.
anu7df 5y ago
No. If I understand correctly, what they are doing is generating "noise" samples with statistics similar to natural images. Then they use unsupervised contrastive learning to create representations of these noise images. This network is then employed in some classification tasks and it does well under a specific training mode (linear classifier training in the final layer only). The details of the evaluation aside, what is being shown is that a network trained to generate representation based on these artificially generated images (noise) can encode a good prior for most vision tasks thereby potentially reducing the need for a very large number of real training images. edited: Typo,clarity
jswulff 5y ago
Hi, author here. To hopefully clarify, our work is in the context of representation learning, which is a bit different from a "standard" classification. For example, to classify a hotdog it might be useful to first generate an intermediate representation of the image (think "cylindrical, brown, meaty thing"). Such a representation can then fairly easily be mapped to the concept "hot dog". These representations can be learned from large image datasets alone (they do not require labels!). In our work we show that you don't even need real images, but that images that are generated from noise processes are enough to train such representations, and that these representations are surprisingly good for classification. Hope this clarifies things a bit, and happy to answer any other questions!
- aaronblohowiak 5y ago
  The part you lost me on is noise processes - what goes in to the noise and how does it help if it is random?
  - deleted 5y ago
    [deleted]
  - Animats 5y ago
    It's not random noise. Look at the images in the paper. Horizontal lines, vertical lines, snakeskin patterns, Minecraft textures. Examples of miscellaneous surface patterns, in other words. Back before deep learning, people used to make recognizers for features like that as a lower level of feature recognition. Now it's expected that features will be derived automatically from real imagery. This is kind of a return to that level. A useful training set might be a big texture library used for game development or animation. Those are easily available.
    - jswulff 5y ago
      That would indeed be an interesting thing to try, use real data, but only in terms of textures - so effects like occlusions, perspective, etc. would not be present. I would expect it to be somewhere in the ballpark of our StyleGAN images, which also look very "textural", but lack these effects that are an result of imaging the 3D world. Interestingly, modelling these effects without realistic textures seems to result in worse performance - this is for example the case for images taken from CLEVR or generated from Minecraft, and both perform worse than the StyleGAN images.
nullc 5y ago
Consider an analog. Say you wanted to design an image compression format in the spirit of jpeg, so you take a bunch of 8x8 blocks of image data and you use a principal component analysis to design an optimal orthogonal transformation to apply to 8x8 blocks to compact their energy into fewer coefficients so you can compress it. But collecting lots of image data is hard so someone comes along and points out that the most important characteristic of your images is that they're appear drawn from an autoregressive process... That is that each pixel is (say) 95% correlated to the pixel 1 away, 95%^2 to the pixel two away, 95^3 three away and so on. So to compute the PCA you only need the covariance matrix anyways, so you just generate an AR0.95 covariance matrix and use the transform derived from that. And you find this works pretty well too. This work is along those likes, they're generating random images with some simple natural-like statistical properties and training the first part of the classifier using them and getting useful results. One interesting promising part is that this line of thinking may result in better network designs or insights that allow skipping training these initial layers: Going back to the block transform example, the next step would be to notice that the PCA of the AR0.95 matrix is the discrete cosign transformation, which is agreeable to an extremely efficient implementation.

TheDudeMan 5y ago

I'm also confused. They seem to be missing a paragraph near the start of the article.

fennecfoxen 5y ago

If I'm reading it right, it seems to be saying that if you throw random noise at your image classifier, it's able to do produce an intermediate artifact that you can train more effectively when you throw hot dogs at it. It won't be quite as good as you get with the best models, but it's strikingly good for random noise. The paper puts it like so: "Although GANs need real images to learn the network parameters, we show in this paper that they introduce a structural prior useful to encode image properties without requiring any training." https://openreview.net/pdf?id=RQUl8gZnN7O https://openreview.net/pdf?id=RQUl8gZnN7O

samkater 5y ago
I’m not an expert, but my light reading of the paper makes it seem like it isn’t “random noise” in the colloquial sense, but more like “there are properties to the randomness even though the images look like noise.” The abstract seems to say that, but again, I could be misinterpreting: “Our findings show that it is important for the noise to capture certain structural properties of real data but that good performance can be achieved even with processes that are far from realistic.”
- jswulff 5y ago
  That is correct -- by "noise", we don't meet pixel noise, but instead a stochastic process (in fact, different processes different properties which we compare in the paper) from which we can sample large amounts of varied training images.
  - Retric 5y ago
    Except your not actually training on noise, your training on procedurally generated images which include random modifications. It’s just the summery is really bad at describing the process.

anu7df 5y ago

No. If I understand correctly, what they are doing is generating "noise" samples with statistics similar to natural images. Then they use unsupervised contrastive learning to create representations of these noise images. This network is then employed in some classification tasks and it does well under a specific training mode (linear classifier training in the final layer only). The details of the evaluation aside, what is being shown is that a network trained to generate representation based on these artificially generated images (noise) can encode a good prior for most vision tasks thereby potentially reducing the need for a very large number of real training images. edited: Typo,clarity

jswulff 5y ago

Hi, author here. To hopefully clarify, our work is in the context of representation learning, which is a bit different from a "standard" classification. For example, to classify a hotdog it might be useful to first generate an intermediate representation of the image (think "cylindrical, brown, meaty thing"). Such a representation can then fairly easily be mapped to the concept "hot dog". These representations can be learned from large image datasets alone (they do not require labels!). In our work we show that you don't even need real images, but that images that are generated from noise processes are enough to train such representations, and that these representations are surprisingly good for classification. Hope this clarifies things a bit, and happy to answer any other questions!

aaronblohowiak 5y ago
The part you lost me on is noise processes - what goes in to the noise and how does it help if it is random?
- deleted 5y ago
  [deleted]
- Animats 5y ago
  It's not random noise. Look at the images in the paper. Horizontal lines, vertical lines, snakeskin patterns, Minecraft textures. Examples of miscellaneous surface patterns, in other words. Back before deep learning, people used to make recognizers for features like that as a lower level of feature recognition. Now it's expected that features will be derived automatically from real imagery. This is kind of a return to that level. A useful training set might be a big texture library used for game development or animation. Those are easily available.
  - jswulff 5y ago
    That would indeed be an interesting thing to try, use real data, but only in terms of textures - so effects like occlusions, perspective, etc. would not be present. I would expect it to be somewhere in the ballpark of our StyleGAN images, which also look very "textural", but lack these effects that are an result of imaging the 3D world. Interestingly, modelling these effects without realistic textures seems to result in worse performance - this is for example the case for images taken from CLEVR or generated from Minecraft, and both perform worse than the StyleGAN images.

nullc 5y ago

Consider an analog. Say you wanted to design an image compression format in the spirit of jpeg, so you take a bunch of 8x8 blocks of image data and you use a principal component analysis to design an optimal orthogonal transformation to apply to 8x8 blocks to compact their energy into fewer coefficients so you can compress it. But collecting lots of image data is hard so someone comes along and points out that the most important characteristic of your images is that they're appear drawn from an autoregressive process... That is that each pixel is (say) 95% correlated to the pixel 1 away, 95%^2 to the pixel two away, 95^3 three away and so on. So to compute the PCA you only need the covariance matrix anyways, so you just generate an AR0.95 covariance matrix and use the transform derived from that. And you find this works pretty well too. This work is along those likes, they're generating random images with some simple natural-like statistical properties and training the first part of the classifier using them and getting useful results. One interesting promising part is that this line of thinking may result in better network designs or insights that allow skipping training these initial layers: Going back to the block transform example, the next step would be to notice that the PCA of the AR0.95 matrix is the discrete cosign transformation, which is agreeable to an extremely efficient implementation.