4 ms·

I don't know much about ML/Deep-Learning and I have a burning question: Say we have 10 images as a starting point. Then we create 10,000 images from those 10 i

I don't know much about ML/Deep-Learning and I have a burning question: Say we have 10 images as a starting point. Then we create 10,000 images from those 10 images by adding noise, filters, flip them, skew them, distort them, etc. Isn't the underlying data the same (or some formal definition of shannon information entropy)? Would that actually improve neural networks? I've always wondered. Is it possible to generate infinite data and get almost perfect neural network accuracy?

colincooke 7y ago
This depends on how well those 10 images are representing the disribution of data for your actual task. With only 10 samples thats highly unlikely. What you are talking about is data augmentation, a strategy we can use to expand our training dataset synthetically, mostly in a bid to prevent over-fitting.
Der_Einzige 7y ago
This is already done. It's called data augmentation and is extremely helpful in computer vision
- fermienrico 7y ago
  How do we generate more "information" from a limited given information? Doesn't that break some law of information theory?
  - claytonjy 7y ago
    it's less "generating more information" and more "presenting the same information in new ways". A more ideal model wouldn't need augmented data, but this is what works well with current architectures. It may be that practical constraints mean we never move away from augmentation, just as we'll never move towards single-layer neural nets, even though theoretically they can fit any model.
  - psb217 7y ago
    With data augmentation, we're effectively injecting additional information about what sorts of transformations of the data the model should be insensitive to. The additional information comes from our (hopefully) well-informed human decisions about how to augment the data. By doing this, we can reduce the tendency for the model to pick up dependencies on patterns that are useful in the context of the (very small) training dataset, but which don't work well on new data that isn't in the training set.
DougBTX 7y ago
> Is it possible to generate infinite data and get almost perfect neural network accuracy? Basic answer is no, but the reason is kind of interesting. Imagine that the input to the model is a list of facts, initially the facts are just: * Image 1 has class A * Image 2 has class B * etc... The idea with data augmentation, in a roundabout way, is to add other facts: * Flipping the image does not change its class * Translating the image does not change its class * Adding a small amount of noise to the image does not change its class * etc... However, it is tricky to express those facts as inputs to the model, but it is easy to generate new images based on those facts which the model should be able to learn from. It would likely be more efficient if those facts could be expressed directly though. So, by generating more data the model can progressively learn those "class invariant transformations", but the model would only reach perfect accuracy if all class invariant transformations were taught. Another way to "teach" a model these rules is to embed the rule into the structure of the model itself, eg the idea behind convolutional neural networks is to embed translation independence into the model, so that it doesn't need to be taught that from large batches of translated images.
ogrisel 7y ago
Data augmentation will help prevent your model from overfitting a bit but the amount of useful information you get from naively augmented data will reach diminishing returns at some point. Data augmentation alone (e.g. rotations / shift / crops / color perturbations / cutout... of a single photo of an husky dog) will never yield the added information that is contained in new pictures showing subtle variations of the phenomenom your are trying to model (e.g. a new photo of a Dalmatian dog if you have no Dalmatian dogs in your original training set).
6gvONxR4sf7o 7y ago
Your big problem with 10 images is going to be overfitting. By modifying an image and training on that too, you're effectively teaching the network that that sort of modification shouldn't change the label. It learns a kind of invariant. That invariant isn't the same as actually seeing the dog from another angle, but it's better than nothing.
arketyp 7y ago
For a standard convolutional net, the low-entropy formulation of for isntance rotation is not immediately accessible, which makes rotation a viable data augmentation and regularization strategy. Some designs try to account for natural symmetries by incorporating the related transformation as priors in the architecture.
jmalicki 7y ago
This is quite common, it's often called data augmentation. For example, most CNNs aren't invariant to skew, distortions, rotations, or even zoom level. So to train a neural net to recognize both 8x8 pixel birds and 10x10 pixel birds, you need to add images of both zoom levels. Of course, this is a weakness, and there is a lot of research to try to rectify this, like Hinton's capsule networks. For things like adding noise, in some cases it is used as regularization to make the model robust to noise, in others, such as GANs, models are trained to learn the difference between the generatively created images that are higher entropy from the true images, to refine the model. But as the sibling noted, and you mention, the underlying data is somewhat the same, so yes, you do need a lot of diversity... but in practice these techniques can be helpful.
rocauc 7y ago
As others have noted, this is data augmentation, and it's incredibly useful to increase variation in training data to help decrease overfitting. It's not a silver bullet. It won't capture the natural variations that happen in the real world. But new forms of augmentation (like OP) are helping us get closer. For example, MixMatch creates "mosaic" images by combining images across the training set [1]. In object detection, bounding box only augmentations are improving models by introducing variation [2]. And an anecdote: I work on https://roboflow.ai https://roboflow.ai , and we've seen customers make production-ready results from datasets <20 images based on techniques like these. [1] https://arxiv.org/abs/1905.02249 https://arxiv.org/abs/1905.02249 [2] https://arxiv.org/pdf/1906.11172.pdf https://arxiv.org/pdf/1906.11172.pdf
ska 7y ago
Short answer is no, certainly to the "perfect" part. The core problem in ML is generalization; simply put - how well does your approach work with new data it hasn't seen before. Think of it this way: there is a large set of all the potential inputs you could see, and you only get to see small subset when you are training; what do you do so your general performance is best? Which of course you can't actually know but you can try and estimate. There are two issues that can give you a lot of trouble here. The first is overfitting (you'll do much better on the training set than "in real life"), the second is bias in in your training samples. Data augmentation (what you are talking about) is one approach to reduce parts of the former effect, and done correctly it can help. Take a simple example, imagine we were trying to recognize simple geometric shapes on images of a page - you want to find triangles, rectangles, ellipses, etc. I only give you a small set images, say 10s of shapes total. Now you suspect that "in the wild" you can have triangles at all sorts of rotations, and sizes, but I've only given you a few examples. So you want your algorithm to learn the shapes, but not the sizes or orientations. If you just train on these, it may not recognize a triangle that is just 2x as big as any it has seen, or rotated 20 degrees left from one it has seen, etc. One approach would be to try and find a rotation and/or scale invariant representation for your inputs - if you "know" that shouldn't matter you've now removed it from the problem. This can be hard or even mathematically impossible to do, depending on the problem space (e.g. there is no rotation and scale invariant manifold for photographic images). So another way you can approach it is empirically; to take the examples I gave you, and generate new examples in different poses and scales. You feed this into your training and should get a much more robust result, one that doesn't hew too closely to the training set (i.e. less over training). So this sounds great, right? What could go wrong? There are a few issues. One is you are now enforcing things outside what you learn from the data, so if you are wrong you will make things worse. More subtly when you do this you can tend to amplify any of the sampling biases you had originally. Imagine that I never gave you an equilateral triangle in the training set. It's quite plausible that by generating millions of inputs from a few examples, this category gets pushed closer to something symmetric, like a sphere , say. Another issue that can be subtle is that the manipulations you are doing for data augmentation can easily introduce new things to the data that you don't see, and your training can pick that up. Consider, for example, rotations of these shapes. I told you we were doing this from images, i.e. discretely sampled grids. This means that other than certain symmetric rotations and flips, you can't do this without resampling. And you can't resample without smoothing. So if you take a dozen or so "crisp" examples and turn them into 10s of thousands of "smoothed" examples, what exactly are you teaching your model. I'm also waving my hands hear about how you are extracting "shapes" from "background" and in a NN context, what your inputs actually look like... but you can introduce issues here also. There are lots of trade offs here. It's a useful technique, but unsurprisingly isn't a silver bullet.

colincooke 7y ago

This depends on how well those 10 images are representing the disribution of data for your actual task. With only 10 samples thats highly unlikely. What you are talking about is data augmentation, a strategy we can use to expand our training dataset synthetically, mostly in a bid to prevent over-fitting.

Der_Einzige 7y ago

This is already done. It's called data augmentation and is extremely helpful in computer vision

fermienrico 7y ago
How do we generate more "information" from a limited given information? Doesn't that break some law of information theory?
- claytonjy 7y ago
  it's less "generating more information" and more "presenting the same information in new ways". A more ideal model wouldn't need augmented data, but this is what works well with current architectures. It may be that practical constraints mean we never move away from augmentation, just as we'll never move towards single-layer neural nets, even though theoretically they can fit any model.
- psb217 7y ago
  With data augmentation, we're effectively injecting additional information about what sorts of transformations of the data the model should be insensitive to. The additional information comes from our (hopefully) well-informed human decisions about how to augment the data. By doing this, we can reduce the tendency for the model to pick up dependencies on patterns that are useful in the context of the (very small) training dataset, but which don't work well on new data that isn't in the training set.

DougBTX 7y ago

> Is it possible to generate infinite data and get almost perfect neural network accuracy? Basic answer is no, but the reason is kind of interesting. Imagine that the input to the model is a list of facts, initially the facts are just: * Image 1 has class A * Image 2 has class B * etc... The idea with data augmentation, in a roundabout way, is to add other facts: * Flipping the image does not change its class * Translating the image does not change its class * Adding a small amount of noise to the image does not change its class * etc... However, it is tricky to express those facts as inputs to the model, but it is easy to generate new images based on those facts which the model should be able to learn from. It would likely be more efficient if those facts could be expressed directly though. So, by generating more data the model can progressively learn those "class invariant transformations", but the model would only reach perfect accuracy if all class invariant transformations were taught. Another way to "teach" a model these rules is to embed the rule into the structure of the model itself, eg the idea behind convolutional neural networks is to embed translation independence into the model, so that it doesn't need to be taught that from large batches of translated images.

ogrisel 7y ago

Data augmentation will help prevent your model from overfitting a bit but the amount of useful information you get from naively augmented data will reach diminishing returns at some point. Data augmentation alone (e.g. rotations / shift / crops / color perturbations / cutout... of a single photo of an husky dog) will never yield the added information that is contained in new pictures showing subtle variations of the phenomenom your are trying to model (e.g. a new photo of a Dalmatian dog if you have no Dalmatian dogs in your original training set).

6gvONxR4sf7o 7y ago

Your big problem with 10 images is going to be overfitting. By modifying an image and training on that too, you're effectively teaching the network that that sort of modification shouldn't change the label. It learns a kind of invariant. That invariant isn't the same as actually seeing the dog from another angle, but it's better than nothing.

arketyp 7y ago

For a standard convolutional net, the low-entropy formulation of for isntance rotation is not immediately accessible, which makes rotation a viable data augmentation and regularization strategy. Some designs try to account for natural symmetries by incorporating the related transformation as priors in the architecture.

jmalicki 7y ago

This is quite common, it's often called data augmentation. For example, most CNNs aren't invariant to skew, distortions, rotations, or even zoom level. So to train a neural net to recognize both 8x8 pixel birds and 10x10 pixel birds, you need to add images of both zoom levels. Of course, this is a weakness, and there is a lot of research to try to rectify this, like Hinton's capsule networks. For things like adding noise, in some cases it is used as regularization to make the model robust to noise, in others, such as GANs, models are trained to learn the difference between the generatively created images that are higher entropy from the true images, to refine the model. But as the sibling noted, and you mention, the underlying data is somewhat the same, so yes, you do need a lot of diversity... but in practice these techniques can be helpful.

rocauc 7y ago

As others have noted, this is data augmentation, and it's incredibly useful to increase variation in training data to help decrease overfitting. It's not a silver bullet. It won't capture the natural variations that happen in the real world. But new forms of augmentation (like OP) are helping us get closer. For example, MixMatch creates "mosaic" images by combining images across the training set [1]. In object detection, bounding box only augmentations are improving models by introducing variation [2]. And an anecdote: I work on https://roboflow.ai https://roboflow.ai , and we've seen customers make production-ready results from datasets <20 images based on techniques like these. [1] https://arxiv.org/abs/1905.02249 https://arxiv.org/abs/1905.02249 [2] https://arxiv.org/pdf/1906.11172.pdf https://arxiv.org/pdf/1906.11172.pdf

ska 7y ago

Short answer is no, certainly to the "perfect" part. The core problem in ML is generalization; simply put - how well does your approach work with new data it hasn't seen before. Think of it this way: there is a large set of all the potential inputs you could see, and you only get to see small subset when you are training; what do you do so your general performance is best? Which of course you can't actually know but you can try and estimate. There are two issues that can give you a lot of trouble here. The first is overfitting (you'll do much better on the training set than "in real life"), the second is bias in in your training samples. Data augmentation (what you are talking about) is one approach to reduce parts of the former effect, and done correctly it can help. Take a simple example, imagine we were trying to recognize simple geometric shapes on images of a page - you want to find triangles, rectangles, ellipses, etc. I only give you a small set images, say 10s of shapes total. Now you suspect that "in the wild" you can have triangles at all sorts of rotations, and sizes, but I've only given you a few examples. So you want your algorithm to learn the shapes, but not the sizes or orientations. If you just train on these, it may not recognize a triangle that is just 2x as big as any it has seen, or rotated 20 degrees left from one it has seen, etc. One approach would be to try and find a rotation and/or scale invariant representation for your inputs - if you "know" that shouldn't matter you've now removed it from the problem. This can be hard or even mathematically impossible to do, depending on the problem space (e.g. there is no rotation and scale invariant manifold for photographic images). So another way you can approach it is empirically; to take the examples I gave you, and generate new examples in different poses and scales. You feed this into your training and should get a much more robust result, one that doesn't hew too closely to the training set (i.e. less over training). So this sounds great, right? What could go wrong? There are a few issues. One is you are now enforcing things outside what you learn from the data, so if you are wrong you will make things worse. More subtly when you do this you can tend to amplify any of the sampling biases you had originally. Imagine that I never gave you an equilateral triangle in the training set. It's quite plausible that by generating millions of inputs from a few examples, this category gets pushed closer to something symmetric, like a sphere , say. Another issue that can be subtle is that the manipulations you are doing for data augmentation can easily introduce new things to the data that you don't see, and your training can pick that up. Consider, for example, rotations of these shapes. I told you we were doing this from images, i.e. discretely sampled grids. This means that other than certain symmetric rotations and flips, you can't do this without resampling. And you can't resample without smoothing. So if you take a dozen or so "crisp" examples and turn them into 10s of thousands of "smoothed" examples, what exactly are you teaching your model. I'm also waving my hands hear about how you are extracting "shapes" from "background" and in a NN context, what your inputs actually look like... but you can introduce issues here also. There are lots of trade offs here. It's a useful technique, but unsurprisingly isn't a silver bullet.