2 ms·

Why wouldn't I be able to train negative classifiers ? Note that I do always have "negative" training data available : background noise for one, and I'm sure I

Why wouldn't I be able to train negative classifiers ? Note that I do always have "negative" training data available : background noise for one, and I'm sure I can generate 10 other kinds but I don't think that's even necessary. Backprop moves things away from equilibrium. It can do so in both directions. It can make a network more accurate, or more "inversely" accurate. I guess what you mean is that seeing only positive examples will lead to a network that always answers "yes". And that is indeed a concern. So perhaps I should adjust my answer to hedge for that: If the network trains a good positive feature for elephants, it will be able to train a good negative feature as well.

rahimnathwani 10y ago
"Note that I do always have "negative" training data available" Yes, you do, but the model being trained with only cat pictures doesn't. "If the network trains a good positive feature for elephants, it will be able to train a good negative feature as well." I'm not sure what you mean here. Let's say a positive feature for an elephant is a patch of grey. What's a good negative feature? And how would the model learn that the presence of that feature should make it less likely what it's seeing is an elephant.
- candiodari 10y ago
  Think of a decision boundary that a neural net makes. And let's assume it's a 3 dimensional surface. That surface is going to have peaks going up and peaks going down. Backprop determines the differential with respect to the error. You can use that to lower the error in classification, but there's nothing preventing you from going the other way. You would normally descend into the valleys, but you can just as well climb the mountain, to find features that are extremely unlikely to be found on an elephant, and with more training you can find stuff that's commonly found on non-elephant things, and train those features on those. What I mean is that if it only sees positive examples, it might conclude that anything means it's an elephant, regardless of what exactly it is. So that's why I have to qualify by saying a "good" feature. Furthermore there are ways to generate good negative examples from the only positive input data. For instance, you can use random static as an input. Given that static (correctly generated) does not have any information content, there's nothing in there that's an elephant, and you can train it away from that. Other obvious ways to generate more data is to do reverse training : determine what input data would make the network say that that is absolutely not an elephant, after seeing a few elephants. Assume it gets that right, that it isn't an elephant. Insert some static to generate more data. Take that input, flip it, rotate it, zoom it, add static, change color distribution and train those as non-elephants. Likewise, take the input pictures. There should be other stuff in there than elephants. So figure out which values trigger the network (not all pixels matter for the end result, in fact, most won't). Find the ones that don't matter for all pictures, and collage them together. Train as non-elephants. Do the same in reverse (find elephant pixels, and use those to paste more elephants in other pictures). You need to do this to keep your data balanced : there should be equal number of positive and negative training generated items. I'm not sure how you could get the adversarial network approach to work for this. At this point, you have still not given it any negative example, but there's lots to train on.
  - rahimnathwani 10y ago
    "Find the ones that don't matter for all pictures, and collage them together. Train as non-elephants." You're just creating negative examples manually, and then feeding them to the model.

rahimnathwani 10y ago

"Note that I do always have "negative" training data available" Yes, you do, but the model being trained with only cat pictures doesn't. "If the network trains a good positive feature for elephants, it will be able to train a good negative feature as well." I'm not sure what you mean here. Let's say a positive feature for an elephant is a patch of grey. What's a good negative feature? And how would the model learn that the presence of that feature should make it less likely what it's seeing is an elephant.

candiodari 10y ago
Think of a decision boundary that a neural net makes. And let's assume it's a 3 dimensional surface. That surface is going to have peaks going up and peaks going down. Backprop determines the differential with respect to the error. You can use that to lower the error in classification, but there's nothing preventing you from going the other way. You would normally descend into the valleys, but you can just as well climb the mountain, to find features that are extremely unlikely to be found on an elephant, and with more training you can find stuff that's commonly found on non-elephant things, and train those features on those. What I mean is that if it only sees positive examples, it might conclude that anything means it's an elephant, regardless of what exactly it is. So that's why I have to qualify by saying a "good" feature. Furthermore there are ways to generate good negative examples from the only positive input data. For instance, you can use random static as an input. Given that static (correctly generated) does not have any information content, there's nothing in there that's an elephant, and you can train it away from that. Other obvious ways to generate more data is to do reverse training : determine what input data would make the network say that that is absolutely not an elephant, after seeing a few elephants. Assume it gets that right, that it isn't an elephant. Insert some static to generate more data. Take that input, flip it, rotate it, zoom it, add static, change color distribution and train those as non-elephants. Likewise, take the input pictures. There should be other stuff in there than elephants. So figure out which values trigger the network (not all pixels matter for the end result, in fact, most won't). Find the ones that don't matter for all pictures, and collage them together. Train as non-elephants. Do the same in reverse (find elephant pixels, and use those to paste more elephants in other pictures). You need to do this to keep your data balanced : there should be equal number of positive and negative training generated items. I'm not sure how you could get the adversarial network approach to work for this. At this point, you have still not given it any negative example, but there's lots to train on.
- rahimnathwani 10y ago
  "Find the ones that don't matter for all pictures, and collage them together. Train as non-elephants." You're just creating negative examples manually, and then feeding them to the model.