3 ms·

Isn't that just a bug?

gumby 8y ago
But who would read an article with the title "We wrote a bug"? No clickbait.
soraki_soladead 8y ago
"Overfitting" would be a little more specific. Basically the task was easier to accomplish by memorizing the data than by generalizing. This was pretty clear from the original CycleGAN paper[0] (figure 4; bottom row) since it was generating trees in the exact positions of the source image where the map had no indication of tree placement. 0. https://arxiv.org/pdf/1703.10593v1.pdf https://arxiv.org/pdf/1703.10593v1.pdf
- phyalow 8y ago
  So in anycase the model had too many degrees of freedom, was not parsimoinous and therefore had a low information criterion. If your model can encode the entire information set, then you arent prediciting you are simply recalling. What a silly article.
  - candiodari 8y ago
    Not at all. A CycleGAN is a trick where the neural network gets to ask another network questions, and both networks get judged on the answers. The first network found a trick to very subtly embed the perfect answer in the question, and therefore, scored very well without solving the actual problem.
- ashelmire 8y ago
  I don’t think this is overfitting. It’s really a form of leakage by the design; it seems that the loss function is a result of the outcome of both networks (cyclegan is really two gans connected it seems), which means the loss gets shared and backpropagated through both networks. I think the architecture is bad. Maybe you could stop the network from embedding extra data with some sort of normalization or max pooling at the very end of the network.
  - soraki_soladead 8y ago
    I believe it's both. The produced images do not encode pixel-perfect details necessary for reconstruction. These details are reconstructed from the imperceptible differences in the outputs which act as markers _and_ the model's memorization of the content at those locations.
    - ashelmire 8y ago
      Ah, that's a good point.
yarg 8y ago
Yes - it's a error in the set up of the experiment. The AI worked within given constraints and came up with the closest solution. It never should've had the opportunity to do what it did.

But who would read an article with the title "We wrote a bug"? No clickbait.

"Overfitting" would be a little more specific. Basically the task was easier to accomplish by memorizing the data than by generalizing. This was pretty clear from the original CycleGAN paper[0] (figure 4; bottom row) since it was generating trees in the exact positions of the source image where the map had no indication of tree placement. 0. https://arxiv.org/pdf/1703.10593v1.pdf https://arxiv.org/pdf/1703.10593v1.pdf

phyalow 8y ago
So in anycase the model had too many degrees of freedom, was not parsimoinous and therefore had a low information criterion. If your model can encode the entire information set, then you arent prediciting you are simply recalling. What a silly article.
- candiodari 8y ago
  Not at all. A CycleGAN is a trick where the neural network gets to ask another network questions, and both networks get judged on the answers. The first network found a trick to very subtly embed the perfect answer in the question, and therefore, scored very well without solving the actual problem.
ashelmire 8y ago
I don’t think this is overfitting. It’s really a form of leakage by the design; it seems that the loss function is a result of the outcome of both networks (cyclegan is really two gans connected it seems), which means the loss gets shared and backpropagated through both networks. I think the architecture is bad. Maybe you could stop the network from embedding extra data with some sort of normalization or max pooling at the very end of the network.
- soraki_soladead 8y ago
  I believe it's both. The produced images do not encode pixel-perfect details necessary for reconstruction. These details are reconstructed from the imperceptible differences in the outputs which act as markers _and_ the model's memorization of the content at those locations.
  - ashelmire 8y ago
    Ah, that's a good point.

yarg 8y ago

Yes - it's a error in the set up of the experiment. The AI worked within given constraints and came up with the closest solution. It never should've had the opportunity to do what it did.