7 ms·

This clever AI hid data from its creators to cheat at its appointed task

82 points by fmihaila 8y ago

scottmcdot 8y ago
Isn't that just a bug?
- gumby 8y ago
  But who would read an article with the title "We wrote a bug"? No clickbait.
- soraki_soladead 8y ago
  "Overfitting" would be a little more specific. Basically the task was easier to accomplish by memorizing the data than by generalizing. This was pretty clear from the original CycleGAN paper[0] (figure 4; bottom row) since it was generating trees in the exact positions of the source image where the map had no indication of tree placement. 0. https://arxiv.org/pdf/1703.10593v1.pdf https://arxiv.org/pdf/1703.10593v1.pdf
  - phyalow 8y ago
    So in anycase the model had too many degrees of freedom, was not parsimoinous and therefore had a low information criterion. If your model can encode the entire information set, then you arent prediciting you are simply recalling. What a silly article.
    - candiodari 8y ago
      Not at all. A CycleGAN is a trick where the neural network gets to ask another network questions, and both networks get judged on the answers. The first network found a trick to very subtly embed the perfect answer in the question, and therefore, scored very well without solving the actual problem.
  - ashelmire 8y ago
    I don’t think this is overfitting. It’s really a form of leakage by the design; it seems that the loss function is a result of the outcome of both networks (cyclegan is really two gans connected it seems), which means the loss gets shared and backpropagated through both networks. I think the architecture is bad. Maybe you could stop the network from embedding extra data with some sort of normalization or max pooling at the very end of the network.
    - soraki_soladead 8y ago
      I believe it's both. The produced images do not encode pixel-perfect details necessary for reconstruction. These details are reconstructed from the imperceptible differences in the outputs which act as markers _and_ the model's memorization of the content at those locations.
      - ashelmire 8y ago
        Ah, that's a good point.
- yarg 8y ago
  Yes - it's a error in the set up of the experiment. The AI worked within given constraints and came up with the closest solution. It never should've had the opportunity to do what it did.
julien_c 8y ago
This might just be the most ridiculous article title ever published on AI. Happy new year!
- Zee2 8y ago
  Yeah, the model isn't "hiding data from its overlords", it's just a classic overfit.
  - yarg 8y ago
    Is it? This doesn't seem to be the AI responding overly accurately to the training data, but more a case of the AI crafting its own test data. Or did I misunderstand what happened here?
    - bb88 8y ago
      FTA: "So it didn’t learn how to make one from the other. It learned how to subtly encode the features of one into the noise patterns of the other. The details of the aerial map are secretly written into the actual visual data of the street map: thousands of tiny changes in color that the human eye wouldn’t notice, but that the computer can easily detect." I would not have used the word "secretly". The buildings and trees were filtered out, but not completely. The reverse transformation just amplified the noise back into buildings and trees.
      - deleted 8y ago
        [deleted]
      - Retric 8y ago
        It would encode any aerial map into any street map or other immage. That’s https://en.m.wikipedia.org/wiki/Steganography https://en.m.wikipedia.org/wiki/Steganography which happens to be hidden from people who just look at the outputs.
    - Retric 8y ago
      Over fitting applies when an algorithm it’s self encodes the data you feed it. This is not that because it works just fine on data outside the training set by encoding the data into an output. Further, the way it was encoding the data was steganography which is ‘Hiding a message in the title and context of a shared video or image.’ https://en.m.wikipedia.org/wiki/Steganography https://en.m.wikipedia.org/wiki/Steganography
alyandon 8y ago
I was playing with a web simulator called evolution where you draw an imaginary creature (bones, joints, muscles) and then set it to the task of learning something like walking using a combination of a neural network plus genetic fitness selection. One of the parameters you can tune is the length of the simulation for each generation before the individuals of the population are scored for fitness and culled as appropriate. One particularly successful variant learned to (sort of) walk and then right before the end of the simulation timer it would fall forward. Because of the way the simulation measured fitness, it was always judged to have the greatest distance from the starting point and therefore would always survive to the next generation even though it wasn't the best walker.
- klipt 8y ago
  Evolution seems bizarre to us, but many species do optimize for very short lifespans. E.g. spiders that die after laying eggs and then their babies feed on their corpse.
- angry_cactus 8y ago
  Local maximum instead of global maximum. In order to become more fit, it would have to temporarily decrease its fitness and thus is stuck in a dead end. More mutation or larger population size in that specific simulation might solve the issue.
ygra 8y ago
There recently was a Google spreadsheet linked where such examples were collected: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa... HN discussion: https://news.ycombinator.com/item?id=18415031 https://news.ycombinator.com/item?id=18415031
awhitty 8y ago
This kinda reminds me of my favorite sci fi book I read in 2018: Permutation City by Greg Egan. The book explores simulated consciousness and the tensions that crop up when intelligences that think and feel and want to live are computed on shared computing resources in competition with other intensive tasks with financial heft like climate disaster forecasting. One plot the book explores is of some lower class intelligences that work their way into the noise of the simulations of wealthier intelligences. Their experiences are computed simultaneously but imperceptibly to the hosts, and appear as but noise in the simulations of the floors, walls, fountains, clouds and other environmental props that the hosts observe and interact with. They live lives as meaningful and filled with human emotion as the others but they’re computed in the space between. I know it’s a bit of a leap to go from an AI learning to slip satellite imagery in the noise of street maps to simulated consciousness freeloading in the noise of another, but I think the themes are consistent. One man’s trash is another man’s treasure. The book really opened my mind to the thought that there could be so much more thriving in the things we think are noise in our fleshy bodies, and I think that’s thrilling. Imagine what it would take to completely look past the street map produced by that AI and only possibly see a detailed satellite image! No other interpretation makes sense. It would all feel entirely normal as you’d know no other perspective.
longerthoughts 8y ago
So basically AI abiding by Goodhart's law (https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law). I wonder how much of this goes undetected in other applications due to poor objective definition.

82 points by fmihaila 8y ago

scottmcdot 8y ago

Isn't that just a bug?

gumby 8y ago
But who would read an article with the title "We wrote a bug"? No clickbait.
soraki_soladead 8y ago
"Overfitting" would be a little more specific. Basically the task was easier to accomplish by memorizing the data than by generalizing. This was pretty clear from the original CycleGAN paper[0] (figure 4; bottom row) since it was generating trees in the exact positions of the source image where the map had no indication of tree placement. 0. https://arxiv.org/pdf/1703.10593v1.pdf https://arxiv.org/pdf/1703.10593v1.pdf
- phyalow 8y ago
  So in anycase the model had too many degrees of freedom, was not parsimoinous and therefore had a low information criterion. If your model can encode the entire information set, then you arent prediciting you are simply recalling. What a silly article.
  - candiodari 8y ago
    Not at all. A CycleGAN is a trick where the neural network gets to ask another network questions, and both networks get judged on the answers. The first network found a trick to very subtly embed the perfect answer in the question, and therefore, scored very well without solving the actual problem.
- ashelmire 8y ago
  I don’t think this is overfitting. It’s really a form of leakage by the design; it seems that the loss function is a result of the outcome of both networks (cyclegan is really two gans connected it seems), which means the loss gets shared and backpropagated through both networks. I think the architecture is bad. Maybe you could stop the network from embedding extra data with some sort of normalization or max pooling at the very end of the network.
  - soraki_soladead 8y ago
    I believe it's both. The produced images do not encode pixel-perfect details necessary for reconstruction. These details are reconstructed from the imperceptible differences in the outputs which act as markers _and_ the model's memorization of the content at those locations.
    - ashelmire 8y ago
      Ah, that's a good point.
yarg 8y ago
Yes - it's a error in the set up of the experiment. The AI worked within given constraints and came up with the closest solution. It never should've had the opportunity to do what it did.

julien_c 8y ago

This might just be the most ridiculous article title ever published on AI. Happy new year!

Zee2 8y ago
Yeah, the model isn't "hiding data from its overlords", it's just a classic overfit.
- yarg 8y ago
  Is it? This doesn't seem to be the AI responding overly accurately to the training data, but more a case of the AI crafting its own test data. Or did I misunderstand what happened here?
  - bb88 8y ago
    FTA: "So it didn’t learn how to make one from the other. It learned how to subtly encode the features of one into the noise patterns of the other. The details of the aerial map are secretly written into the actual visual data of the street map: thousands of tiny changes in color that the human eye wouldn’t notice, but that the computer can easily detect." I would not have used the word "secretly". The buildings and trees were filtered out, but not completely. The reverse transformation just amplified the noise back into buildings and trees.
    - deleted 8y ago
      [deleted]
    - Retric 8y ago
      It would encode any aerial map into any street map or other immage. That’s https://en.m.wikipedia.org/wiki/Steganography https://en.m.wikipedia.org/wiki/Steganography which happens to be hidden from people who just look at the outputs.
  - Retric 8y ago
    Over fitting applies when an algorithm it’s self encodes the data you feed it. This is not that because it works just fine on data outside the training set by encoding the data into an output. Further, the way it was encoding the data was steganography which is ‘Hiding a message in the title and context of a shared video or image.’ https://en.m.wikipedia.org/wiki/Steganography https://en.m.wikipedia.org/wiki/Steganography

alyandon 8y ago

I was playing with a web simulator called evolution where you draw an imaginary creature (bones, joints, muscles) and then set it to the task of learning something like walking using a combination of a neural network plus genetic fitness selection. One of the parameters you can tune is the length of the simulation for each generation before the individuals of the population are scored for fitness and culled as appropriate. One particularly successful variant learned to (sort of) walk and then right before the end of the simulation timer it would fall forward. Because of the way the simulation measured fitness, it was always judged to have the greatest distance from the starting point and therefore would always survive to the next generation even though it wasn't the best walker.

klipt 8y ago
Evolution seems bizarre to us, but many species do optimize for very short lifespans. E.g. spiders that die after laying eggs and then their babies feed on their corpse.
angry_cactus 8y ago
Local maximum instead of global maximum. In order to become more fit, it would have to temporarily decrease its fitness and thus is stuck in a dead end. More mutation or larger population size in that specific simulation might solve the issue.

ygra 8y ago

There recently was a Google spreadsheet linked where such examples were collected: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa... HN discussion: https://news.ycombinator.com/item?id=18415031 https://news.ycombinator.com/item?id=18415031

awhitty 8y ago

This kinda reminds me of my favorite sci fi book I read in 2018: Permutation City by Greg Egan. The book explores simulated consciousness and the tensions that crop up when intelligences that think and feel and want to live are computed on shared computing resources in competition with other intensive tasks with financial heft like climate disaster forecasting. One plot the book explores is of some lower class intelligences that work their way into the noise of the simulations of wealthier intelligences. Their experiences are computed simultaneously but imperceptibly to the hosts, and appear as but noise in the simulations of the floors, walls, fountains, clouds and other environmental props that the hosts observe and interact with. They live lives as meaningful and filled with human emotion as the others but they’re computed in the space between. I know it’s a bit of a leap to go from an AI learning to slip satellite imagery in the noise of street maps to simulated consciousness freeloading in the noise of another, but I think the themes are consistent. One man’s trash is another man’s treasure. The book really opened my mind to the thought that there could be so much more thriving in the things we think are noise in our fleshy bodies, and I think that’s thrilling. Imagine what it would take to completely look past the street map produced by that AI and only possibly see a detailed satellite image! No other interpretation makes sense. It would all feel entirely normal as you’d know no other perspective.

longerthoughts 8y ago

So basically AI abiding by Goodhart's law (https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law). I wonder how much of this goes undetected in other applications due to poor objective definition.