3 ms·

Where are these images from? Are there more?

by bla3 6y ago

sillysaurusx 6y ago
Oh, you! I'm so flattered. You're making me blush. Sure, you can have as many as you want. Watch it train in real time: http://song.tensorfork.com:8097/#images http://song.tensorfork.com:8097/#images Though I imagine HN might swamp our little training server running tensorboard, so here you go. https://i.imgur.com/BkGxbo7.png https://i.imgur.com/BkGxbo7.png We've been training a BigGAN-Deep model for 1.5 months now. Though that sounds like a long time, in reality it's completely automatic and I've been leaving it running just to see what will happen. Every single other BigGAN implementation reports that eventually the training run will collapse. We observed the same thing. But gwern came up with a brilliantly simple way to solve this: if D_loss < 0.2: D_loss = 0 It takes some thinking about why this solves collapse. But in short, the discriminator isn't allowed to get too intelligent. Therefore the generator is never forced to degenerate into single examples that happen to fool the discriminator, i.e. collapse. If you like this sort of thing in general, I encourage you to come join the Tensorfork community discord server, which we affectionately named the "TPU podcast": https://discord.com/invite/x52Xz3y https://discord.com/invite/x52Xz3y There are some 1,200 of us now, and people are always showing off stuff like this in our (way too many) channels.
- Jabbles 6y ago
  Do you take advantage of previous iterations of the generator and discriminator? i.e. the generator should be able to fool all previous discriminators, and the discriminator should be able to recognise the work of all previous generators?
  - sillysaurusx 6y ago
    Nope! It's an interesting balance. The truth of the situation seems to be: the generator and discriminator provide a "signal" to each other, like two planets orbiting around each other. If you cut the signal from one, the other will rapidly veer off into infinity, i.e. collapse quickly. Or it will veer off in the other direction, i.e. all progress will stop and the model won't improve. So it's a constant "signal", you see, where one is dependent on the other in the current state. Therefore I am skeptical of attempts to use previous states of discriminators. However! One of the counterintuitive aspects of AI is that the strangest-sounding ideas often have a chance of being good ideas. It's also so hard to try new ideas that you have to pick specific ones. So, roll up your sleeves and implement yours; I would personally be delighted to see what the code would look like for "the current generator can fool all previous discriminators". I really do not mean that in any sort of negative or dismissive way. I really hope that you will come try it, because DL has never been more accessible. And the time is ripe for fresh takes on old ideas; there's a very real chance that you'll stumble across something that works quite well, if you follow your line of thinking. But for practical purposes, the current theory with generators and discriminators is that they react to their current states. So there's not really any way of testing "can the generator fool all previous discriminators?" because in reality, the generator isn't fooling the discriminator at all -- they simply notice when each other deviates by a small amount, and they make a corresponding "small delta change" in response. Kind of like an ongoing chess game.
    - Jabbles 6y ago
      Thanks for the detailed answer. I don't claim it to be a novel idea, I just remember the Alpha Go (zero?) paper that said they played it against older versions to make sure it hadn't got into a bad state.
      - sillysaurusx 6y ago
        Ah! This is an interesting difference, and illustrates one fun aspect of GANs vs other types of models: Alpha Go had a very specific "win condition" that you can measure precisely. (Can the model win the game?) Whereas it's very difficult to quantify what it means to be "better" at generating images, once you get to a certain threshold of realism. (Was Leonardo better than michelangelo? Probably, but it's hard to measure precisely.) The way that Alpha Go worked was, it gathered a bunch of experiences, i.e. it played a bunch of games. Then, after playing tons of games -- tens of dozens! just kidding, probably like 20 million -- it then performed a single gradient update. In other words, you gather your current experiences, and then you react to them. It's a two-phase commit. There's an explicit "gather" step, which you then react to by updating your database of parameters, so to speak. Whereas with GANs, that happens continuously. There's no "gather" step. The generator simply tries to maximize the discrimiantor's loss, and the discriminator tries to minimize it. Balancing the two has been very tricky. But the results speak for themselves.

Where are these images from? Are there more?

by bla3 6y ago

Where are these images from? Are there more?

Oh, you! I'm so flattered. You're making me blush. Sure, you can have as many as you want. Watch it train in real time: http://song.tensorfork.com:8097/#images http://song.tensorfork.com:8097/#images Though I imagine HN might swamp our little training server running tensorboard, so here you go. https://i.imgur.com/BkGxbo7.png https://i.imgur.com/BkGxbo7.png We've been training a BigGAN-Deep model for 1.5 months now. Though that sounds like a long time, in reality it's completely automatic and I've been leaving it running just to see what will happen. Every single other BigGAN implementation reports that eventually the training run will collapse. We observed the same thing. But gwern came up with a brilliantly simple way to solve this: if D_loss < 0.2: D_loss = 0 It takes some thinking about why this solves collapse. But in short, the discriminator isn't allowed to get too intelligent. Therefore the generator is never forced to degenerate into single examples that happen to fool the discriminator, i.e. collapse. If you like this sort of thing in general, I encourage you to come join the Tensorfork community discord server, which we affectionately named the "TPU podcast": https://discord.com/invite/x52Xz3y https://discord.com/invite/x52Xz3y There are some 1,200 of us now, and people are always showing off stuff like this in our (way too many) channels.

Jabbles 6y ago
Do you take advantage of previous iterations of the generator and discriminator? i.e. the generator should be able to fool all previous discriminators, and the discriminator should be able to recognise the work of all previous generators?
- sillysaurusx 6y ago
  Nope! It's an interesting balance. The truth of the situation seems to be: the generator and discriminator provide a "signal" to each other, like two planets orbiting around each other. If you cut the signal from one, the other will rapidly veer off into infinity, i.e. collapse quickly. Or it will veer off in the other direction, i.e. all progress will stop and the model won't improve. So it's a constant "signal", you see, where one is dependent on the other in the current state. Therefore I am skeptical of attempts to use previous states of discriminators. However! One of the counterintuitive aspects of AI is that the strangest-sounding ideas often have a chance of being good ideas. It's also so hard to try new ideas that you have to pick specific ones. So, roll up your sleeves and implement yours; I would personally be delighted to see what the code would look like for "the current generator can fool all previous discriminators". I really do not mean that in any sort of negative or dismissive way. I really hope that you will come try it, because DL has never been more accessible. And the time is ripe for fresh takes on old ideas; there's a very real chance that you'll stumble across something that works quite well, if you follow your line of thinking. But for practical purposes, the current theory with generators and discriminators is that they react to their current states. So there's not really any way of testing "can the generator fool all previous discriminators?" because in reality, the generator isn't fooling the discriminator at all -- they simply notice when each other deviates by a small amount, and they make a corresponding "small delta change" in response. Kind of like an ongoing chess game.
  - Jabbles 6y ago
    Thanks for the detailed answer. I don't claim it to be a novel idea, I just remember the Alpha Go (zero?) paper that said they played it against older versions to make sure it hadn't got into a bad state.
    - sillysaurusx 6y ago
      Ah! This is an interesting difference, and illustrates one fun aspect of GANs vs other types of models: Alpha Go had a very specific "win condition" that you can measure precisely. (Can the model win the game?) Whereas it's very difficult to quantify what it means to be "better" at generating images, once you get to a certain threshold of realism. (Was Leonardo better than michelangelo? Probably, but it's hard to measure precisely.) The way that Alpha Go worked was, it gathered a bunch of experiences, i.e. it played a bunch of games. Then, after playing tons of games -- tens of dozens! just kidding, probably like 20 million -- it then performed a single gradient update. In other words, you gather your current experiences, and then you react to them. It's a two-phase commit. There's an explicit "gather" step, which you then react to by updating your database of parameters, so to speak. Whereas with GANs, that happens continuously. There's no "gather" step. The generator simply tries to maximize the discrimiantor's loss, and the discriminator tries to minimize it. Balancing the two has been very tricky. But the results speak for themselves.