3 ms·

I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google'

I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google's proprietary 100-million image dataset, the model's accuracy increases to 87%-whatever. That's pretty interesting. It implies that the original accuracy rating might be legit. The concern is that we're chasing the imagenet benchmark as if it's the holy grail, when in fact it's a very narrow slice of what we normally care about as ML researchers. However, the fact that pretraining on JFT increases the accuracy means that the model is generalizing, which is very interesting; it implies that models might be "just that good now." Or more succinctly, if the result was bogus, you'd expect JFT pretraining to have no effect whatsoever (or a negative effect). But it has a positive result. The other thing worth mentioning is that AJMooch seems to have killed batch normalization dead, which is very strange to think about. BN has had a long reign of some ~4 years, but the drawbacks are significant: you have to maintain counters yourself, for example, which was quite annoying. It always seemed like a neural net ought to be able to learn what BN forces you to keep track of. And AJMooch et al seem to prove this is true. I recommend giving evonorm-s a try; it worked perfectly for us the first time, with no loss in generality, and it's basically a copy-paste replacement. (Our BigGAN-Deep model is so good that I doubt you can tell the difference vs the official model. It uses AJMooch's evonorm-s rather than batchnorm: [1] https://i.imgur.com/sfGVbuq.png https://i.imgur.com/sfGVbuq.png [2] https://i.imgur.com/JMJ1Ll0.png https://i.imgur.com/JMJ1Ll0.png and lol at the fake speedometer.)

trhway 6y ago
> pretraining on JFT increases the accuracy means that the model is generalizing not necessarily, it may be mostly a bonus of the transfer, especially considering that JFT is that much larger - getting for example the first conv layers kernels to converge to Gabor-like takes time, yet those layers kernels are very similar across the well trained image nets (and there were some works showing that it is optimal in a sense, and that it is one of the reasons it is in our visual cortex) and thus transferrable, and can practically be treated as fixed in the new model (especially if those layers were pretrained on very large model and reached the state of generic feature extraction). I suspect the similar is applicable for the low level feature aggregating layers too.
bla3 6y ago
Where are these images from? Are there more?
- sillysaurusx 6y ago
  Oh, you! I'm so flattered. You're making me blush. Sure, you can have as many as you want. Watch it train in real time: http://song.tensorfork.com:8097/#images http://song.tensorfork.com:8097/#images Though I imagine HN might swamp our little training server running tensorboard, so here you go. https://i.imgur.com/BkGxbo7.png https://i.imgur.com/BkGxbo7.png We've been training a BigGAN-Deep model for 1.5 months now. Though that sounds like a long time, in reality it's completely automatic and I've been leaving it running just to see what will happen. Every single other BigGAN implementation reports that eventually the training run will collapse. We observed the same thing. But gwern came up with a brilliantly simple way to solve this: if D_loss < 0.2: D_loss = 0 It takes some thinking about why this solves collapse. But in short, the discriminator isn't allowed to get too intelligent. Therefore the generator is never forced to degenerate into single examples that happen to fool the discriminator, i.e. collapse. If you like this sort of thing in general, I encourage you to come join the Tensorfork community discord server, which we affectionately named the "TPU podcast": https://discord.com/invite/x52Xz3y https://discord.com/invite/x52Xz3y There are some 1,200 of us now, and people are always showing off stuff like this in our (way too many) channels.
  - Jabbles 6y ago
    Do you take advantage of previous iterations of the generator and discriminator? i.e. the generator should be able to fool all previous discriminators, and the discriminator should be able to recognise the work of all previous generators?
    - sillysaurusx 6y ago
      Nope! It's an interesting balance. The truth of the situation seems to be: the generator and discriminator provide a "signal" to each other, like two planets orbiting around each other. If you cut the signal from one, the other will rapidly veer off into infinity, i.e. collapse quickly. Or it will veer off in the other direction, i.e. all progress will stop and the model won't improve. So it's a constant "signal", you see, where one is dependent on the other in the current state. Therefore I am skeptical of attempts to use previous states of discriminators. However! One of the counterintuitive aspects of AI is that the strangest-sounding ideas often have a chance of being good ideas. It's also so hard to try new ideas that you have to pick specific ones. So, roll up your sleeves and implement yours; I would personally be delighted to see what the code would look like for "the current generator can fool all previous discriminators". I really do not mean that in any sort of negative or dismissive way. I really hope that you will come try it, because DL has never been more accessible. And the time is ripe for fresh takes on old ideas; there's a very real chance that you'll stumble across something that works quite well, if you follow your line of thinking. But for practical purposes, the current theory with generators and discriminators is that they react to their current states. So there's not really any way of testing "can the generator fool all previous discriminators?" because in reality, the generator isn't fooling the discriminator at all -- they simply notice when each other deviates by a small amount, and they make a corresponding "small delta change" in response. Kind of like an ongoing chess game.
Jabbles 6y ago
What safeguards are there or what assurances do we have that JFT is not contaminated with images from (or extremely similar to) the validation set?
- sillysaurusx 6y ago
  Haha. None whatsoever. The assurance is that everyone in the field seems to take the work seriously. But the reality is that errors creep in from a variety of corners. I would not be even slightly surprised to find that the validation data is substantially similar. We're still at the "bangs rocks together to make fire" phase of ML, which is both exciting and challenging; we're building the future from the ground up. People rarely take the time to look at the actual images, but if you do, you'll notice they have some interesting errors in them: https://twitter.com/theshawwn/status/1262535747975868418 https://twitter.com/theshawwn/status/1262535747975868418 I built an interactive viewer for the tagging site: https://tags.tagpls.com/ https://tags.tagpls.com/ (Someone tagged all 70 shoes in this one, which was kind of impressive... https://tags.shawwn.com/tags/https://battle.shawwn.com/sdc/imagenet/validation/ILSVRC2012_val_00049459.JPEG https://tags.shawwn.com/tags/https://battle.shawwn.com/sdc/i... ) Anyway, some of the validation images happen to be rotated 90 degrees and no one noticed. That made me wonder what other sorts of unexpected errors are in these specific 50,000 validation images that the world just-so-happened to decide were Super Important to the future of AI. The trouble is, images in general are substantially similar to the imagenet validation dataset. In other words, it's tempting to try to think of some way of "dividing up" the data so that there's some sort of validation phase that you can cleanly separate. But reality isn't so kind. When you're at the scale of millions of images, holding out 10% is just a way of sanity checking that your model isn't memorizing the training data; nothing more. Besides, random 90 degree rotations are introduced on purpose now, so it's funny that old mistakes tend not to matter.
- quantumofalpha 6y ago
  Just the sheer size of JFT (latest versions I heard approach 1B images), so it's probably impractical to train on it till overfitting.
gameshot911 6y ago
Those example pictures are trippy! Some of them look like those weird DeepMind sick-fever dream creations. Except I assume they are all real photos, not generated. It's very possible that a trained AI would be better-able to identify some of the candidates than I would. For example, from OP's post, w/ coordinate system starting at lower left, I have no idea what I'm looking at in these examples, except they look organic-ish: [1]: [1,4], [3,2], [4,1] sillysaurusx: I've never seen conglomerate pictures like this used in AI training. Do you train models on these 4x4 images? What's the purpose vs a single picture at a time? Does the model know that you're feeding it 4x4 examples, or does it have to figure that out itself? Aside: Another awesome 'sick-fever dream creation' example if you missed it when it made the rounds on HN is this[3]. Slide the creativity filter up for weirdness! [3] https://thisanimedoesnotexist.ai/ https://thisanimedoesnotexist.ai/
- sillysaurusx 6y ago
  I'm surprised so many people want to see our BigGAN images. Thank you for asking :) You can watch the training process here: http://song.tensorfork.com:8097/#images http://song.tensorfork.com:8097/#images It's been going on for a month and a half, but I leave it running mostly as a fishtank rather than to get to a specific objective. It's fun to load it up and look at a new random image whenever I want. Plus I like the idea of my little TPU being like "look at me! I'm doing work! Here's what I've prepared for you!" so I try to keep my little fella online all the time. - https://i.imgur.com/0O5KZdE.png https://i.imgur.com/0O5KZdE.png - Plus stuff like this makes me laugh really hard. https://i.imgur.com/EnfIBz3.png https://i.imgur.com/EnfIBz3.png - Some nice flowers and a boat. https://i.imgur.com/mrFkIx0.png https://i.imgur.com/mrFkIx0.png The model is getting quite good. I kind of forgot about it over the past few weeks. StyleGAN could never get anywhere close to this level of detail. I had to spend roughly a year tracking down a crucial bug in the implementation that prevented biggan from working very well until now: https://github.com/google/compare_gan/issues/54 https://github.com/google/compare_gan/issues/54 And we also seemed to solve BigGAN collapse, so theoretically the model can improve forever now. I leave it running to see how good it can get. I've never seen conglomerate pictures like this used in AI training. Do you train models on these 4x4 images? What's the purpose vs a single picture at a time? Does the model know that you're feeding it 4x4 examples, or does it have to figure that out itself? Nah, the grid is just for convenient viewing for humans. Robots see one image at a time. (Or more specifically, a batch of images; we happen to use batch size 2 or 4, I forget, so each core sees two images at a time, and then all 8 cores broadcast their gradients to each other and average, so it's really seeing 16 or 32 images at a time.) I feel a bit silly plugging our community so much, but it's really true. If you like tricks like this, join the Tensorfork discord: https://discord.com/invite/x52Xz3y https://discord.com/invite/x52Xz3y My theory when I set it up was that everyone has little tricks like this, but there's no central repository of knowledge / place to ask questions. But now that there are 1,200+ of us, it's become the de facto place to pop in and share random ideas and tricks. For what it's worth, https://thisanimedoesnotexist.ai/ https://thisanimedoesnotexist.ai/ was a joint collaboration of several Tensorfork discord members. :) If you want future updates about this specific BigGAN model, twitter is your best bet: https://twitter.com/search?q=(from%3Atheshawwn)%20biggan&src=typed_query https://twitter.com/search?q=(from%3Atheshawwn)%20biggan&src...
logane 6y ago
Not sure if I follow your JFT argument, but there's a large body of work on both (a) studying whether chasing ImageNet accuracy yields models that generalize well to out of distribution data [1, 2, 3] and (b) contextualizing progress on ImageNet (i.e., what does high accuracy on ImageNet really mean?) [4, 5, 6]. For (a), maybe surprisingly the answer is mostly yes! Better ImageNet accuracy generally corresponds to better out of distribution accuracy. For (b), it turns out that the ImageNet dataset is full of contradictions---many images have multiple ImageNet-relevant objects, and often are ambiguously or mis-labeled, etc---so it's hard to disentangle progress in identifying objects vs. models overfitting to the quirks of the benchmark. [1] ObjectNet: https://objectnet.dev https://objectnet.dev / associated paper [2] ImageNet-v2: https://arxiv.org/abs/1902.10811 https://arxiv.org/abs/1902.10811 [3] An Unbiased Lookat Dataset Bias: https://people.csail.mit.edu/torralba/publications/datasets_cvpr11.pdf https://people.csail.mit.edu/torralba/publications/datasets_... (pre-AlexNet!) [4] From ImageNet to Image Classification: https://arxiv.org/abs/2005.11295 https://arxiv.org/abs/2005.11295 [5] Are we done with ImageNet? https://arxiv.org/abs/2006.07159 https://arxiv.org/abs/2006.07159 [6] Evaluating Machine Accuracy on ImageNet: http://proceedings.mlr.press/v119/shankar20c.html http://proceedings.mlr.press/v119/shankar20c.html
code-scope 6y ago
Is there any (influence) SW framework that takes youtube video as input and split out object/timestamp as output?
indiv0 6y ago
That’s awesome! Is your model available publically? I run a site [0] where users can generate images from text prompts using models like the official BigGAN-Deep one and I’d love to try it out for this purpose. Do you also have somewhere whereupon discuss this stuff? I’m new to ML in general and was wondering if there’s somewhere where y’all experts gather. [0]: https://dank.xyz https://dank.xyz

trhway 6y ago

> pretraining on JFT increases the accuracy means that the model is generalizing not necessarily, it may be mostly a bonus of the transfer, especially considering that JFT is that much larger - getting for example the first conv layers kernels to converge to Gabor-like takes time, yet those layers kernels are very similar across the well trained image nets (and there were some works showing that it is optimal in a sense, and that it is one of the reasons it is in our visual cortex) and thus transferrable, and can practically be treated as fixed in the new model (especially if those layers were pretrained on very large model and reached the state of generic feature extraction). I suspect the similar is applicable for the low level feature aggregating layers too.

bla3 6y ago

Where are these images from? Are there more?

sillysaurusx 6y ago
Oh, you! I'm so flattered. You're making me blush. Sure, you can have as many as you want. Watch it train in real time: http://song.tensorfork.com:8097/#images http://song.tensorfork.com:8097/#images Though I imagine HN might swamp our little training server running tensorboard, so here you go. https://i.imgur.com/BkGxbo7.png https://i.imgur.com/BkGxbo7.png We've been training a BigGAN-Deep model for 1.5 months now. Though that sounds like a long time, in reality it's completely automatic and I've been leaving it running just to see what will happen. Every single other BigGAN implementation reports that eventually the training run will collapse. We observed the same thing. But gwern came up with a brilliantly simple way to solve this: if D_loss < 0.2: D_loss = 0 It takes some thinking about why this solves collapse. But in short, the discriminator isn't allowed to get too intelligent. Therefore the generator is never forced to degenerate into single examples that happen to fool the discriminator, i.e. collapse. If you like this sort of thing in general, I encourage you to come join the Tensorfork community discord server, which we affectionately named the "TPU podcast": https://discord.com/invite/x52Xz3y https://discord.com/invite/x52Xz3y There are some 1,200 of us now, and people are always showing off stuff like this in our (way too many) channels.
- Jabbles 6y ago
  Do you take advantage of previous iterations of the generator and discriminator? i.e. the generator should be able to fool all previous discriminators, and the discriminator should be able to recognise the work of all previous generators?
  - sillysaurusx 6y ago
    Nope! It's an interesting balance. The truth of the situation seems to be: the generator and discriminator provide a "signal" to each other, like two planets orbiting around each other. If you cut the signal from one, the other will rapidly veer off into infinity, i.e. collapse quickly. Or it will veer off in the other direction, i.e. all progress will stop and the model won't improve. So it's a constant "signal", you see, where one is dependent on the other in the current state. Therefore I am skeptical of attempts to use previous states of discriminators. However! One of the counterintuitive aspects of AI is that the strangest-sounding ideas often have a chance of being good ideas. It's also so hard to try new ideas that you have to pick specific ones. So, roll up your sleeves and implement yours; I would personally be delighted to see what the code would look like for "the current generator can fool all previous discriminators". I really do not mean that in any sort of negative or dismissive way. I really hope that you will come try it, because DL has never been more accessible. And the time is ripe for fresh takes on old ideas; there's a very real chance that you'll stumble across something that works quite well, if you follow your line of thinking. But for practical purposes, the current theory with generators and discriminators is that they react to their current states. So there's not really any way of testing "can the generator fool all previous discriminators?" because in reality, the generator isn't fooling the discriminator at all -- they simply notice when each other deviates by a small amount, and they make a corresponding "small delta change" in response. Kind of like an ongoing chess game.

Jabbles 6y ago

What safeguards are there or what assurances do we have that JFT is not contaminated with images from (or extremely similar to) the validation set?

sillysaurusx 6y ago
Haha. None whatsoever. The assurance is that everyone in the field seems to take the work seriously. But the reality is that errors creep in from a variety of corners. I would not be even slightly surprised to find that the validation data is substantially similar. We're still at the "bangs rocks together to make fire" phase of ML, which is both exciting and challenging; we're building the future from the ground up. People rarely take the time to look at the actual images, but if you do, you'll notice they have some interesting errors in them: https://twitter.com/theshawwn/status/1262535747975868418 https://twitter.com/theshawwn/status/1262535747975868418 I built an interactive viewer for the tagging site: https://tags.tagpls.com/ https://tags.tagpls.com/ (Someone tagged all 70 shoes in this one, which was kind of impressive... https://tags.shawwn.com/tags/https://battle.shawwn.com/sdc/imagenet/validation/ILSVRC2012_val_00049459.JPEG https://tags.shawwn.com/tags/https://battle.shawwn.com/sdc/i... ) Anyway, some of the validation images happen to be rotated 90 degrees and no one noticed. That made me wonder what other sorts of unexpected errors are in these specific 50,000 validation images that the world just-so-happened to decide were Super Important to the future of AI. The trouble is, images in general are substantially similar to the imagenet validation dataset. In other words, it's tempting to try to think of some way of "dividing up" the data so that there's some sort of validation phase that you can cleanly separate. But reality isn't so kind. When you're at the scale of millions of images, holding out 10% is just a way of sanity checking that your model isn't memorizing the training data; nothing more. Besides, random 90 degree rotations are introduced on purpose now, so it's funny that old mistakes tend not to matter.
quantumofalpha 6y ago
Just the sheer size of JFT (latest versions I heard approach 1B images), so it's probably impractical to train on it till overfitting.

gameshot911 6y ago

Those example pictures are trippy! Some of them look like those weird DeepMind sick-fever dream creations. Except I assume they are all real photos, not generated. It's very possible that a trained AI would be better-able to identify some of the candidates than I would. For example, from OP's post, w/ coordinate system starting at lower left, I have no idea what I'm looking at in these examples, except they look organic-ish: [1]: [1,4], [3,2], [4,1] sillysaurusx: I've never seen conglomerate pictures like this used in AI training. Do you train models on these 4x4 images? What's the purpose vs a single picture at a time? Does the model know that you're feeding it 4x4 examples, or does it have to figure that out itself? Aside: Another awesome 'sick-fever dream creation' example if you missed it when it made the rounds on HN is this[3]. Slide the creativity filter up for weirdness! [3] https://thisanimedoesnotexist.ai/ https://thisanimedoesnotexist.ai/

sillysaurusx 6y ago
I'm surprised so many people want to see our BigGAN images. Thank you for asking :) You can watch the training process here: http://song.tensorfork.com:8097/#images http://song.tensorfork.com:8097/#images It's been going on for a month and a half, but I leave it running mostly as a fishtank rather than to get to a specific objective. It's fun to load it up and look at a new random image whenever I want. Plus I like the idea of my little TPU being like "look at me! I'm doing work! Here's what I've prepared for you!" so I try to keep my little fella online all the time. - https://i.imgur.com/0O5KZdE.png https://i.imgur.com/0O5KZdE.png - Plus stuff like this makes me laugh really hard. https://i.imgur.com/EnfIBz3.png https://i.imgur.com/EnfIBz3.png - Some nice flowers and a boat. https://i.imgur.com/mrFkIx0.png https://i.imgur.com/mrFkIx0.png The model is getting quite good. I kind of forgot about it over the past few weeks. StyleGAN could never get anywhere close to this level of detail. I had to spend roughly a year tracking down a crucial bug in the implementation that prevented biggan from working very well until now: https://github.com/google/compare_gan/issues/54 https://github.com/google/compare_gan/issues/54 And we also seemed to solve BigGAN collapse, so theoretically the model can improve forever now. I leave it running to see how good it can get. I've never seen conglomerate pictures like this used in AI training. Do you train models on these 4x4 images? What's the purpose vs a single picture at a time? Does the model know that you're feeding it 4x4 examples, or does it have to figure that out itself? Nah, the grid is just for convenient viewing for humans. Robots see one image at a time. (Or more specifically, a batch of images; we happen to use batch size 2 or 4, I forget, so each core sees two images at a time, and then all 8 cores broadcast their gradients to each other and average, so it's really seeing 16 or 32 images at a time.) I feel a bit silly plugging our community so much, but it's really true. If you like tricks like this, join the Tensorfork discord: https://discord.com/invite/x52Xz3y https://discord.com/invite/x52Xz3y My theory when I set it up was that everyone has little tricks like this, but there's no central repository of knowledge / place to ask questions. But now that there are 1,200+ of us, it's become the de facto place to pop in and share random ideas and tricks. For what it's worth, https://thisanimedoesnotexist.ai/ https://thisanimedoesnotexist.ai/ was a joint collaboration of several Tensorfork discord members. :) If you want future updates about this specific BigGAN model, twitter is your best bet: https://twitter.com/search?q=(from%3Atheshawwn)%20biggan&src=typed_query https://twitter.com/search?q=(from%3Atheshawwn)%20biggan&src...

logane 6y ago

Not sure if I follow your JFT argument, but there's a large body of work on both (a) studying whether chasing ImageNet accuracy yields models that generalize well to out of distribution data [1, 2, 3] and (b) contextualizing progress on ImageNet (i.e., what does high accuracy on ImageNet really mean?) [4, 5, 6]. For (a), maybe surprisingly the answer is mostly yes! Better ImageNet accuracy generally corresponds to better out of distribution accuracy. For (b), it turns out that the ImageNet dataset is full of contradictions---many images have multiple ImageNet-relevant objects, and often are ambiguously or mis-labeled, etc---so it's hard to disentangle progress in identifying objects vs. models overfitting to the quirks of the benchmark. [1] ObjectNet: https://objectnet.dev https://objectnet.dev / associated paper [2] ImageNet-v2: https://arxiv.org/abs/1902.10811 https://arxiv.org/abs/1902.10811 [3] An Unbiased Lookat Dataset Bias: https://people.csail.mit.edu/torralba/publications/datasets_cvpr11.pdf https://people.csail.mit.edu/torralba/publications/datasets_... (pre-AlexNet!) [4] From ImageNet to Image Classification: https://arxiv.org/abs/2005.11295 https://arxiv.org/abs/2005.11295 [5] Are we done with ImageNet? https://arxiv.org/abs/2006.07159 https://arxiv.org/abs/2006.07159 [6] Evaluating Machine Accuracy on ImageNet: http://proceedings.mlr.press/v119/shankar20c.html http://proceedings.mlr.press/v119/shankar20c.html

code-scope 6y ago

Is there any (influence) SW framework that takes youtube video as input and split out object/timestamp as output?

indiv0 6y ago

That’s awesome! Is your model available publically? I run a site [0] where users can generate images from text prompts using models like the official BigGAN-Deep one and I’d love to try it out for this purpose. Do you also have somewhere whereupon discuss this stuff? I’m new to ML in general and was wondering if there’s somewhere where y’all experts gather. [0]: https://dank.xyz https://dank.xyz