3 ms·

What safeguards are there or what assurances do we have that JFT is not contaminated with images from (or extremely similar to) the validation set?

sillysaurusx 6y ago
Haha. None whatsoever. The assurance is that everyone in the field seems to take the work seriously. But the reality is that errors creep in from a variety of corners. I would not be even slightly surprised to find that the validation data is substantially similar. We're still at the "bangs rocks together to make fire" phase of ML, which is both exciting and challenging; we're building the future from the ground up. People rarely take the time to look at the actual images, but if you do, you'll notice they have some interesting errors in them: https://twitter.com/theshawwn/status/1262535747975868418 https://twitter.com/theshawwn/status/1262535747975868418 I built an interactive viewer for the tagging site: https://tags.tagpls.com/ https://tags.tagpls.com/ (Someone tagged all 70 shoes in this one, which was kind of impressive... https://tags.shawwn.com/tags/https://battle.shawwn.com/sdc/imagenet/validation/ILSVRC2012_val_00049459.JPEG https://tags.shawwn.com/tags/https://battle.shawwn.com/sdc/i... ) Anyway, some of the validation images happen to be rotated 90 degrees and no one noticed. That made me wonder what other sorts of unexpected errors are in these specific 50,000 validation images that the world just-so-happened to decide were Super Important to the future of AI. The trouble is, images in general are substantially similar to the imagenet validation dataset. In other words, it's tempting to try to think of some way of "dividing up" the data so that there's some sort of validation phase that you can cleanly separate. But reality isn't so kind. When you're at the scale of millions of images, holding out 10% is just a way of sanity checking that your model isn't memorizing the training data; nothing more. Besides, random 90 degree rotations are introduced on purpose now, so it's funny that old mistakes tend not to matter.
quantumofalpha 6y ago
Just the sheer size of JFT (latest versions I heard approach 1B images), so it's probably impractical to train on it till overfitting.

sillysaurusx 6y ago

Haha. None whatsoever. The assurance is that everyone in the field seems to take the work seriously. But the reality is that errors creep in from a variety of corners. I would not be even slightly surprised to find that the validation data is substantially similar. We're still at the "bangs rocks together to make fire" phase of ML, which is both exciting and challenging; we're building the future from the ground up. People rarely take the time to look at the actual images, but if you do, you'll notice they have some interesting errors in them: https://twitter.com/theshawwn/status/1262535747975868418 https://twitter.com/theshawwn/status/1262535747975868418 I built an interactive viewer for the tagging site: https://tags.tagpls.com/ https://tags.tagpls.com/ (Someone tagged all 70 shoes in this one, which was kind of impressive... https://tags.shawwn.com/tags/https://battle.shawwn.com/sdc/imagenet/validation/ILSVRC2012_val_00049459.JPEG https://tags.shawwn.com/tags/https://battle.shawwn.com/sdc/i... ) Anyway, some of the validation images happen to be rotated 90 degrees and no one noticed. That made me wonder what other sorts of unexpected errors are in these specific 50,000 validation images that the world just-so-happened to decide were Super Important to the future of AI. The trouble is, images in general are substantially similar to the imagenet validation dataset. In other words, it's tempting to try to think of some way of "dividing up" the data so that there's some sort of validation phase that you can cleanly separate. But reality isn't so kind. When you're at the scale of millions of images, holding out 10% is just a way of sanity checking that your model isn't memorizing the training data; nothing more. Besides, random 90 degree rotations are introduced on purpose now, so it's funny that old mistakes tend not to matter.

quantumofalpha 6y ago

Just the sheer size of JFT (latest versions I heard approach 1B images), so it's probably impractical to train on it till overfitting.