7 ms·

Introducing the Open Images Dataset

168 points by hurrycane 10y ago

devindotcom 10y ago
First video, now images - wonder if speech and others are on the way? It's nice that they're doing this, helps advance the art I think. But it also puts a lot of smaller operations in unis sort of under the Google system in that they're best compared to Google's ML work and others using these datasets. It's a small way of stacking the deck to make Google and DeepMind more embedded in the community. That said, its utility for others surely outweighs the strategic advantage gained here, so I for one welcome these libraries. A lot of work goes into them. Hopefully others will release theirs as well.
- imh 10y ago
  I really want real-world speech data. I think speech-to-text could open the doors to lots of creative tech to help hard of hearing folks, but the dataset barrier is so massive :(
  - Houshalter 10y ago
    I wonder why semi supervised learning hasn't taken off more. There is so much unlabelled data out there. E.g. in podcasts and youtube videos and television. A small amount of it is labelled, like captioned television programs and youtube videos. You could use those captions to train a weak model which could provide labels for the unlabelled data. You can correct many of it's errors with simple language models. For instance the phrase "wreck a nice beach" has much less probability than "recognize speech". So if the model isn't sure which one it is, you can assume it's the more probable one. Then train on that, and it will get even better at recognizing those words.
    - rm999 10y ago
      >You can correct many of it's errors with simple language models. For instance the phrase "wreck a nice beach" has much less probability than "recognize speech". So if the model isn't sure which one it is, you can assume it's the more probable one. Then train on that, and it will get even better at recognizing those words. Why not just build that heuristic into your initial supervised "weak" model? Training on data labeled by a model introduces no new information, so you're not gaining anything there.
      - Houshalter 10y ago
        It does introduce new information. This is how semi-supervised learning works. Many words may be missed by the weak model, but can be inferred correctly from their context. Then you have new labels to train the weak model on, to make it better at those words. The way I'm describing it is probably not the optimal way to do it. I don't know if there is a better way. But the point is it must be possible to take advantage of the vast quantities of unlabelled data we have. Humans brains somehow do something similar. Semi supervised learning is really cool. I saw one successful example where they labelled just a few emails as spam and not spam. Then they trained used their weak classifier to label thousands of unclassified emails, and then used those as training data for an even stronger model. It actually works: http://matpalm.com/semi_supervised_naive_bayes/semi_supervised_bayes.html http://matpalm.com/semi_supervised_naive_bayes/semi_supervis... https://en.wikipedia.org/wiki/Semi-supervised_learning https://en.wikipedia.org/wiki/Semi-supervised_learning
    - visarga 10y ago
      If you make your training set the output of Google's speech recognition algo from YT there will be lots of label noise. It's better to go with movies and audiobooks because the text 'transcription' is human generated. Aligning known text to audio is much more accurate than speech recognition.
  - Smerity 10y ago
    Have you seen the LibriSpeech ASR corpus[1]? It's large scale (1k hours) Creative Commons licensed English speech with the original book text. The data is derived from read audiobooks from the LibriVox project and has been carefully segmented and aligned. If you mean real world as in "with realistic noise", Baidu's DeepSpeech showed that adding noise to clean audio can make a resilient model. Indeed, having clean audio is better as you can add different types of noise to clean data to, in effect, create regularization. [1]: http://www.openslr.org/12/ http://www.openslr.org/12/
  - mynameislegion 10y ago
    http://voxforge.org/ http://voxforge.org/
- chillydawg 10y ago
  TV show audio tracks with subtitles? Not perfect, but a very, very large available dataset if you feel like downloading enough dodgy torrents. Plus points would be that any model trained would also need to be able to distinguish and separate speakers, which is something even the human brain struggles with.
  - PavlovsCat 10y ago
    Record people with a good microphone as they use their cell phone / landline (or other communication apps that compress heavily), and also record what comes out at the other end, the other end being all sorts of places, including overseas. Train a huge neural network with that, until it can extrapolate bad phone quality into something superb. Next up: x-ray glasses, with several modes, to turn people into the hottest or ugliest possible version that could hide under those clothes.
transcranial 10y ago
Interesting that the base data consists of URLs. I guess it makes sense given copyright issues. Anybody know what the ballpark expected half-life of such URLs?
- DarkLinkXXXX 10y ago
  I was wondering about that as well. I'd hope that all these images are cached by archive.org or the like.
  - Smerity 10y ago
    This is the tactic used by the CNN / Dailymail dataset[1] released by Google Deepmind. In that situation you want everyone to have the same and original file. The contents of the URL may be updated and/or disappear. The original URLs are recorded but they're retrieved from the Wayback Machine. Even with that a few of the pages are still essentially lost however - or at least I was never able to retrieve them. Kyunghyun Cho hosts a copy of the processed data on his site at NYU - who may be less likely to receive legal requests compared to Google or a similar commercial company hosting them. Distribution of datasets is going to be a continued painpoint for machine learning in the future. No-one wants to be the legal guinea pig for exactly what entails fair use in ML. [1]: https://github.com/deepmind/rc-data https://github.com/deepmind/rc-data
    - DarkLinkXXXX 10y ago
      Agreed. Do you think maybe these datasets should include checksums so that they might be recovered with more certainty, from less obvious sources? P.S.: Also, archive.org retroactively removes (or delists?) site content when blocked by robots.txt, as domain parkers often do, so I could see that causing trouble down the line. https://archive.org/post/1019415/retroactive-robotstxt-removal-of-past-crawls-aka-oakland-archive-policy https://archive.org/post/1019415/retroactive-robotstxt-remov...
      - krasin 10y ago
rocky1138 10y ago
Are there any other libraries that are similar?
dharma1 10y ago
Is there a link to the trained model somewhere?
- goddamnsteve 10y ago
  Even I've been searching for some links for the trained models. May be its too early for now.
  - krasin 10y ago
    Stay tuned: https://github.com/openimages/dataset/issues/3 https://github.com/openimages/dataset/issues/3
imh 10y ago
Lawyers are funny: >Today, we introduce Open Images, a dataset consisting of ~9 million URLs ... having a Creative Commons Attribution license* . Then the footnote below: >* While we tried to identify images that are licensed under a Creative Commons Attribution license, we make no representations or warranties regarding the license status of each image and you should verify the license for each image yourself. I think this might be the most blatant instance I've ever seen of, "We have to write this even though it's essentially impossible for you to actually follow our directions."
- zappo2938 10y ago
  Any image where it is impossible to find the originating source, already used thousands of times on tumbler, blogs, and websites, and is under a certain resolution and size (thumbnail for example) falls into the category of fair use.
  - teddyh 10y ago
    Nope. Everything is copyrighted by default, and if you don’t have permission (“license”) to copy it, it’s illegal for you to copy it. There are the “fair use” exceptions, but those have nothing to do with what you wrote.
    - zappo2938 10y ago
      I clearly misunderstood what is meant by "Would it be considered impossible to obtain permission from the original source?" as criteria for fair use.[0] I've seen numerous images on Wikipedia that use this as reason for fair use. Someone uploads an image to 4chan as anonymous, for example, without a path to ask permission it might be fair use to use that image.This is different from someone posting an image to facebook where there is a way to ask the author for permission. Please, explain my misunderstanding here. [0]: https://i.kinja-img.com/gawker-media/image/upload/qxptplnxqtrhme965gal.jpg https://i.kinja-img.com/gawker-media/image/upload/qxptplnxqt...
      - teddyh 10y ago
diyseguy 10y ago
Any guesses on how large the resulting dataset would be if you actually downloaded all the images? I imagine the urls will get removed in a hurry as everybody starts automating it.
- krasin 10y ago
  (disclosure: I am one of the contributors to the dataset) ~1TB for 640x480 thumbnails, ~3TB for 1600x1200 thumbnails. The originals are about 20TB, though.
Omnipresent 10y ago
Looking forward to someone trying tensorFlow CNN on this
zappo2938 10y ago
I'm glad I'm getting a return on all the effort clicking street signs and store fronts on reCaptcha.
pilooch 10y ago
I've put an efficient downloader here for the interested crowd: https://github.com/beniz/openimages_downloader https://github.com/beniz/openimages_downloader It's a fork of the one script I used to grab Imagenet.

168 points by hurrycane 10y ago

First video, now images - wonder if speech and others are on the way? It's nice that they're doing this, helps advance the art I think. But it also puts a lot of smaller operations in unis sort of under the Google system in that they're best compared to Google's ML work and others using these datasets. It's a small way of stacking the deck to make Google and DeepMind more embedded in the community. That said, its utility for others surely outweighs the strategic advantage gained here, so I for one welcome these libraries. A lot of work goes into them. Hopefully others will release theirs as well.

imh 10y ago
I really want real-world speech data. I think speech-to-text could open the doors to lots of creative tech to help hard of hearing folks, but the dataset barrier is so massive :(
- Houshalter 10y ago
  I wonder why semi supervised learning hasn't taken off more. There is so much unlabelled data out there. E.g. in podcasts and youtube videos and television. A small amount of it is labelled, like captioned television programs and youtube videos. You could use those captions to train a weak model which could provide labels for the unlabelled data. You can correct many of it's errors with simple language models. For instance the phrase "wreck a nice beach" has much less probability than "recognize speech". So if the model isn't sure which one it is, you can assume it's the more probable one. Then train on that, and it will get even better at recognizing those words.
  - rm999 10y ago
    >You can correct many of it's errors with simple language models. For instance the phrase "wreck a nice beach" has much less probability than "recognize speech". So if the model isn't sure which one it is, you can assume it's the more probable one. Then train on that, and it will get even better at recognizing those words. Why not just build that heuristic into your initial supervised "weak" model? Training on data labeled by a model introduces no new information, so you're not gaining anything there.
    - Houshalter 10y ago
      It does introduce new information. This is how semi-supervised learning works. Many words may be missed by the weak model, but can be inferred correctly from their context. Then you have new labels to train the weak model on, to make it better at those words. The way I'm describing it is probably not the optimal way to do it. I don't know if there is a better way. But the point is it must be possible to take advantage of the vast quantities of unlabelled data we have. Humans brains somehow do something similar. Semi supervised learning is really cool. I saw one successful example where they labelled just a few emails as spam and not spam. Then they trained used their weak classifier to label thousands of unclassified emails, and then used those as training data for an even stronger model. It actually works: http://matpalm.com/semi_supervised_naive_bayes/semi_supervised_bayes.html http://matpalm.com/semi_supervised_naive_bayes/semi_supervis... https://en.wikipedia.org/wiki/Semi-supervised_learning https://en.wikipedia.org/wiki/Semi-supervised_learning
  - visarga 10y ago
    If you make your training set the output of Google's speech recognition algo from YT there will be lots of label noise. It's better to go with movies and audiobooks because the text 'transcription' is human generated. Aligning known text to audio is much more accurate than speech recognition.
- Smerity 10y ago
  Have you seen the LibriSpeech ASR corpus[1]? It's large scale (1k hours) Creative Commons licensed English speech with the original book text. The data is derived from read audiobooks from the LibriVox project and has been carefully segmented and aligned. If you mean real world as in "with realistic noise", Baidu's DeepSpeech showed that adding noise to clean audio can make a resilient model. Indeed, having clean audio is better as you can add different types of noise to clean data to, in effect, create regularization. [1]: http://www.openslr.org/12/ http://www.openslr.org/12/
- mynameislegion 10y ago
  http://voxforge.org/ http://voxforge.org/
chillydawg 10y ago
TV show audio tracks with subtitles? Not perfect, but a very, very large available dataset if you feel like downloading enough dodgy torrents. Plus points would be that any model trained would also need to be able to distinguish and separate speakers, which is something even the human brain struggles with.
- PavlovsCat 10y ago
  Record people with a good microphone as they use their cell phone / landline (or other communication apps that compress heavily), and also record what comes out at the other end, the other end being all sorts of places, including overseas. Train a huge neural network with that, until it can extrapolate bad phone quality into something superb. Next up: x-ray glasses, with several modes, to turn people into the hottest or ugliest possible version that could hide under those clothes.

transcranial 10y ago

Interesting that the base data consists of URLs. I guess it makes sense given copyright issues. Anybody know what the ballpark expected half-life of such URLs?

DarkLinkXXXX 10y ago
I was wondering about that as well. I'd hope that all these images are cached by archive.org or the like.
- Smerity 10y ago
  This is the tactic used by the CNN / Dailymail dataset[1] released by Google Deepmind. In that situation you want everyone to have the same and original file. The contents of the URL may be updated and/or disappear. The original URLs are recorded but they're retrieved from the Wayback Machine. Even with that a few of the pages are still essentially lost however - or at least I was never able to retrieve them. Kyunghyun Cho hosts a copy of the processed data on his site at NYU - who may be less likely to receive legal requests compared to Google or a similar commercial company hosting them. Distribution of datasets is going to be a continued painpoint for machine learning in the future. No-one wants to be the legal guinea pig for exactly what entails fair use in ML. [1]: https://github.com/deepmind/rc-data https://github.com/deepmind/rc-data
  - DarkLinkXXXX 10y ago
    Agreed. Do you think maybe these datasets should include checksums so that they might be recovered with more certainty, from less obvious sources? P.S.: Also, archive.org retroactively removes (or delists?) site content when blocked by robots.txt, as domain parkers often do, so I could see that causing trouble down the line. https://archive.org/post/1019415/retroactive-robotstxt-removal-of-past-crawls-aka-oakland-archive-policy https://archive.org/post/1019415/retroactive-robotstxt-remov...
    - krasin 10y ago

rocky1138 10y ago

Are there any other libraries that are similar?

dharma1 10y ago

Is there a link to the trained model somewhere?

goddamnsteve 10y ago
Even I've been searching for some links for the trained models. May be its too early for now.
- krasin 10y ago
  Stay tuned: https://github.com/openimages/dataset/issues/3 https://github.com/openimages/dataset/issues/3

imh 10y ago

Lawyers are funny: >Today, we introduce Open Images, a dataset consisting of ~9 million URLs ... having a Creative Commons Attribution license* . Then the footnote below: >* While we tried to identify images that are licensed under a Creative Commons Attribution license, we make no representations or warranties regarding the license status of each image and you should verify the license for each image yourself. I think this might be the most blatant instance I've ever seen of, "We have to write this even though it's essentially impossible for you to actually follow our directions."

zappo2938 10y ago
Any image where it is impossible to find the originating source, already used thousands of times on tumbler, blogs, and websites, and is under a certain resolution and size (thumbnail for example) falls into the category of fair use.
- teddyh 10y ago
  Nope. Everything is copyrighted by default, and if you don’t have permission (“license”) to copy it, it’s illegal for you to copy it. There are the “fair use” exceptions, but those have nothing to do with what you wrote.
  - zappo2938 10y ago
    I clearly misunderstood what is meant by "Would it be considered impossible to obtain permission from the original source?" as criteria for fair use.[0] I've seen numerous images on Wikipedia that use this as reason for fair use. Someone uploads an image to 4chan as anonymous, for example, without a path to ask permission it might be fair use to use that image.This is different from someone posting an image to facebook where there is a way to ask the author for permission. Please, explain my misunderstanding here. [0]: https://i.kinja-img.com/gawker-media/image/upload/qxptplnxqtrhme965gal.jpg https://i.kinja-img.com/gawker-media/image/upload/qxptplnxqt...
    - teddyh 10y ago

diyseguy 10y ago

Any guesses on how large the resulting dataset would be if you actually downloaded all the images? I imagine the urls will get removed in a hurry as everybody starts automating it.

krasin 10y ago
(disclosure: I am one of the contributors to the dataset) ~1TB for 640x480 thumbnails, ~3TB for 1600x1200 thumbnails. The originals are about 20TB, though.

Omnipresent 10y ago

Looking forward to someone trying tensorFlow CNN on this

zappo2938 10y ago

I'm glad I'm getting a return on all the effort clicking street signs and store fronts on reCaptcha.

pilooch 10y ago

I've put an efficient downloader here for the interested crowd: https://github.com/beniz/openimages_downloader https://github.com/beniz/openimages_downloader It's a fork of the one script I used to grab Imagenet.