6 ms·

Real story. I worked on nudity detection for mobile realtime video chat app with on-device deep learning model. (It was successful.) Should be easy: label some

Real story. I worked on nudity detection for mobile realtime video chat app with on-device deep learning model. (It was successful.) Should be easy: label some data, start from a good network pretrained on ImageNet (we used SqueezeNet), fine tune. The problem was that ImageNet is photos, not video frames, and distribution of video frames from mobile video is somewhat different. Incredibly large proportion of frames are either blank (camera is blocked) or ceiling (camera is misdirected). We ended up adding lots of data for blank and ceiling explicitly labeled as blank and ceiling. It became an excellent ceiling detector, detecting ceilings it never have seen before. Is there a mobile video frames dataset for labeling ceilings? Why not? I can't believe I am the only one who experienced this. Why is not a ceiling dataset worthy of, say, CVPR? It will improve mobile video on-device deep learning more than most of CVPR papers. This is a serious problem. Edit: I understand why niche datasets remain in industry and not academia. Public datasets are better if they are of general interest. But ceiling dataset is of general interest to anyone who wants to process video frames originating from blocked or misdirected camera aka smartphones, and it's hard to imagine topics of more general interest once you have obvious things like face.

ethbr0 6y ago
Because fewer PhD are minted by "Improved on SotA ceiling detection rates by 50%."
- sanxiyn 6y ago
  Maybe not PhD, but as a research topic, I think doing this well is an interesting task. I learned that ceilings are not featureless. (I know, it's a kind of ridiculous to say I "learned" this, since I could have learned this from, oh well, looking at my ceiling just now, but it really was a surprise when I encountered this in machine learning data context.) Ceilings often have lightings! Lightings (either on and off) vary between geography and culture! (They do! Extreme variations!) Lightings, if on, will cause flares! We just sampled from actual distribution (privacy policy included we can monitor connection for anti-abuse and we can use data for research and development of anti-abuse), but dataset collected with care will be valuable. I am sure actually doing this will reveal other considerations, and even just with points I mentioned, it should be good enough for a paper, if not for PhD.
  - ethbr0 6y ago
    That sounds like the problem in a nutshell. "X is simple, so just gin me up a data set of it." (Four months later) "X is not simple, in all these ways..." Which is another way of saying that discovery and communication with SMEs is an important side skill in project success.
deleted 6y ago
[deleted]
sillysaurusx 6y ago
Little crafty trick: grab the flickr dataset (yfcc100m) and snag photos tagged as whatever you want. It's like a janky google image search. I've put together datasets of airplanes, bicycles, dogs, etc this way. It's not entirely accurate, but it's good enough. Within a few hours you can have a pretty large dataset of whatever you want, really. (Yay for massive dataset plus user tags.) You can snag a copy from my server, if you want. (Warning: It's a direct link to a 54GB json file.) https://battle.shawwn.com/sdc/f100m/yfcc100m_dataset.json https://battle.shawwn.com/sdc/f100m/yfcc100m_dataset.json I have a script called "janky-image-search" that returns random results from searching this dataset. Here are a few random hits for `janky-image-search ceiling`: https://www.flickr.com/photos/35981213@N00/9374577304 https://www.flickr.com/photos/35981213@N00/9374577304 https://farm3.staticflickr.com/2084/2513539756_7a768de44c.jpg https://farm3.staticflickr.com/2084/2513539756_7a768de44c.jp... http://www.flickr.com/photos/96453841@N00/7077625757/ http://www.flickr.com/photos/96453841@N00/7077625757/ https://www.flickr.com/photos/28742299@N04/4683245369/ https://www.flickr.com/photos/28742299@N04/4683245369/ etc. The quality is hit or miss, but it seems better than 70%. EDIT: Here's a gallery of the first 100 hits for "ceiling": https://cdn.gather.town/storage.googleapis.com/gather-town.appspot.com/uploads//8COCgBeMAjI2etwdmTEJSB?:id=ceiling.html https://cdn.gather.town/storage.googleapis.com/gather-town.a... But as you can see, it's not effortless. Most of those ceilings are old. So it depends what you want. It's why labeled data is worth billions of dollars (scale.ai et al).
- sanxiyn 6y ago
  I found that people don't upload misdirected ceiling photos and videos to YouTube and Flickr. How unfortunate. Yes, it's better than nothing.
  - sillysaurusx 6y ago
    Now you've made me want to create a random-ceilings dataset. I spent about an hour trying to think of some way to get you a bunch of un-curated ceilings, to no avail. If you ended up using the ceilings uploaded to your app directly from your users, it might be possible that there wasn't any better solution than the one you came up with.
notsuoh 6y ago
Not an ML person, but I generally understand the concept. Why do you care about ceilings here? I get that the camera is pointed weird ways, but if you're trying to detect nudity presumably if trained on a bunch of images of that subject, it would just call ceilings False.
- sanxiyn 6y ago
  No, because training set does not include ceilings.
  - notsuoh 6y ago
    I think I understand, so you basically need in your training set anything that the production model would presumably see for it to work well? You can't just say here are a bunch of positives and a bunch of negatives, the negatives have to be actual things the model will see.
    - sanxiyn 6y ago
      Yes, with emphasis on "well". It does work without ceilings. But your training set should include everything you will frequently encounter in test set. Ceilings are too frequent to miss.
    - sanxiyn 6y ago
      If you think about it, this can't be otherwise. Positive/negative is misleading, since that distinction is the entire thing we are trying to teach. Let's call two labels A and B. You tell the model: This long penis is A. This short penis is A. This cat is B. This dog is B. Now, what is this ceiling? The model, looking at the ceiling, discovers a long fluorescent tube. It is long! Neither cat nor dog is long (The model is yet to discover longcat), and while penis comes in long and short variety, all penises seem long-ish. The ceiling is A.
    - jononor 6y ago
      In the typical ML task formulation, closed-set classification, the model is forced to make a choice between the provided categories, even if the output is not really either. And neural networks tend to perform erratically outside of the training data distribution. Adding common inputs to the training (or at least validation and test) sets is a good solution. Its hard data work, but will pay off. There are some techniques outside of closed-set classification that can help reduce the problems, or make the process of improving it more effective: - Couple the classifier with a out-of-distribution (novelty/anomaly) detector. Samples that score high are considered "Unknown" and can be flagged for review. - Learn a distance metric for "nudity" instead of a classifier, potentially with unsupervised or self-supervised learning (no labels needed). This has higher chance of doing well on novel examples, but it still needs to be validated/monitored. - Use one-class classifier, trained only on positive samples of nudity. This has the disadvantage that novel nudity is very likely to be classified as "not nudity", which could be an issue.
amelius 6y ago
Why didn't you train on random images from the web? If you carefully select your sources, most of them will be non-nudes.
- Cthulhu_ 6y ago
  I'm sure there's ways to extract loads of e.g. tiktok videos, especially the non trending / highly produced ones (I'm making assumptions about the use case here).
- collyw 6y ago
  If you "carefully select your sources" then you aren't training on "random images from the web". If you are using random images from the web, there is a decent chance that you will get a result like this. https://www.theguardian.com/commentisfree/2021/feb/03/what-a-picture-of-alexandria-ocasio-cortez-in-a-bikini-tells-us-about-the-disturbing-future-of-ai https://www.theguardian.com/commentisfree/2021/feb/03/what-a... The Guardian labels it sexist. I assumed that it was because the majority of young attractive women on the internet are posting bikini pictures.
  - gwd 6y ago
    > The Guardian labels it sexist. I assumed that it was because the majority of young attractive women on the internet are posting bikini pictures. They're both true at the same time, which is exactly the point. The people in the article in the paper weren't trying to teach their neural net that men usually wear suits and women usually wear bikinis, but they did. They only noticed because the failure mode was really obvious. In some other situation, the exact same kind of sexist bias might exist in the data -- and important decisions, affecting people's lives, may be made -- without anybody noticing first.
iaml 6y ago
I'm wondering if you could just take some 3d scenes of say flat interiors and then render them in a good quality with camera looking in random directions to use that as training data. You get actually correct labels because you can calculate it when rendering and label accordingly.
bonoboTP 6y ago
What would be the novel scientific contribution over any other image classifier? What's the storyline? How do I get excited? You collected a dataset for some boring class and trained a standard sota model on it. This is an undergrad project report, not a research paper. The main question people ask when considering reading a paper is "So what??"

ethbr0 6y ago

Because fewer PhD are minted by "Improved on SotA ceiling detection rates by 50%."

sanxiyn 6y ago
Maybe not PhD, but as a research topic, I think doing this well is an interesting task. I learned that ceilings are not featureless. (I know, it's a kind of ridiculous to say I "learned" this, since I could have learned this from, oh well, looking at my ceiling just now, but it really was a surprise when I encountered this in machine learning data context.) Ceilings often have lightings! Lightings (either on and off) vary between geography and culture! (They do! Extreme variations!) Lightings, if on, will cause flares! We just sampled from actual distribution (privacy policy included we can monitor connection for anti-abuse and we can use data for research and development of anti-abuse), but dataset collected with care will be valuable. I am sure actually doing this will reveal other considerations, and even just with points I mentioned, it should be good enough for a paper, if not for PhD.
- ethbr0 6y ago
  That sounds like the problem in a nutshell. "X is simple, so just gin me up a data set of it." (Four months later) "X is not simple, in all these ways..." Which is another way of saying that discovery and communication with SMEs is an important side skill in project success.

deleted 6y ago

[deleted]

sillysaurusx 6y ago

Little crafty trick: grab the flickr dataset (yfcc100m) and snag photos tagged as whatever you want. It's like a janky google image search. I've put together datasets of airplanes, bicycles, dogs, etc this way. It's not entirely accurate, but it's good enough. Within a few hours you can have a pretty large dataset of whatever you want, really. (Yay for massive dataset plus user tags.) You can snag a copy from my server, if you want. (Warning: It's a direct link to a 54GB json file.) https://battle.shawwn.com/sdc/f100m/yfcc100m_dataset.json https://battle.shawwn.com/sdc/f100m/yfcc100m_dataset.json I have a script called "janky-image-search" that returns random results from searching this dataset. Here are a few random hits for `janky-image-search ceiling`: https://www.flickr.com/photos/35981213@N00/9374577304 https://www.flickr.com/photos/35981213@N00/9374577304 https://farm3.staticflickr.com/2084/2513539756_7a768de44c.jpg https://farm3.staticflickr.com/2084/2513539756_7a768de44c.jp... http://www.flickr.com/photos/96453841@N00/7077625757/ http://www.flickr.com/photos/96453841@N00/7077625757/ https://www.flickr.com/photos/28742299@N04/4683245369/ https://www.flickr.com/photos/28742299@N04/4683245369/ etc. The quality is hit or miss, but it seems better than 70%. EDIT: Here's a gallery of the first 100 hits for "ceiling": https://cdn.gather.town/storage.googleapis.com/gather-town.appspot.com/uploads//8COCgBeMAjI2etwdmTEJSB?:id=ceiling.html https://cdn.gather.town/storage.googleapis.com/gather-town.a... But as you can see, it's not effortless. Most of those ceilings are old. So it depends what you want. It's why labeled data is worth billions of dollars (scale.ai et al).

sanxiyn 6y ago
I found that people don't upload misdirected ceiling photos and videos to YouTube and Flickr. How unfortunate. Yes, it's better than nothing.
- sillysaurusx 6y ago
  Now you've made me want to create a random-ceilings dataset. I spent about an hour trying to think of some way to get you a bunch of un-curated ceilings, to no avail. If you ended up using the ceilings uploaded to your app directly from your users, it might be possible that there wasn't any better solution than the one you came up with.

notsuoh 6y ago

Not an ML person, but I generally understand the concept. Why do you care about ceilings here? I get that the camera is pointed weird ways, but if you're trying to detect nudity presumably if trained on a bunch of images of that subject, it would just call ceilings False.

sanxiyn 6y ago
No, because training set does not include ceilings.
- notsuoh 6y ago
  I think I understand, so you basically need in your training set anything that the production model would presumably see for it to work well? You can't just say here are a bunch of positives and a bunch of negatives, the negatives have to be actual things the model will see.
  - sanxiyn 6y ago
    Yes, with emphasis on "well". It does work without ceilings. But your training set should include everything you will frequently encounter in test set. Ceilings are too frequent to miss.
  - sanxiyn 6y ago
    If you think about it, this can't be otherwise. Positive/negative is misleading, since that distinction is the entire thing we are trying to teach. Let's call two labels A and B. You tell the model: This long penis is A. This short penis is A. This cat is B. This dog is B. Now, what is this ceiling? The model, looking at the ceiling, discovers a long fluorescent tube. It is long! Neither cat nor dog is long (The model is yet to discover longcat), and while penis comes in long and short variety, all penises seem long-ish. The ceiling is A.
  - jononor 6y ago
    In the typical ML task formulation, closed-set classification, the model is forced to make a choice between the provided categories, even if the output is not really either. And neural networks tend to perform erratically outside of the training data distribution. Adding common inputs to the training (or at least validation and test) sets is a good solution. Its hard data work, but will pay off. There are some techniques outside of closed-set classification that can help reduce the problems, or make the process of improving it more effective: - Couple the classifier with a out-of-distribution (novelty/anomaly) detector. Samples that score high are considered "Unknown" and can be flagged for review. - Learn a distance metric for "nudity" instead of a classifier, potentially with unsupervised or self-supervised learning (no labels needed). This has higher chance of doing well on novel examples, but it still needs to be validated/monitored. - Use one-class classifier, trained only on positive samples of nudity. This has the disadvantage that novel nudity is very likely to be classified as "not nudity", which could be an issue.

amelius 6y ago

Why didn't you train on random images from the web? If you carefully select your sources, most of them will be non-nudes.

Cthulhu_ 6y ago
I'm sure there's ways to extract loads of e.g. tiktok videos, especially the non trending / highly produced ones (I'm making assumptions about the use case here).
collyw 6y ago
If you "carefully select your sources" then you aren't training on "random images from the web". If you are using random images from the web, there is a decent chance that you will get a result like this. https://www.theguardian.com/commentisfree/2021/feb/03/what-a-picture-of-alexandria-ocasio-cortez-in-a-bikini-tells-us-about-the-disturbing-future-of-ai https://www.theguardian.com/commentisfree/2021/feb/03/what-a... The Guardian labels it sexist. I assumed that it was because the majority of young attractive women on the internet are posting bikini pictures.
- gwd 6y ago
  > The Guardian labels it sexist. I assumed that it was because the majority of young attractive women on the internet are posting bikini pictures. They're both true at the same time, which is exactly the point. The people in the article in the paper weren't trying to teach their neural net that men usually wear suits and women usually wear bikinis, but they did. They only noticed because the failure mode was really obvious. In some other situation, the exact same kind of sexist bias might exist in the data -- and important decisions, affecting people's lives, may be made -- without anybody noticing first.

iaml 6y ago

I'm wondering if you could just take some 3d scenes of say flat interiors and then render them in a good quality with camera looking in random directions to use that as training data. You get actually correct labels because you can calculate it when rendering and label accordingly.

bonoboTP 6y ago

What would be the novel scientific contribution over any other image classifier? What's the storyline? How do I get excited? You collected a dataset for some boring class and trained a standard sota model on it. This is an undergrad project report, not a research paper. The main question people ask when considering reading a paper is "So what??"