10 ms·

Psychedelic stickers that interfere with AI image recognition

113 points by zipwitch 9y ago

tw1010 9y ago
These are the types of fun details that are totally going to be part of future history lessons (no matter which direction, positive or negative, all of this is heading towards).
- zitterbewegung 9y ago
  Hardening production ML systems are going to be fun. I think exploiting ML will be the new kid on the block just like how we were introduced to XSS exploits. See https://github.com/cchio/deep-pwning https://github.com/cchio/deep-pwning
  - mtgx 9y ago
    Indeed. It's already starting to happen: https://www.csail.mit.edu/news/fooling-googles-image-recognition-ai-1000x-faster https://www.csail.mit.edu/news/fooling-googles-image-recogni...
    - username223 9y ago
      Trying to "harden" million-parameter models trained on a relatively small number of relevant examples will be a nightmare to make web security look easy. PS -- NIPS is in Long Beach now? What a shame.
      - robotresearcher 9y ago
        It left Vancouver a few years ago and has been moving around. It's getting too big for many cities now.
        username223 9y ago
        Too bad... Vancouver/Whistler was an awesome place to have a conference, even if the weather was rainy and the skiing so-so. A NIPS too big to be hosted by anything but a mega-city sounds depressing.
Iv 9y ago
Psychedelic stickers that interfere with one specific model used in AI image recognition. If necessary, next week these system can learn to ignore these.
- kaybe 9y ago
  Check out the 34C3 talk on adversarial AI. They found that the percentages of fooling are still high if one gets the adversary model completely wrong when designing the attack generating network, so it seems surprisingly stable.
  - Iv 9y ago
    I think that working around these type of fooling is easy but not really worthwhile for now. After all, adversarial models are designed to improve the performances of the models. Also in the article, they test a detector that has to identify a single object in an image that contains two: place an actual toaster next to the banana and call it fooled.
    - solarkraft 9y ago
      The point there may have been that the sticker over-powered the banana. The article does leave me with more questions than answers.
      - Iv 9y ago
        Ask them, I may have some answers.
  - glogla 9y ago
    Exactly! More than that, you can fool models that work completely differently (like decision trees, SVM and kNN) with false data made for the other model, which shows some kind of underlying similarity in there that we don't know yet. Or maybe just similarity of the training sets?
    - 9y ago
dasil003 9y ago
It actually does kind of look like a toaster.
- kaybe 9y ago
  I agree, especially on a small screen. Though it does seem to have elements of a pot plant too.
- waynecochran 9y ago
  Indeed ... looks like a toaster... I would say the AI is pretty dang good.
  - ascorbic 9y ago
    I think the point is that even though it's smaller and less obvious that the other objects, it's still sufficient to "hijack" the recognition for the whole image. If they had a little sticker with a normal picture of a toaster on it it's unlikely that it would've prevented the banana from being recognised, and the image only looks a bit like a toaster, whereas the banana is unambiguously recognisable.
ModernMech 9y ago
It's pretty funny seeing this post directly under this: "Beijing bets on facial recognition in a big drive for total surveillance" I guess we'll see people sticking these on their faces?
- QAPereo 9y ago
  I would guess that if that becomes an issue, such behavior will rapidly be criminalized, and in China, harshly punished.
ryandrake 9y ago
A potential application of this could be some kind of “privacy sticker” that you’d wear on your hat or your face in order to disable automated facial recognition systems.
- monksy 9y ago
  Unless things have changed. This shouldn't affect face recognition systems. Facial recognition systems look for 2 things. 1. They look for the general shape of the face 2. They look for a skin region and classify it based on shape. From the segment, then it becomes a complex search problem to match the face to a known face.
  - bigiain 9y ago
    I'm guessing the opportunity here is to disrupt the face detection step - make the face detector zero in on your non-face sticker/image, so the face recogniser never gets passed an actual face.
    - tpxl 9y ago
      Face detectors generally check the whole image for faces, outputting everything above a threshold, so as long as a face is present, it's getting detected.
    - monksy 9y ago
      I would suggest that you disrupt the face's shape or wear some kind facepaint to avoid the skin dection part.
- ajr0 9y ago
  I prefer a rings of Agus type solution (RE: Freedom (part two of Daemon by Danial Suarez)
- zipwitch 9y ago
  I've seen discussion floating around of dazzle camouflage for the face - a sort of blend of WWI meet 80s glam rock.
dghughes 9y ago
When I saw the "oily" legs on reddit I was curious if such an illusion could be used to fool AI camera surveillance. The recent article on China's surveillance network came to mind. Oily legs illusion https://i.imgur.com/14U9rqn.jpg https://i.imgur.com/14U9rqn.jpg
nopinsight 9y ago
It appears to me that this mostly fools whole-image classification algorithms. If the system performs object segmentation first and then applies classification to each object in the scene, this method is unlikely to be effective. One can paste such a sticker on top of the face or other objects to be disguised and it might reduce recognition accuracy a bit, but applying some “paint-in” algorithms to fill in the blank covered by the sticker would basically remove its effect. That is unless it is used to cover some prominent features, although that is often unpractical in many circumstances.
- asdfaefasdf 9y ago
  This sticker only works against this classifier. If you start changing the algorithm, you'd need to change the attack to match. If you think you can write a better image classifier by first segmenting the image before using ML, then I encourage you to get your own computer vision paper published and see how that works for you.
ted_dunning 9y ago
I think that the technical term for these should be "squirrels". https://www.youtube.com/watch?v=SSUXXzN26zg https://www.youtube.com/watch?v=SSUXXzN26zg
vog 9y ago
That reminds of the old days when automatic "self-learning" SPAM classification began. Back then, spammers sent deliberately gibberish messages. The goal was that users (rightfully) marked those as SPAM, somehow disturbing the machine learning and thus weakening the overall SPAM recognition. Alas, I don't know if this was actually working, and if so, how large the effect was. This would be an interesting bit of history.
wallstprog 9y ago
Interestingly, William Gibson includes something very much like this as a plot point at the end of "Zero History."

113 points by zipwitch 9y ago

These are the types of fun details that are totally going to be part of future history lessons (no matter which direction, positive or negative, all of this is heading towards).

zitterbewegung 9y ago
Hardening production ML systems are going to be fun. I think exploiting ML will be the new kid on the block just like how we were introduced to XSS exploits. See https://github.com/cchio/deep-pwning https://github.com/cchio/deep-pwning
- mtgx 9y ago
  Indeed. It's already starting to happen: https://www.csail.mit.edu/news/fooling-googles-image-recognition-ai-1000x-faster https://www.csail.mit.edu/news/fooling-googles-image-recogni...
  - username223 9y ago
    Trying to "harden" million-parameter models trained on a relatively small number of relevant examples will be a nightmare to make web security look easy. PS -- NIPS is in Long Beach now? What a shame.
    - robotresearcher 9y ago
      It left Vancouver a few years ago and has been moving around. It's getting too big for many cities now.
      - username223 9y ago
        Too bad... Vancouver/Whistler was an awesome place to have a conference, even if the weather was rainy and the skiing so-so. A NIPS too big to be hosted by anything but a mega-city sounds depressing.

Iv 9y ago

Psychedelic stickers that interfere with one specific model used in AI image recognition. If necessary, next week these system can learn to ignore these.

kaybe 9y ago
Check out the 34C3 talk on adversarial AI. They found that the percentages of fooling are still high if one gets the adversary model completely wrong when designing the attack generating network, so it seems surprisingly stable.
- Iv 9y ago
  I think that working around these type of fooling is easy but not really worthwhile for now. After all, adversarial models are designed to improve the performances of the models. Also in the article, they test a detector that has to identify a single object in an image that contains two: place an actual toaster next to the banana and call it fooled.
  - solarkraft 9y ago
    The point there may have been that the sticker over-powered the banana. The article does leave me with more questions than answers.
    - Iv 9y ago
      Ask them, I may have some answers.
- glogla 9y ago
  Exactly! More than that, you can fool models that work completely differently (like decision trees, SVM and kNN) with false data made for the other model, which shows some kind of underlying similarity in there that we don't know yet. Or maybe just similarity of the training sets?
  - 9y ago

dasil003 9y ago

It actually does kind of look like a toaster.

kaybe 9y ago
I agree, especially on a small screen. Though it does seem to have elements of a pot plant too.
waynecochran 9y ago
Indeed ... looks like a toaster... I would say the AI is pretty dang good.
- ascorbic 9y ago
  I think the point is that even though it's smaller and less obvious that the other objects, it's still sufficient to "hijack" the recognition for the whole image. If they had a little sticker with a normal picture of a toaster on it it's unlikely that it would've prevented the banana from being recognised, and the image only looks a bit like a toaster, whereas the banana is unambiguously recognisable.

ModernMech 9y ago

It's pretty funny seeing this post directly under this: "Beijing bets on facial recognition in a big drive for total surveillance" I guess we'll see people sticking these on their faces?

QAPereo 9y ago
I would guess that if that becomes an issue, such behavior will rapidly be criminalized, and in China, harshly punished.

ryandrake 9y ago

A potential application of this could be some kind of “privacy sticker” that you’d wear on your hat or your face in order to disable automated facial recognition systems.

monksy 9y ago
Unless things have changed. This shouldn't affect face recognition systems. Facial recognition systems look for 2 things. 1. They look for the general shape of the face 2. They look for a skin region and classify it based on shape. From the segment, then it becomes a complex search problem to match the face to a known face.
- bigiain 9y ago
  I'm guessing the opportunity here is to disrupt the face detection step - make the face detector zero in on your non-face sticker/image, so the face recogniser never gets passed an actual face.
  - tpxl 9y ago
    Face detectors generally check the whole image for faces, outputting everything above a threshold, so as long as a face is present, it's getting detected.
  - monksy 9y ago
    I would suggest that you disrupt the face's shape or wear some kind facepaint to avoid the skin dection part.
ajr0 9y ago
I prefer a rings of Agus type solution (RE: Freedom (part two of Daemon by Danial Suarez)
zipwitch 9y ago
I've seen discussion floating around of dazzle camouflage for the face - a sort of blend of WWI meet 80s glam rock.

dghughes 9y ago

When I saw the "oily" legs on reddit I was curious if such an illusion could be used to fool AI camera surveillance. The recent article on China's surveillance network came to mind. Oily legs illusion https://i.imgur.com/14U9rqn.jpg https://i.imgur.com/14U9rqn.jpg

nopinsight 9y ago

It appears to me that this mostly fools whole-image classification algorithms. If the system performs object segmentation first and then applies classification to each object in the scene, this method is unlikely to be effective. One can paste such a sticker on top of the face or other objects to be disguised and it might reduce recognition accuracy a bit, but applying some “paint-in” algorithms to fill in the blank covered by the sticker would basically remove its effect. That is unless it is used to cover some prominent features, although that is often unpractical in many circumstances.

asdfaefasdf 9y ago
This sticker only works against this classifier. If you start changing the algorithm, you'd need to change the attack to match. If you think you can write a better image classifier by first segmenting the image before using ML, then I encourage you to get your own computer vision paper published and see how that works for you.

ted_dunning 9y ago

I think that the technical term for these should be "squirrels". https://www.youtube.com/watch?v=SSUXXzN26zg https://www.youtube.com/watch?v=SSUXXzN26zg

vog 9y ago

That reminds of the old days when automatic "self-learning" SPAM classification began. Back then, spammers sent deliberately gibberish messages. The goal was that users (rightfully) marked those as SPAM, somehow disturbing the machine learning and thus weakening the overall SPAM recognition. Alas, I don't know if this was actually working, and if so, how large the effect was. This would be an interesting bit of history.

wallstprog 9y ago

Interestingly, William Gibson includes something very much like this as a plot point at the end of "Zero History."