5 ms·

Making an ML Invisibility Cloak

138 points by behnamoh 4y ago

bsenftner 4y ago
This makes a lot of sense, and upon reflection is kinda of obvious when one considers how image pyramids and resolution independence works in computer vision.
godelski 4y ago
Anyone else find it weird they used a ML classifier to mask peoples' identities? It doesn't do a good job at constantly masking any of the bystanders. Which their identity is probably more important than the researchers? I totally thought this was going to be about something completely different. But I don't suspect these kinds of methods to work in the long run. Honestly maybe someone can convince me otherwise. These kinds of attacks are always going to be extremely difficult. Your attack (the sweatshirt) changes as you move and walk through different lightings, etc. It just doesn't seem like a good direction for real protection. Plus, a model can always be tuned to correct for the attack. It just seems like the research is more about robustness which it feels like this is just flashy presentation. I can get that tbh, but these always seem like they are being presented as ways people can attack these systems in the real world (I don't buy this). So privacy ML people, can you explain to a CV researcher why this is a useful direction?
- TheMiddleMan 4y ago
  I don't think privacy is the only application. Imagine a bank using an algorithm like this with their security cameras to reduce the number of guards needed on premises. I could imagine thieves targeting their system with something like this. Or maybe I've been watching too many heist movies...
  - Semaphor 4y ago
    I’m pretty sure GP didn’t mean the actual project, but their presentation and the anonymization of it.
    - godelski 4y ago
      I mean the entire premise of wearing something that fools the classifier and not the camera. I see visual attacks as a cat and mouse game except the cat can see you when you move the wrong way or someone shines a light on you. I get that security always has been and always will be a cat and mouse game, but this seems like a path where the cat is always going to have huge advantages.
brentonmallen 4y ago
That’s an interesting way to put on a sweater
- elefanten 4y ago
  Yeah, that's incredibly bizarre. He does it in both scenes right before putting it on, but also once while just handling the sweatshirt in the second scene. Can't come up with a good guess other than some personal or performative idiosyncrasy.
  - TuringTest 4y ago
    Folding the sweater so that the bottom will be close to the neck makes it easy to put the head through the neck. I do the same, although I fold it from the bottom hole, not the neck hole.
    - galangalalgol 4y ago
      Same here. It's like bunching up your socks to put them on. I'm not the only one that does that am I? Are there people that put in milk and then the cereal? I findnthe idea of people wriggling into unfolded sweaters deeply disconcerting somehow. It is like the test in Dune to see if someone is actually human or a collection of instincts.
      - elefanten 4y ago
        But he's doing more than just that. He's flipping it 180 degrees side-to-side, twice, right before bunching it up in the way you two are describing. He holds it facing "forward" (as if he were wearing it correctly), then he flips it 180 to be oriented "backwards" as he bunches it up, and finally flips it 180 degrees back to the "forward" orientation right before putting it on. But that seems to achieve nothing? He could just bunch it up (through the neckhole, from below like he's doing otherwise) in the "forward" orientation to begin with... without the self-negating 180 flips.
durpkingOP 4y ago
wear this sweater, get hit by a tesla, sue elon, roll around in a golden wheelchair forever
- deleted 4y ago
  [deleted]
- dev_snd 4y ago
  Your joke is not as far fetched as one might think! There have been multiple instances [0][1] of a Tesla (and also other car makes, for that matter) where the image recognition system could be tricked. I wonder at which point we'll deem fooling a autonomous car a misdemeanor instead of changing the underlying AI system. [0] https://www.technologyreview.com/2020/02/19/868188/hackers-can-trick-a-tesla-into-accelerating-by-50-miles-per-hour/ https://www.technologyreview.com/2020/02/19/868188/hackers-c... [1] https://www.technologyreview.com/2019/04/01/65915/hackers-trick-teslas-autopilot-into-veering-towards-oncoming-traffic/ https://www.technologyreview.com/2019/04/01/65915/hackers-tr...
  - sllabres 4y ago
    You doesn't need a "ML Invisibility Cloak", it seems as a shopping cart [1] was at least some time ago (2018?) sufficient. [1] "Tesla does not see you if you are pushing a shopping cart" https://www.youtube.com/watch?v=Zw51rMUJFfY https://www.youtube.com/watch?v=Zw51rMUJFfY
    - mmcdermott 4y ago
      More recently, Tesla seems to struggle with motorcyclists sometimes: https://www.youtube.com/watch?v=yRdzIs4FJJg https://www.youtube.com/watch?v=yRdzIs4FJJg
      - salty_biscuits 4y ago
        Too be fair, so do a lot of human drivers. Something about motorbikes makes them difficult to perceive. Not sure if it is car pillar design, not expecting to see them there (a prior of just looking for cars) or just a small frontal area.
popcalc 4y ago
Paper- https://arxiv.org/pdf/1910.14667.pdf https://arxiv.org/pdf/1910.14667.pdf Code- https://github.com/zxwu/adv_cloak https://github.com/zxwu/adv_cloak
heeen2 4y ago
missed opportunity to add a sponsored link to on-demand print this through redbubble or a similar service to fund the chairs coffee stash or something.
generalizations 4y ago
Doesn't look like it totally worked, based on that last image. Pretty good though. I wonder if it'll be possible to come up with something that generalizes to arbitrary models.
- pelorat 4y ago
  No, that's not possible. This approach can only fool models that identifies people as a whole based on a 2D array of pixels. It can't fool models that understand that a person is a composite of other features. If you create models that can detect individual features like eyes, nose, ears, head, arms, hands, legs, neck, fingers, hair and further feed those into other models that can validate how parts belong together for them to form a human (pose estimation, outline detection to 3D mesh mapping), this research would not work at all. This research works only on straightforward single network models where input is a 2D array of pixels, and output is a yes/no.
adnanaga 4y ago
Someone from my graduate program created a full body suit of this exact concept for their thesis last year - https://www.traceyshi.me/watch-it-from-where-they https://www.traceyshi.me/watch-it-from-where-they

138 points by behnamoh 4y ago

This makes a lot of sense, and upon reflection is kinda of obvious when one considers how image pyramids and resolution independence works in computer vision.

godelski 4y ago

Anyone else find it weird they used a ML classifier to mask peoples' identities? It doesn't do a good job at constantly masking any of the bystanders. Which their identity is probably more important than the researchers? I totally thought this was going to be about something completely different. But I don't suspect these kinds of methods to work in the long run. Honestly maybe someone can convince me otherwise. These kinds of attacks are always going to be extremely difficult. Your attack (the sweatshirt) changes as you move and walk through different lightings, etc. It just doesn't seem like a good direction for real protection. Plus, a model can always be tuned to correct for the attack. It just seems like the research is more about robustness which it feels like this is just flashy presentation. I can get that tbh, but these always seem like they are being presented as ways people can attack these systems in the real world (I don't buy this). So privacy ML people, can you explain to a CV researcher why this is a useful direction?

TheMiddleMan 4y ago
I don't think privacy is the only application. Imagine a bank using an algorithm like this with their security cameras to reduce the number of guards needed on premises. I could imagine thieves targeting their system with something like this. Or maybe I've been watching too many heist movies...
- Semaphor 4y ago
  I’m pretty sure GP didn’t mean the actual project, but their presentation and the anonymization of it.
  - godelski 4y ago
    I mean the entire premise of wearing something that fools the classifier and not the camera. I see visual attacks as a cat and mouse game except the cat can see you when you move the wrong way or someone shines a light on you. I get that security always has been and always will be a cat and mouse game, but this seems like a path where the cat is always going to have huge advantages.

brentonmallen 4y ago

That’s an interesting way to put on a sweater

elefanten 4y ago
Yeah, that's incredibly bizarre. He does it in both scenes right before putting it on, but also once while just handling the sweatshirt in the second scene. Can't come up with a good guess other than some personal or performative idiosyncrasy.
- TuringTest 4y ago
  Folding the sweater so that the bottom will be close to the neck makes it easy to put the head through the neck. I do the same, although I fold it from the bottom hole, not the neck hole.
  - galangalalgol 4y ago
    Same here. It's like bunching up your socks to put them on. I'm not the only one that does that am I? Are there people that put in milk and then the cereal? I findnthe idea of people wriggling into unfolded sweaters deeply disconcerting somehow. It is like the test in Dune to see if someone is actually human or a collection of instincts.
    - elefanten 4y ago
      But he's doing more than just that. He's flipping it 180 degrees side-to-side, twice, right before bunching it up in the way you two are describing. He holds it facing "forward" (as if he were wearing it correctly), then he flips it 180 to be oriented "backwards" as he bunches it up, and finally flips it 180 degrees back to the "forward" orientation right before putting it on. But that seems to achieve nothing? He could just bunch it up (through the neckhole, from below like he's doing otherwise) in the "forward" orientation to begin with... without the self-negating 180 flips.

durpkingOP 4y ago

wear this sweater, get hit by a tesla, sue elon, roll around in a golden wheelchair forever

deleted 4y ago
[deleted]
dev_snd 4y ago
Your joke is not as far fetched as one might think! There have been multiple instances [0][1] of a Tesla (and also other car makes, for that matter) where the image recognition system could be tricked. I wonder at which point we'll deem fooling a autonomous car a misdemeanor instead of changing the underlying AI system. [0] https://www.technologyreview.com/2020/02/19/868188/hackers-can-trick-a-tesla-into-accelerating-by-50-miles-per-hour/ https://www.technologyreview.com/2020/02/19/868188/hackers-c... [1] https://www.technologyreview.com/2019/04/01/65915/hackers-trick-teslas-autopilot-into-veering-towards-oncoming-traffic/ https://www.technologyreview.com/2019/04/01/65915/hackers-tr...
- sllabres 4y ago
  You doesn't need a "ML Invisibility Cloak", it seems as a shopping cart [1] was at least some time ago (2018?) sufficient. [1] "Tesla does not see you if you are pushing a shopping cart" https://www.youtube.com/watch?v=Zw51rMUJFfY https://www.youtube.com/watch?v=Zw51rMUJFfY
  - mmcdermott 4y ago
    More recently, Tesla seems to struggle with motorcyclists sometimes: https://www.youtube.com/watch?v=yRdzIs4FJJg https://www.youtube.com/watch?v=yRdzIs4FJJg
    - salty_biscuits 4y ago
      Too be fair, so do a lot of human drivers. Something about motorbikes makes them difficult to perceive. Not sure if it is car pillar design, not expecting to see them there (a prior of just looking for cars) or just a small frontal area.

popcalc 4y ago

Paper- https://arxiv.org/pdf/1910.14667.pdf https://arxiv.org/pdf/1910.14667.pdf Code- https://github.com/zxwu/adv_cloak https://github.com/zxwu/adv_cloak

heeen2 4y ago

missed opportunity to add a sponsored link to on-demand print this through redbubble or a similar service to fund the chairs coffee stash or something.

generalizations 4y ago

Doesn't look like it totally worked, based on that last image. Pretty good though. I wonder if it'll be possible to come up with something that generalizes to arbitrary models.

pelorat 4y ago
No, that's not possible. This approach can only fool models that identifies people as a whole based on a 2D array of pixels. It can't fool models that understand that a person is a composite of other features. If you create models that can detect individual features like eyes, nose, ears, head, arms, hands, legs, neck, fingers, hair and further feed those into other models that can validate how parts belong together for them to form a human (pose estimation, outline detection to 3D mesh mapping), this research would not work at all. This research works only on straightforward single network models where input is a 2D array of pixels, and output is a yes/no.

adnanaga 4y ago

Someone from my graduate program created a full body suit of this exact concept for their thesis last year - https://www.traceyshi.me/watch-it-from-where-they https://www.traceyshi.me/watch-it-from-where-they