12 ms·

Tool can make photos undetectable to facial recognition without ruining them

228 points by 1cvmask 6y ago

airstrike 6y ago
...until someone writes a workaround
- Quarrelsome 6y ago
  everything eventually devolves into war games doesn't it?
  - tremon 6y ago
    If we keep building a more and more adversarial society, yes.
- ReactiveJelly 6y ago
  1. This will become popular 2. People will start posting photos edited with it "We're safe now! Everyone post your faces!" 3. It'll get added to training sets 4. It'll be sunk within 3 years
  - georgeplusplus 6y ago
    If you can defeat something for 3 years, id call that a win.
  - badkitty99 6y ago
    Try 3 minutes
  - danso 6y ago
    3 years is practically an era in information technology.
greggyb 6y ago
I have a couple questions I didn't see addressed in the article. I like this tech, and am curious to learn more. 1. Are the changes detectable? 2. Are the changes reversible? 3. Are the changes deterministic? 1 and 2 go toward whether this can be counteracted in some way. 3 is concerned with whether we end up making an alternate population of photos that can eventually tie back to the same real person.
kube-system 6y ago
How much of the effect is actually just a result of them flipping the image? TinEye, as I understand, doesn’t even use facial recognition.
- SamBam 6y ago
  Yes, the TinEye example doesn't seem like a great metric. TinEye is pretty great -- it can still match images after scaling and resolution changes, which this technique obviously foils (for now) -- but it's not trying to do facial recognition. I think this is simply beating TinEye's "perceptual hashing" technique, mostly just by flipping the picture, and has nothing at all to do with facial recognition. A better test would be to load a few dozen pictures of Biden into Google Photos, name the face, and see if it correctly names their Biden example. I'd bet dollars to doughnuts it can. I don't think this is affecting facial recognition at all. Google Photos can often recognize people in the far background, turned away, and out of focus, so I don't see how bit-flipping a few pixels will affect that. ---- Edit: Indeed, the TinEye result can trivially be achieved by flipping the image, as they do. Link to a search of a flipped photo of Obama it can't find. I contend this has zero application to "facial recognition," which is not what TinEye attempts to do. https://tineye.com/search/b9d6f95346a7636bbd85e1bcfb931a9e8216145b?sort=score&order=desc&page=1 https://tineye.com/search/b9d6f95346a7636bbd85e1bcfb931a9e82...
maurits 6y ago
Stanford cs231n has a little demo illustrating how easy this is, if you know the model weights, called "Fooling images". In short, you backprop to the data to find out which pixels were important for the classification result. Then you ever so slightly modify these, right upon the point the model miss classifies. [1] [1]: https://nbviewer.jupyter.org/github/madalinabuzau/cs231n-convolutional-neural-networks-solutions/blob/master/2017%20Spring%20Assignments/assignment3/NetworkVisualization-PyTorch.ipynb https://nbviewer.jupyter.org/github/madalinabuzau/cs231n-con...
Medicineguy 6y ago
The fight against facial recognition will be a back and forth between the facial recognition itself and algorithms trying to prevent it (like this one). One issue is that parties using facial recognition have a big advantage: They can run new algorithms against old pictures. Tools like these might be useful in some cases, like preventing stalking _right now_, but can't (sadly) guarantee a prevention in any way for the future. I assume this tool will be marketed differently.
- AxesPushPatty 6y ago
  The problem is more in the case that if you have access to the ground truth data, and use such a tool to create images without detectable faces, you can artificially create more testing data. In the end this training data will be more robust, and the underlying method, like any form of CNN, will be able to generalize better.
caster_cp 6y ago
Sounds very interesting! However, I am very curious on whether this works for simple workarounds such as taking a screenshot of the picture or reducing image quality. Since the original and the altered images are visually identical to the human eye, from the limited knowledge I have on the topic, I infer that the changes made by the algorithm are all in the high-frequency side of the spectrum.
sunnyque 6y ago
just tested example picture in tweet by ms azure facial recognition service and it worked pretty well, idk maybe test picture is just marketing crap, wanna see real world example of this tech before believing in it
TeMPOraL 6y ago
A silly thing. As others point out, if it spreads, photos altered by it will quickly become part of the training set, and the method will be defeated. And in general, it's an unwinnable quest: by definition, you cannot have a system that makes photos "undetectable to facial recognition without ruining them", because if another human can recognize you on a photo, then a machine can be taught to do the same.
- kierenj 6y ago
  It's an arms race then, sure. But why not stay in the race?
  - tantalor 6y ago
    Because arms races are pointless. See War Games (1983)
    - danso 6y ago
      But WarGames is a fictional movie about a mutually assured destruction scenario? And even though it was famously watched by President Reagan [0], his response was decidedly not to abide by the philosophy of "the only winning move is not to play". Reagan presided over massive budget increases to keep up and attempt to win the arms race, including the extremely expensive but impotent "Star Wars" program [1]. Unlike "Star Wars", increasing the computational complexity and costs of mass face surveillance is not a decades-to-soon monolith project. [0] https://en.wikipedia.org/wiki/WarGames#Influence https://en.wikipedia.org/wiki/WarGames#Influence [1] https://www.armscontrol.org/act/2004_07-08/Reagan https://www.armscontrol.org/act/2004_07-08/Reagan
      - HPsquared 6y ago
        Regarding the Star Wars program, perhaps the high cost was part of the motivation? (To saddle the opponent with similar costs)
        6y ago
imchillyb 6y ago
For the next several minutes, hours, days, weeks, months, or years. The technology that detects faces LEARNS. The training may take a bit of time, but this obfuscation will last about as long as masks did. The Chinese Government has had over 90% success in detecting faces that are more than 70% obscured. How long before their software finds this 'hiding' trivial to unmask?
SiempreViernes 6y ago
Yandex will not be fooled by the small screenshot of Biden they provide (if you crop out the text), and overall yandex is just much better at reverse image lookup than google or tineye. I think I've seen rumours that the accuracy of the google reverse lookup is artificially lowered because of privacy concerns or something like that though, so it might not simply be superior Russian engineering behind this result.
leecarraher 6y ago
What doesn't defeat AI just makes the GAN smarter
Someone1234 6y ago
I've been meaning to print the EURion constellation[0] pattern (repeating) onto a t-shirt just to see what it might break. But unsurprisingly getting it printed onto a t-shirt is easier said than done due to multiple printers/software/etc directly blocking it. [0] https://en.wikipedia.org/wiki/EURion_constellation https://en.wikipedia.org/wiki/EURion_constellation
- 91aintprime 6y ago
  That is very funny. You could call it double irony: what you expected to happen is exactly what happened, but not in the way you expected it to happen. I guess it's time for some old school analog printing. I'll see if I can whip up a 3d printable stamp
  - 91aintprime 6y ago
    clearance=17; // [50] height=150; // [300] r1=25-17/2; r2=25+17/2; for(xy = [[269,73],[85,170],[237,228],[475,280],[263,487]]){ translate([xy[0],-xy[1],0]) difference(){ cylinder(r=r2, h=clearance*2, center=true); cylinder(r=r1, h=clearance*2+.1, center=true); } hull(){ translate([xy[0],-xy[1],0]) cylinder(r=r2, h=clearance); translate([237,-228,0]) cylinder(r=r2, h=height-clearance); } } https://www.thingiverse.com/thing:4841547 https://www.thingiverse.com/thing:4841547
- durnygbur 6y ago
  I literally just printed it. Copy pasted the image from wiki into Libreoffice Writer, and printed on my monochrome laser printer.
barbazoo 6y ago
> The new DoNotPay app acts as a “robot lawyer,” letting you “sue anyone by pressing a button.” > DoNotPay also offers services that help you renegotiate a cheaper flight or hotel price. > “We are in an AI arms race between good and evil,” Browder added. “We hope to help consumers fight back, protect their privacy, and avoid stalkers on dating apps!” They should add video streaming and messaging, then it would be the only app you ever need \s
- jvanderbot 6y ago
  DoNotPay asks for my bank account information (routing, checking) before even letting me set a password. That's a bit too quick for me, DoNotPay. Let's have dinner first.
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
frereubu 6y ago
This says that it "does not ruin the image," but that's somewhat subjective. Faces are naturally asymmetric, and it looks like they always flip the image, which in many cases will lead to a bit of an uncanny valley feeling on many photos, particularly for people you know well.
philjohn 6y ago
Putting the photo of Biden through PimEyes shows that it finds matches just fine using facial recognition - https://pimeyes.com/en/search/fcfcfcff98181800fefec8187e400048d19ed2d4877878a1 https://pimeyes.com/en/search/fcfcfcff98181800fefec8187e4000...
- oh_sigh 6y ago
  I suspect they only tested this on google image search, which doesn't actually try to do any kind of fancy facial recognition - it just (I believe) drastically downsizes the image, and looks for images that are similar to the downsized image. So flipping the image 180 degrees would be good enough to "trick" google.
hivacruz 6y ago
I use to flip horizontally some of my movie snapshots I feature on my movie quiz game [1]. I do that randomly, not on all images (like the ones who have text on it). It fools Google Images Reverse Search a lot. Not a great solution but at least it slows down the cheaters and since it's a game where answering fast is important, it works nicely. I tried adding noise to fool Google AI but that didn't work at all. [1]: https://twitter.com/whattheshot https://twitter.com/whattheshot
ivanhoe 6y ago
Took both versions of a lady on a yellow background photo, flipped the altered one horizontally and overlapped them in Photoshop. Looking now at the layer difference and I see nothing other than slight outline highlight and some artifacts on the background (that can easily be cropped). The faces are identical, none of the biometrical features has been altered, once you flip the photo back to normal orientation, it's basically the same photo - diff is almost completely black.
- shiftpgdn 6y ago
  You're right. As far as I can tell all they have done is flip the image and then put out a press release that they've created some new technology to beat "AI" without actually doing anything. The average reporter is beyond non-technical and doesn't have the chops to suss out charlatans.
  - potatoman22 6y ago
    Look up adversarial noise. That's a technology that can fool SOTA methods.
    - kevin_thibedeau 6y ago
      For specific models.
    - baq 6y ago
      always wondered why a low pass filter isn't a standard part of the training pipeline?
      - gugagore 6y ago
        The early convolution layers could implement a low-pass filter with the appropriate weights. Presumably the learning algorithm would do so if it were beneficial.
        6y ago
aparsons 6y ago
Cue arms race
phw 6y ago
There's research on the same idea but with a (presumably) sounder method behind it: https://sandlab.cs.uchicago.edu/fawkes/ https://sandlab.cs.uchicago.edu/fawkes/
kolinko 6y ago
I’m flagging this as false news - both me and others pointed out that even the example in the text doesn’t fool even the basic online services for face recognition. https://imgur.com/gallery/5beg8qg https://imgur.com/gallery/5beg8qg
ionwake 6y ago
Is there a service that can give you as much information as possible regarding a photo? I want to know how anonymous my photos are. Thank you.
lupire 6y ago
Article about a picture has no pictures, and link to picture is 404. The whole thing is based on the premise that the inventor doesn't understand what Reverse Image Search is (it's not facial recognition) > But the changes confuse the A.I; even an altered photo of Joe Biden will return zero results in Google/Tineye Reverse Search!
propogandist 6y ago
PR spam
m3kw9 6y ago
Maybe they are just using the fawkes app

228 points by 1cvmask 6y ago

airstrike 6y ago

...until someone writes a workaround

Quarrelsome 6y ago
everything eventually devolves into war games doesn't it?
- tremon 6y ago
  If we keep building a more and more adversarial society, yes.
ReactiveJelly 6y ago
1. This will become popular 2. People will start posting photos edited with it "We're safe now! Everyone post your faces!" 3. It'll get added to training sets 4. It'll be sunk within 3 years
- georgeplusplus 6y ago
  If you can defeat something for 3 years, id call that a win.
- badkitty99 6y ago
  Try 3 minutes
- danso 6y ago
  3 years is practically an era in information technology.

greggyb 6y ago

I have a couple questions I didn't see addressed in the article. I like this tech, and am curious to learn more. 1. Are the changes detectable? 2. Are the changes reversible? 3. Are the changes deterministic? 1 and 2 go toward whether this can be counteracted in some way. 3 is concerned with whether we end up making an alternate population of photos that can eventually tie back to the same real person.

kube-system 6y ago

How much of the effect is actually just a result of them flipping the image? TinEye, as I understand, doesn’t even use facial recognition.

SamBam 6y ago
Yes, the TinEye example doesn't seem like a great metric. TinEye is pretty great -- it can still match images after scaling and resolution changes, which this technique obviously foils (for now) -- but it's not trying to do facial recognition. I think this is simply beating TinEye's "perceptual hashing" technique, mostly just by flipping the picture, and has nothing at all to do with facial recognition. A better test would be to load a few dozen pictures of Biden into Google Photos, name the face, and see if it correctly names their Biden example. I'd bet dollars to doughnuts it can. I don't think this is affecting facial recognition at all. Google Photos can often recognize people in the far background, turned away, and out of focus, so I don't see how bit-flipping a few pixels will affect that. ---- Edit: Indeed, the TinEye result can trivially be achieved by flipping the image, as they do. Link to a search of a flipped photo of Obama it can't find. I contend this has zero application to "facial recognition," which is not what TinEye attempts to do. https://tineye.com/search/b9d6f95346a7636bbd85e1bcfb931a9e8216145b?sort=score&order=desc&page=1 https://tineye.com/search/b9d6f95346a7636bbd85e1bcfb931a9e82...

maurits 6y ago

Stanford cs231n has a little demo illustrating how easy this is, if you know the model weights, called "Fooling images". In short, you backprop to the data to find out which pixels were important for the classification result. Then you ever so slightly modify these, right upon the point the model miss classifies. [1] [1]: https://nbviewer.jupyter.org/github/madalinabuzau/cs231n-convolutional-neural-networks-solutions/blob/master/2017%20Spring%20Assignments/assignment3/NetworkVisualization-PyTorch.ipynb https://nbviewer.jupyter.org/github/madalinabuzau/cs231n-con...

Medicineguy 6y ago

The fight against facial recognition will be a back and forth between the facial recognition itself and algorithms trying to prevent it (like this one). One issue is that parties using facial recognition have a big advantage: They can run new algorithms against old pictures. Tools like these might be useful in some cases, like preventing stalking _right now_, but can't (sadly) guarantee a prevention in any way for the future. I assume this tool will be marketed differently.

AxesPushPatty 6y ago
The problem is more in the case that if you have access to the ground truth data, and use such a tool to create images without detectable faces, you can artificially create more testing data. In the end this training data will be more robust, and the underlying method, like any form of CNN, will be able to generalize better.

caster_cp 6y ago

Sounds very interesting! However, I am very curious on whether this works for simple workarounds such as taking a screenshot of the picture or reducing image quality. Since the original and the altered images are visually identical to the human eye, from the limited knowledge I have on the topic, I infer that the changes made by the algorithm are all in the high-frequency side of the spectrum.

sunnyque 6y ago

just tested example picture in tweet by ms azure facial recognition service and it worked pretty well, idk maybe test picture is just marketing crap, wanna see real world example of this tech before believing in it

TeMPOraL 6y ago

A silly thing. As others point out, if it spreads, photos altered by it will quickly become part of the training set, and the method will be defeated. And in general, it's an unwinnable quest: by definition, you cannot have a system that makes photos "undetectable to facial recognition without ruining them", because if another human can recognize you on a photo, then a machine can be taught to do the same.

kierenj 6y ago
It's an arms race then, sure. But why not stay in the race?
- tantalor 6y ago
  Because arms races are pointless. See War Games (1983)
  - danso 6y ago
    But WarGames is a fictional movie about a mutually assured destruction scenario? And even though it was famously watched by President Reagan [0], his response was decidedly not to abide by the philosophy of "the only winning move is not to play". Reagan presided over massive budget increases to keep up and attempt to win the arms race, including the extremely expensive but impotent "Star Wars" program [1]. Unlike "Star Wars", increasing the computational complexity and costs of mass face surveillance is not a decades-to-soon monolith project. [0] https://en.wikipedia.org/wiki/WarGames#Influence https://en.wikipedia.org/wiki/WarGames#Influence [1] https://www.armscontrol.org/act/2004_07-08/Reagan https://www.armscontrol.org/act/2004_07-08/Reagan
    - HPsquared 6y ago
      Regarding the Star Wars program, perhaps the high cost was part of the motivation? (To saddle the opponent with similar costs)
      - 6y ago

imchillyb 6y ago

For the next several minutes, hours, days, weeks, months, or years. The technology that detects faces LEARNS. The training may take a bit of time, but this obfuscation will last about as long as masks did. The Chinese Government has had over 90% success in detecting faces that are more than 70% obscured. How long before their software finds this 'hiding' trivial to unmask?

SiempreViernes 6y ago

Yandex will not be fooled by the small screenshot of Biden they provide (if you crop out the text), and overall yandex is just much better at reverse image lookup than google or tineye. I think I've seen rumours that the accuracy of the google reverse lookup is artificially lowered because of privacy concerns or something like that though, so it might not simply be superior Russian engineering behind this result.

leecarraher 6y ago

What doesn't defeat AI just makes the GAN smarter

Someone1234 6y ago

I've been meaning to print the EURion constellation[0] pattern (repeating) onto a t-shirt just to see what it might break. But unsurprisingly getting it printed onto a t-shirt is easier said than done due to multiple printers/software/etc directly blocking it. [0] https://en.wikipedia.org/wiki/EURion_constellation https://en.wikipedia.org/wiki/EURion_constellation

91aintprime 6y ago
That is very funny. You could call it double irony: what you expected to happen is exactly what happened, but not in the way you expected it to happen. I guess it's time for some old school analog printing. I'll see if I can whip up a 3d printable stamp
- 91aintprime 6y ago
  clearance=17; // [50] height=150; // [300] r1=25-17/2; r2=25+17/2; for(xy = [[269,73],[85,170],[237,228],[475,280],[263,487]]){ translate([xy[0],-xy[1],0]) difference(){ cylinder(r=r2, h=clearance*2, center=true); cylinder(r=r1, h=clearance*2+.1, center=true); } hull(){ translate([xy[0],-xy[1],0]) cylinder(r=r2, h=clearance); translate([237,-228,0]) cylinder(r=r2, h=height-clearance); } } https://www.thingiverse.com/thing:4841547 https://www.thingiverse.com/thing:4841547
durnygbur 6y ago
I literally just printed it. Copy pasted the image from wiki into Libreoffice Writer, and printed on my monochrome laser printer.

barbazoo 6y ago

> The new DoNotPay app acts as a “robot lawyer,” letting you “sue anyone by pressing a button.” > DoNotPay also offers services that help you renegotiate a cheaper flight or hotel price. > “We are in an AI arms race between good and evil,” Browder added. “We hope to help consumers fight back, protect their privacy, and avoid stalkers on dating apps!” They should add video streaming and messaging, then it would be the only app you ever need \s

jvanderbot 6y ago
DoNotPay asks for my bank account information (routing, checking) before even letting me set a password. That's a bit too quick for me, DoNotPay. Let's have dinner first.

deleted 6y ago

[deleted]

deleted 6y ago

[deleted]

frereubu 6y ago

This says that it "does not ruin the image," but that's somewhat subjective. Faces are naturally asymmetric, and it looks like they always flip the image, which in many cases will lead to a bit of an uncanny valley feeling on many photos, particularly for people you know well.

philjohn 6y ago

Putting the photo of Biden through PimEyes shows that it finds matches just fine using facial recognition - https://pimeyes.com/en/search/fcfcfcff98181800fefec8187e400048d19ed2d4877878a1 https://pimeyes.com/en/search/fcfcfcff98181800fefec8187e4000...

oh_sigh 6y ago
I suspect they only tested this on google image search, which doesn't actually try to do any kind of fancy facial recognition - it just (I believe) drastically downsizes the image, and looks for images that are similar to the downsized image. So flipping the image 180 degrees would be good enough to "trick" google.

hivacruz 6y ago

I use to flip horizontally some of my movie snapshots I feature on my movie quiz game [1]. I do that randomly, not on all images (like the ones who have text on it). It fools Google Images Reverse Search a lot. Not a great solution but at least it slows down the cheaters and since it's a game where answering fast is important, it works nicely. I tried adding noise to fool Google AI but that didn't work at all. [1]: https://twitter.com/whattheshot https://twitter.com/whattheshot

ivanhoe 6y ago

Took both versions of a lady on a yellow background photo, flipped the altered one horizontally and overlapped them in Photoshop. Looking now at the layer difference and I see nothing other than slight outline highlight and some artifacts on the background (that can easily be cropped). The faces are identical, none of the biometrical features has been altered, once you flip the photo back to normal orientation, it's basically the same photo - diff is almost completely black.

shiftpgdn 6y ago
You're right. As far as I can tell all they have done is flip the image and then put out a press release that they've created some new technology to beat "AI" without actually doing anything. The average reporter is beyond non-technical and doesn't have the chops to suss out charlatans.
- potatoman22 6y ago
  Look up adversarial noise. That's a technology that can fool SOTA methods.
  - kevin_thibedeau 6y ago
    For specific models.
  - baq 6y ago
    always wondered why a low pass filter isn't a standard part of the training pipeline?
    - gugagore 6y ago
      The early convolution layers could implement a low-pass filter with the appropriate weights. Presumably the learning algorithm would do so if it were beneficial.
      - 6y ago

aparsons 6y ago

Cue arms race

phw 6y ago

There's research on the same idea but with a (presumably) sounder method behind it: https://sandlab.cs.uchicago.edu/fawkes/ https://sandlab.cs.uchicago.edu/fawkes/

kolinko 6y ago

I’m flagging this as false news - both me and others pointed out that even the example in the text doesn’t fool even the basic online services for face recognition. https://imgur.com/gallery/5beg8qg https://imgur.com/gallery/5beg8qg

ionwake 6y ago

Is there a service that can give you as much information as possible regarding a photo? I want to know how anonymous my photos are. Thank you.

lupire 6y ago

Article about a picture has no pictures, and link to picture is 404. The whole thing is based on the premise that the inventor doesn't understand what Reverse Image Search is (it's not facial recognition) > But the changes confuse the A.I; even an altered photo of Joe Biden will return zero results in Google/Tineye Reverse Search!

propogandist 6y ago

PR spam

m3kw9 6y ago

Maybe they are just using the fawkes app