5 ms·

From the linked article: Q: Can't you just apply some filter, or compression, or blurring algorithm, or add some noise to the image to destroy image cloaks? A

by woah 6y ago

From the linked article: Q: Can't you just apply some filter, or compression, or blurring algorithm, or add some noise to the image to destroy image cloaks? A: As counterintuitive as this may be, the high level answer is no simple tools work to destroy the perturbation that form image cloaks. To make sense of this, it helps to first understand that Fawkes does not use high-intensity pixels, or rely on bright patterns to distort the classification value of the image in the feature space. It is a precisely computed combination of a number of pixels that do not easily stand out, that produce the distortion in the feature space. If you're interested in seeing some details, we encourage you to take a look at the technical paper (also linked above). In it we present detailed experimental results showing how robust Fawkes is to things like image compression and distortion/noise injection. The quick takeaway is that as you increase the magnitude of these noisy disruptions to the image, protection of image cloaking does fall, but slower than normal image classification accuracy. Translated: Yes, it is possible to add noise and distortions at a high enough level to distort image cloaks. But such distortions will hurt normal classification far more and faster. By the time a distortion is large enough to break cloaking, it has already broken normal image classification and made the image useless for facial recognition.

akersten 6y ago
It was pretty frustrating that they did not readily offer any example images for inspection, so against my better judgement I downloaded their binaries to run some experiements. First, a source image at an approximate resolution that you might find on a social networking site: https://imgur.com/a/9szcC1m https://imgur.com/a/9szcC1m Text output of the tool, which ran for about 3 minutes: https://imgur.com/a/fZtfrmm https://imgur.com/a/fZtfrmm The resulting cloaked image: https://imgur.com/a/OSHXdbO https://imgur.com/a/OSHXdbO I applied a difference filter between the two images in Photoshop, to show an example of the actual pertubations performed: https://imgur.com/a/q4zC7Ms https://imgur.com/a/q4zC7Ms Since it's hard to see, I compressed the output to highlight what the program actually changed. It does seem like there is a good amount of disturbance to the image: https://imgur.com/a/1Sx68o3 https://imgur.com/a/1Sx68o3 Now, the real test. First, a Google reverse image search for the original file - identification is pretty bang-on: https://imgur.com/a/5HJwjPx https://imgur.com/a/5HJwjPx A Google reverse image search for the cloaked file: https://imgur.com/a/QByXBfS https://imgur.com/a/QByXBfS The only difference I'm seeing is a few images that are one or two images swapped in the "visually similar images" category. So, I figured that that's the "best case" for the cloaked image - giving the search algorithm the full, unfiltered data, and the program still failed to disguise it. For fun, I thought I would use a "low-pass filter" (Google Lens pointed at my computer screen) as well, just for thoroughness. And the result surprised me! Here's Google Lens pointed at my screen with the original image open: https://imgur.com/a/1BVRFG0 https://imgur.com/a/1BVRFG0 And here's Google Lens pointed at my screen with the cloaked image open: https://imgur.com/a/uoppuit https://imgur.com/a/uoppuit So, it would seem that the algorithm's distortions more effectively come through in worse quality images! But, based on my full-resolution result, I wouldn't trust it to disguise something that is being directly uploaded to a social network. Now, one important note is that reverse image search is probably not using a facial recognition model, but more like image chunk hashing - although I would also consider that something a privacy tool should defend against, which is why I included it. All in all, very interesting and thanks for convincing me that I should actually test it out.
- Bjartr 6y ago
  There's also a chance Google image search is looking at the filename of the image to get a bit more context. Does the reverse image search of the cloaked image still work if you rename it something unrelated?
  - akersten 6y ago
    Fair shake, this time I cropped the image too so that it wasn't looking at any of the boundary to identify it either. Here's the original image: https://imgur.com/a/Td4rhoy https://imgur.com/a/Td4rhoy And the cloaked: https://imgur.com/a/cPCiCZo https://imgur.com/a/cPCiCZo These were both saved as JPG with compression level 8/12. I searched for the cloaked crop (96.jpg) first this time: https://imgur.com/a/FSehQWO https://imgur.com/a/FSehQWO And the original crop (10.jpg) next: https://imgur.com/a/yx4jF0B https://imgur.com/a/yx4jF0B This time, Google reverse image search did better at identifying the name of the singer in the cloaked image, instead of just giving the band name for the uncloaked. Not super scientific since we don't really know what's going on behind-the-scenes with Google reverse image search, but it's certainly one adversary that doesn't seem to be easily fooled if there are other images of "you" out there for it to find. I also tried these small crops in Google Lens with less success (I got unrelated portraits for both images, cloaked or not).
    - hadeson 6y ago
      Yandex seem to give even more precise matches than Google. Maybe image of someone less popular would work better.
    - cookiecaper 6y ago
      The goal, afaik, is to stop facial detection software from learning to recognize you and put a face to a name, not to frustrate visual similarity searches. The images are supposed to be visually similar -- so similar that they're indistinguishable to a human viewer. Surveillance software that purports to accurately identify a person across multiple images is not just looking for the same content with some visually insignificant modifications. It's reading your facial structure, attaching your name to it, and searching for it in every image received. Fawkes is working to defeat that specific use case, not all fuzzy matchers in general. P.S. If you have a human assailant running a reverse image search for photos of you, I think you're well past the point that something like this could be expected to help.

by woah 6y ago

akersten 6y ago

It was pretty frustrating that they did not readily offer any example images for inspection, so against my better judgement I downloaded their binaries to run some experiements. First, a source image at an approximate resolution that you might find on a social networking site: https://imgur.com/a/9szcC1m https://imgur.com/a/9szcC1m Text output of the tool, which ran for about 3 minutes: https://imgur.com/a/fZtfrmm https://imgur.com/a/fZtfrmm The resulting cloaked image: https://imgur.com/a/OSHXdbO https://imgur.com/a/OSHXdbO I applied a difference filter between the two images in Photoshop, to show an example of the actual pertubations performed: https://imgur.com/a/q4zC7Ms https://imgur.com/a/q4zC7Ms Since it's hard to see, I compressed the output to highlight what the program actually changed. It does seem like there is a good amount of disturbance to the image: https://imgur.com/a/1Sx68o3 https://imgur.com/a/1Sx68o3 Now, the real test. First, a Google reverse image search for the original file - identification is pretty bang-on: https://imgur.com/a/5HJwjPx https://imgur.com/a/5HJwjPx A Google reverse image search for the cloaked file: https://imgur.com/a/QByXBfS https://imgur.com/a/QByXBfS The only difference I'm seeing is a few images that are one or two images swapped in the "visually similar images" category. So, I figured that that's the "best case" for the cloaked image - giving the search algorithm the full, unfiltered data, and the program still failed to disguise it. For fun, I thought I would use a "low-pass filter" (Google Lens pointed at my computer screen) as well, just for thoroughness. And the result surprised me! Here's Google Lens pointed at my screen with the original image open: https://imgur.com/a/1BVRFG0 https://imgur.com/a/1BVRFG0 And here's Google Lens pointed at my screen with the cloaked image open: https://imgur.com/a/uoppuit https://imgur.com/a/uoppuit So, it would seem that the algorithm's distortions more effectively come through in worse quality images! But, based on my full-resolution result, I wouldn't trust it to disguise something that is being directly uploaded to a social network. Now, one important note is that reverse image search is probably not using a facial recognition model, but more like image chunk hashing - although I would also consider that something a privacy tool should defend against, which is why I included it. All in all, very interesting and thanks for convincing me that I should actually test it out.

Bjartr 6y ago
There's also a chance Google image search is looking at the filename of the image to get a bit more context. Does the reverse image search of the cloaked image still work if you rename it something unrelated?
- akersten 6y ago
  Fair shake, this time I cropped the image too so that it wasn't looking at any of the boundary to identify it either. Here's the original image: https://imgur.com/a/Td4rhoy https://imgur.com/a/Td4rhoy And the cloaked: https://imgur.com/a/cPCiCZo https://imgur.com/a/cPCiCZo These were both saved as JPG with compression level 8/12. I searched for the cloaked crop (96.jpg) first this time: https://imgur.com/a/FSehQWO https://imgur.com/a/FSehQWO And the original crop (10.jpg) next: https://imgur.com/a/yx4jF0B https://imgur.com/a/yx4jF0B This time, Google reverse image search did better at identifying the name of the singer in the cloaked image, instead of just giving the band name for the uncloaked. Not super scientific since we don't really know what's going on behind-the-scenes with Google reverse image search, but it's certainly one adversary that doesn't seem to be easily fooled if there are other images of "you" out there for it to find. I also tried these small crops in Google Lens with less success (I got unrelated portraits for both images, cloaked or not).
  - hadeson 6y ago
    Yandex seem to give even more precise matches than Google. Maybe image of someone less popular would work better.
  - cookiecaper 6y ago
    The goal, afaik, is to stop facial detection software from learning to recognize you and put a face to a name, not to frustrate visual similarity searches. The images are supposed to be visually similar -- so similar that they're indistinguishable to a human viewer. Surveillance software that purports to accurately identify a person across multiple images is not just looking for the same content with some visually insignificant modifications. It's reading your facial structure, attaching your name to it, and searching for it in every image received. Fawkes is working to defeat that specific use case, not all fuzzy matchers in general. P.S. If you have a human assailant running a reverse image search for photos of you, I think you're well past the point that something like this could be expected to help.