4 ms·

It was pretty frustrating that they did not readily offer any example images for inspection, so against my better judgement I downloaded their binaries to run s

It was pretty frustrating that they did not readily offer any example images for inspection, so against my better judgement I downloaded their binaries to run some experiements. First, a source image at an approximate resolution that you might find on a social networking site: https://imgur.com/a/9szcC1m https://imgur.com/a/9szcC1m Text output of the tool, which ran for about 3 minutes: https://imgur.com/a/fZtfrmm https://imgur.com/a/fZtfrmm The resulting cloaked image: https://imgur.com/a/OSHXdbO https://imgur.com/a/OSHXdbO I applied a difference filter between the two images in Photoshop, to show an example of the actual pertubations performed: https://imgur.com/a/q4zC7Ms https://imgur.com/a/q4zC7Ms Since it's hard to see, I compressed the output to highlight what the program actually changed. It does seem like there is a good amount of disturbance to the image: https://imgur.com/a/1Sx68o3 https://imgur.com/a/1Sx68o3 Now, the real test. First, a Google reverse image search for the original file - identification is pretty bang-on: https://imgur.com/a/5HJwjPx https://imgur.com/a/5HJwjPx A Google reverse image search for the cloaked file: https://imgur.com/a/QByXBfS https://imgur.com/a/QByXBfS The only difference I'm seeing is a few images that are one or two images swapped in the "visually similar images" category. So, I figured that that's the "best case" for the cloaked image - giving the search algorithm the full, unfiltered data, and the program still failed to disguise it. For fun, I thought I would use a "low-pass filter" (Google Lens pointed at my computer screen) as well, just for thoroughness. And the result surprised me! Here's Google Lens pointed at my screen with the original image open: https://imgur.com/a/1BVRFG0 https://imgur.com/a/1BVRFG0 And here's Google Lens pointed at my screen with the cloaked image open: https://imgur.com/a/uoppuit https://imgur.com/a/uoppuit So, it would seem that the algorithm's distortions more effectively come through in worse quality images! But, based on my full-resolution result, I wouldn't trust it to disguise something that is being directly uploaded to a social network. Now, one important note is that reverse image search is probably not using a facial recognition model, but more like image chunk hashing - although I would also consider that something a privacy tool should defend against, which is why I included it. All in all, very interesting and thanks for convincing me that I should actually test it out.

Bjartr 6y ago
There's also a chance Google image search is looking at the filename of the image to get a bit more context. Does the reverse image search of the cloaked image still work if you rename it something unrelated?
- akersten 6y ago
  Fair shake, this time I cropped the image too so that it wasn't looking at any of the boundary to identify it either. Here's the original image: https://imgur.com/a/Td4rhoy https://imgur.com/a/Td4rhoy And the cloaked: https://imgur.com/a/cPCiCZo https://imgur.com/a/cPCiCZo These were both saved as JPG with compression level 8/12. I searched for the cloaked crop (96.jpg) first this time: https://imgur.com/a/FSehQWO https://imgur.com/a/FSehQWO And the original crop (10.jpg) next: https://imgur.com/a/yx4jF0B https://imgur.com/a/yx4jF0B This time, Google reverse image search did better at identifying the name of the singer in the cloaked image, instead of just giving the band name for the uncloaked. Not super scientific since we don't really know what's going on behind-the-scenes with Google reverse image search, but it's certainly one adversary that doesn't seem to be easily fooled if there are other images of "you" out there for it to find. I also tried these small crops in Google Lens with less success (I got unrelated portraits for both images, cloaked or not).
  - hadeson 6y ago
    Yandex seem to give even more precise matches than Google. Maybe image of someone less popular would work better.
  - cookiecaper 6y ago
    The goal, afaik, is to stop facial detection software from learning to recognize you and put a face to a name, not to frustrate visual similarity searches. The images are supposed to be visually similar -- so similar that they're indistinguishable to a human viewer. Surveillance software that purports to accurately identify a person across multiple images is not just looking for the same content with some visually insignificant modifications. It's reading your facial structure, attaching your name to it, and searching for it in every image received. Fawkes is working to defeat that specific use case, not all fuzzy matchers in general. P.S. If you have a human assailant running a reverse image search for photos of you, I think you're well past the point that something like this could be expected to help.
kevingadd 6y ago
GIS is specifically designed to be good at finding similar images so it's going to work great for your test case. Facial recognition algorithms are solving a different problem. If they were promising that cloaking would work well on GIS, that'd be a different matter. I can imagine wanting your images to not show up on GIS (because people would use them to try and find the source image on your profile, or something) but it's a different set of constraints at that point. For cloaking a big use case would be "I took a selfie with a friend and want to share it on my instagram" and your goal is for that instagram selfie to not automatically connect with, for example, a surveillance photo of you at a protest. GIS is obviously not relevant to that scenario.
throwaway189262 6y ago
You're not testing against facial recognition models. "similar image" search generally uses "perceptual hashing" which is not related to facial recognition, and not really a privacy risk. You're not testing against what this tool was designed to defeat :-/
im3w1l 6y ago
When I look at the cloaked image it seems that the tool is doing something legit. It's actually changing the shape of the face ever so slightly, rather than playing tricks with noise. I echo the sentiment of other posters regarding reverse-image search. The original image should not be available to match against. That would be operator failure.

Bjartr 6y ago

There's also a chance Google image search is looking at the filename of the image to get a bit more context. Does the reverse image search of the cloaked image still work if you rename it something unrelated?

akersten 6y ago
Fair shake, this time I cropped the image too so that it wasn't looking at any of the boundary to identify it either. Here's the original image: https://imgur.com/a/Td4rhoy https://imgur.com/a/Td4rhoy And the cloaked: https://imgur.com/a/cPCiCZo https://imgur.com/a/cPCiCZo These were both saved as JPG with compression level 8/12. I searched for the cloaked crop (96.jpg) first this time: https://imgur.com/a/FSehQWO https://imgur.com/a/FSehQWO And the original crop (10.jpg) next: https://imgur.com/a/yx4jF0B https://imgur.com/a/yx4jF0B This time, Google reverse image search did better at identifying the name of the singer in the cloaked image, instead of just giving the band name for the uncloaked. Not super scientific since we don't really know what's going on behind-the-scenes with Google reverse image search, but it's certainly one adversary that doesn't seem to be easily fooled if there are other images of "you" out there for it to find. I also tried these small crops in Google Lens with less success (I got unrelated portraits for both images, cloaked or not).
- hadeson 6y ago
  Yandex seem to give even more precise matches than Google. Maybe image of someone less popular would work better.
- cookiecaper 6y ago
  The goal, afaik, is to stop facial detection software from learning to recognize you and put a face to a name, not to frustrate visual similarity searches. The images are supposed to be visually similar -- so similar that they're indistinguishable to a human viewer. Surveillance software that purports to accurately identify a person across multiple images is not just looking for the same content with some visually insignificant modifications. It's reading your facial structure, attaching your name to it, and searching for it in every image received. Fawkes is working to defeat that specific use case, not all fuzzy matchers in general. P.S. If you have a human assailant running a reverse image search for photos of you, I think you're well past the point that something like this could be expected to help.

kevingadd 6y ago

GIS is specifically designed to be good at finding similar images so it's going to work great for your test case. Facial recognition algorithms are solving a different problem. If they were promising that cloaking would work well on GIS, that'd be a different matter. I can imagine wanting your images to not show up on GIS (because people would use them to try and find the source image on your profile, or something) but it's a different set of constraints at that point. For cloaking a big use case would be "I took a selfie with a friend and want to share it on my instagram" and your goal is for that instagram selfie to not automatically connect with, for example, a surveillance photo of you at a protest. GIS is obviously not relevant to that scenario.

throwaway189262 6y ago

You're not testing against facial recognition models. "similar image" search generally uses "perceptual hashing" which is not related to facial recognition, and not really a privacy risk. You're not testing against what this tool was designed to defeat :-/

im3w1l 6y ago

When I look at the cloaked image it seems that the tool is doing something legit. It's actually changing the shape of the face ever so slightly, rather than playing tricks with noise. I echo the sentiment of other posters regarding reverse-image search. The original image should not be available to match against. That would be operator failure.