5 ms·

Color me extremely skeptical. A low-pass filter will make short work of any "tiny, pixel-level" changes designed to thwart ML. After all, one of the most tell-t

Color me extremely skeptical. A low-pass filter will make short work of any "tiny, pixel-level" changes designed to thwart ML. After all, one of the most tell-tale identifiers (space between eyes/nose/mouth) is still plainly observable and unaltered in the "cloaked" image. If a human's neural network can correctly correlate the before/after examples, so can a computer's. They might have found an issue with some modern implementations of facial recognition, sure. But it's a false sense of security to claim "when someone tries to identify you using an unaltered image of you [...] they will fail."

calvinmorrison 6y ago
Or more simply, what if you just take a real life photo of the image and scan it back in and use the same classification techniques?
- akersten 6y ago
  Reproducing the image in this way is essentially a manual low-pass filter (although with little control over the parameters), so it's certainly one valid data point with which to test the hypothesis.
- dijksterhuis 6y ago
  This can be thwarted fairly easily https://arxiv.org/pdf/1607.02533.pdf https://arxiv.org/pdf/1607.02533.pdf
skylanh 6y ago
This was my first impression as well, except with less knowledge on the subject. Someone, somewhere said "huh...", and placed another filter into the pipeline to handle these types of images. While looking for something to sound smart wrt the tank-training myth, I found this interesting page: https://www.gwern.net/Tanks https://www.gwern.net/Tanks "The Neural Net Tank Urban Legend" And interestingly, looking at the link for "superresolution needing learned downscalers" found this: https://arxiv.org/abs/1907.12904 https://arxiv.org/abs/1907.12904 "Learned Image Downscaling for Upscaling using Content Adaptive Resampler", code available at https://github.com/sunwj/CAR https://github.com/sunwj/CAR So, IDK, seems like this Fawkes approach will be an interesting paper.
DabbyDabberson 6y ago
> If a human's neural network can correctly correlate the before/after examples, so can a computer's. color _me_ skeptical, but this is like saying we have functioning AGI; that artificial NNs are the same as the ones we have in our skulls. This to me, is an effect of the over-anthropomorphization of machine learning. It's a bad intuition to have. However, I do agree. This is just one step in an arms race, and one iteration from being worthless.
- akersten 6y ago
  That bit was more of a forward-looking statement about the future capabilities of image recognition, but yes it is somewhat hyperbolic in the general case. I don't believe we'll ever achieve AGI, but I do believe we'll have super reliable application-specific classifiers that vastly outperform humans and won't be fooled by tricks like this.
  - azernik 6y ago
    Machine vision will likely be one of the last of these, though; the visual cortex is a pretty large chunk of very sophisticated hardware.
dijksterhuis 6y ago
Applying some form of transformation is a weak countermeasure to adversarial examples. Especially simple ones such as low pass filters. https://arxiv.org/pdf/1707.07397.pdf https://arxiv.org/pdf/1707.07397.pdf Neural Networks do not learn what humans learn. They can learn completely different and sometimes much smaller features.
woah 6y ago
From the linked article: Q: Can't you just apply some filter, or compression, or blurring algorithm, or add some noise to the image to destroy image cloaks? A: As counterintuitive as this may be, the high level answer is no simple tools work to destroy the perturbation that form image cloaks. To make sense of this, it helps to first understand that Fawkes does not use high-intensity pixels, or rely on bright patterns to distort the classification value of the image in the feature space. It is a precisely computed combination of a number of pixels that do not easily stand out, that produce the distortion in the feature space. If you're interested in seeing some details, we encourage you to take a look at the technical paper (also linked above). In it we present detailed experimental results showing how robust Fawkes is to things like image compression and distortion/noise injection. The quick takeaway is that as you increase the magnitude of these noisy disruptions to the image, protection of image cloaking does fall, but slower than normal image classification accuracy. Translated: Yes, it is possible to add noise and distortions at a high enough level to distort image cloaks. But such distortions will hurt normal classification far more and faster. By the time a distortion is large enough to break cloaking, it has already broken normal image classification and made the image useless for facial recognition.
- akersten 6y ago
  It was pretty frustrating that they did not readily offer any example images for inspection, so against my better judgement I downloaded their binaries to run some experiements. First, a source image at an approximate resolution that you might find on a social networking site: https://imgur.com/a/9szcC1m https://imgur.com/a/9szcC1m Text output of the tool, which ran for about 3 minutes: https://imgur.com/a/fZtfrmm https://imgur.com/a/fZtfrmm The resulting cloaked image: https://imgur.com/a/OSHXdbO https://imgur.com/a/OSHXdbO I applied a difference filter between the two images in Photoshop, to show an example of the actual pertubations performed: https://imgur.com/a/q4zC7Ms https://imgur.com/a/q4zC7Ms Since it's hard to see, I compressed the output to highlight what the program actually changed. It does seem like there is a good amount of disturbance to the image: https://imgur.com/a/1Sx68o3 https://imgur.com/a/1Sx68o3 Now, the real test. First, a Google reverse image search for the original file - identification is pretty bang-on: https://imgur.com/a/5HJwjPx https://imgur.com/a/5HJwjPx A Google reverse image search for the cloaked file: https://imgur.com/a/QByXBfS https://imgur.com/a/QByXBfS The only difference I'm seeing is a few images that are one or two images swapped in the "visually similar images" category. So, I figured that that's the "best case" for the cloaked image - giving the search algorithm the full, unfiltered data, and the program still failed to disguise it. For fun, I thought I would use a "low-pass filter" (Google Lens pointed at my computer screen) as well, just for thoroughness. And the result surprised me! Here's Google Lens pointed at my screen with the original image open: https://imgur.com/a/1BVRFG0 https://imgur.com/a/1BVRFG0 And here's Google Lens pointed at my screen with the cloaked image open: https://imgur.com/a/uoppuit https://imgur.com/a/uoppuit So, it would seem that the algorithm's distortions more effectively come through in worse quality images! But, based on my full-resolution result, I wouldn't trust it to disguise something that is being directly uploaded to a social network. Now, one important note is that reverse image search is probably not using a facial recognition model, but more like image chunk hashing - although I would also consider that something a privacy tool should defend against, which is why I included it. All in all, very interesting and thanks for convincing me that I should actually test it out.
raxxorrax 6y ago
Haven't read the paper yet but sure will do. I wonder how it works, that so many face rec implementations are fooled but I don't know how similar they work. I would have guessed they know which features are extracted and modify the relevant regions. I guess that face rec software will quickly adapt though. That said, we have invisible watermarks that are very resistant to compression or other filters.

calvinmorrison 6y ago

Or more simply, what if you just take a real life photo of the image and scan it back in and use the same classification techniques?

akersten 6y ago
Reproducing the image in this way is essentially a manual low-pass filter (although with little control over the parameters), so it's certainly one valid data point with which to test the hypothesis.
dijksterhuis 6y ago
This can be thwarted fairly easily https://arxiv.org/pdf/1607.02533.pdf https://arxiv.org/pdf/1607.02533.pdf

skylanh 6y ago

This was my first impression as well, except with less knowledge on the subject. Someone, somewhere said "huh...", and placed another filter into the pipeline to handle these types of images. While looking for something to sound smart wrt the tank-training myth, I found this interesting page: https://www.gwern.net/Tanks https://www.gwern.net/Tanks "The Neural Net Tank Urban Legend" And interestingly, looking at the link for "superresolution needing learned downscalers" found this: https://arxiv.org/abs/1907.12904 https://arxiv.org/abs/1907.12904 "Learned Image Downscaling for Upscaling using Content Adaptive Resampler", code available at https://github.com/sunwj/CAR https://github.com/sunwj/CAR So, IDK, seems like this Fawkes approach will be an interesting paper.

DabbyDabberson 6y ago

> If a human's neural network can correctly correlate the before/after examples, so can a computer's. color _me_ skeptical, but this is like saying we have functioning AGI; that artificial NNs are the same as the ones we have in our skulls. This to me, is an effect of the over-anthropomorphization of machine learning. It's a bad intuition to have. However, I do agree. This is just one step in an arms race, and one iteration from being worthless.

akersten 6y ago
That bit was more of a forward-looking statement about the future capabilities of image recognition, but yes it is somewhat hyperbolic in the general case. I don't believe we'll ever achieve AGI, but I do believe we'll have super reliable application-specific classifiers that vastly outperform humans and won't be fooled by tricks like this.
- azernik 6y ago
  Machine vision will likely be one of the last of these, though; the visual cortex is a pretty large chunk of very sophisticated hardware.

dijksterhuis 6y ago

Applying some form of transformation is a weak countermeasure to adversarial examples. Especially simple ones such as low pass filters. https://arxiv.org/pdf/1707.07397.pdf https://arxiv.org/pdf/1707.07397.pdf Neural Networks do not learn what humans learn. They can learn completely different and sometimes much smaller features.

woah 6y ago

From the linked article: Q: Can't you just apply some filter, or compression, or blurring algorithm, or add some noise to the image to destroy image cloaks? A: As counterintuitive as this may be, the high level answer is no simple tools work to destroy the perturbation that form image cloaks. To make sense of this, it helps to first understand that Fawkes does not use high-intensity pixels, or rely on bright patterns to distort the classification value of the image in the feature space. It is a precisely computed combination of a number of pixels that do not easily stand out, that produce the distortion in the feature space. If you're interested in seeing some details, we encourage you to take a look at the technical paper (also linked above). In it we present detailed experimental results showing how robust Fawkes is to things like image compression and distortion/noise injection. The quick takeaway is that as you increase the magnitude of these noisy disruptions to the image, protection of image cloaking does fall, but slower than normal image classification accuracy. Translated: Yes, it is possible to add noise and distortions at a high enough level to distort image cloaks. But such distortions will hurt normal classification far more and faster. By the time a distortion is large enough to break cloaking, it has already broken normal image classification and made the image useless for facial recognition.

akersten 6y ago
It was pretty frustrating that they did not readily offer any example images for inspection, so against my better judgement I downloaded their binaries to run some experiements. First, a source image at an approximate resolution that you might find on a social networking site: https://imgur.com/a/9szcC1m https://imgur.com/a/9szcC1m Text output of the tool, which ran for about 3 minutes: https://imgur.com/a/fZtfrmm https://imgur.com/a/fZtfrmm The resulting cloaked image: https://imgur.com/a/OSHXdbO https://imgur.com/a/OSHXdbO I applied a difference filter between the two images in Photoshop, to show an example of the actual pertubations performed: https://imgur.com/a/q4zC7Ms https://imgur.com/a/q4zC7Ms Since it's hard to see, I compressed the output to highlight what the program actually changed. It does seem like there is a good amount of disturbance to the image: https://imgur.com/a/1Sx68o3 https://imgur.com/a/1Sx68o3 Now, the real test. First, a Google reverse image search for the original file - identification is pretty bang-on: https://imgur.com/a/5HJwjPx https://imgur.com/a/5HJwjPx A Google reverse image search for the cloaked file: https://imgur.com/a/QByXBfS https://imgur.com/a/QByXBfS The only difference I'm seeing is a few images that are one or two images swapped in the "visually similar images" category. So, I figured that that's the "best case" for the cloaked image - giving the search algorithm the full, unfiltered data, and the program still failed to disguise it. For fun, I thought I would use a "low-pass filter" (Google Lens pointed at my computer screen) as well, just for thoroughness. And the result surprised me! Here's Google Lens pointed at my screen with the original image open: https://imgur.com/a/1BVRFG0 https://imgur.com/a/1BVRFG0 And here's Google Lens pointed at my screen with the cloaked image open: https://imgur.com/a/uoppuit https://imgur.com/a/uoppuit So, it would seem that the algorithm's distortions more effectively come through in worse quality images! But, based on my full-resolution result, I wouldn't trust it to disguise something that is being directly uploaded to a social network. Now, one important note is that reverse image search is probably not using a facial recognition model, but more like image chunk hashing - although I would also consider that something a privacy tool should defend against, which is why I included it. All in all, very interesting and thanks for convincing me that I should actually test it out.

raxxorrax 6y ago

Haven't read the paper yet but sure will do. I wonder how it works, that so many face rec implementations are fooled but I don't know how similar they work. I would have guessed they know which features are extracted and modify the relevant regions. I guess that face rec software will quickly adapt though. That said, we have invisible watermarks that are very resistant to compression or other filters.