4 ms·

Meta's Un-Stable Signature

146 points by ementally 3mo ago

flaxxer 3mo ago
also, easily bypassed now: https://twotensors.ai/ https://twotensors.ai/
- tarpitt 3mo ago
  There is actually an older method for countering steganography and adversarial image generation attacks: https://en.wikipedia.org/wiki/Gaussian_blur https://en.wikipedia.org/wiki/Gaussian_blur
  - miohtama 3mo ago
    AFAIK these methods claim to be blur resistant
    - masfuerte 3mo ago
      Yes. These methods don't work reliably. Apply a blur and they still don't work reliably.
miohtama 3mo ago
Related to this, the EU AI Act requires mandatory watermarking that is cannot be removed or is illegal to remove. https://digital-strategy.ec.europa.eu/en/policies/eu-icons-labelling-ai-generated-content https://digital-strategy.ec.europa.eu/en/policies/eu-icons-l... If Facebook already embeds user IDs in images (AI or no AI) I can only drool to think what kind tracking, advertising and mass surveillance opportunities are coming.
- N19PEDL2 3mo ago
  How can a watermark be unremovable?
  - embedding-shape 3mo ago
    The actual rules don't say that I think, it's more about the intention that the watermark is embedded with the image/multimedia itself, so it's persisting even if someone "right-click > save" the image or takes a screenshot, not literally regulated the watermark has to be unremovable. > (Summary) The icon should be directly embedded into the deep fake or published text (except for creative works), unless equivalent alternatives are available such as a user interface overlay. The icon must be visible when content is reshared or downloaded.
    - charcircuit 3mo ago
      It says the icons are optional. So that icon must not be what the other person was talking about.
      - embedding-shape 3mo ago
        > It says the icons are optional. So that icon must not be what the other person was talking about. What "watermark" are they talking about if not the label/icons? The label/icon in question are what the whole "EU Icons for labelling AI-generated content" thing is about, someone correct me if I'm having a big brain fart.
        Ohentis 3mo ago
        I imagine the goal is for everything to use something like Google's synthid.
        embedding-shape 3mo ago
        That sounds like one possible implementation, not the goal per se. The goal (the explicit/stated one at least) is to give people a heads up what's AI generated vs not, when that's unclear.
  - deleted 3mo ago
    [deleted]
- InsideOutSanta 3mo ago
  I don't see anything about watermarking in the linked article, it's about labelling requirements. It describes situations where you are required to disclose if an image was AI-generated.
richardfey 3mo ago
This is a great statistical analysis and it was a pleasure to read, but I wasn't expecting the claims to be so poorly supported. There's also a reply from one of the Meta authors there, worth checking out.
- hackerfactor1 3mo ago
  Which claims do you think are poorly supported? I'm the author; I tried to include everything other people need to repeat the same experiments. I've even had two people write in directly to me, stating that they have been able to replicate my findings. Or are you referring to the claims from Meta, Google, and Adobe -- which failed to hold up under independent evaluation. You are also correct that one of the meta authors wrote in a comment. However, he demonstrated a clear lack of understanding regarding what makes the bits "independent" or how to resolve the independence problem.
  - richardfey 3mo ago
    > Or are you referring to the claims from Meta, Google, and Adobe -- which failed to hold up under independent evaluation. This. > However, he demonstrated a clear lack of understanding regarding what makes the bits "independent" or how to resolve the independence problem. Yes, I didn’t want to call it out explicitly, but this is exactly the kind of thing that would have made my undergraduate statistics professor lose patience.
itake 3mo ago
A watermark is not just “transparency.” It can reveal what tool someone used, how they work, or that an image came from a stigmatized platform. In sensitive contexts—politics, sexuality, medical issues, protest material, or private expression—that can become surveillance. I am working on Saigon Watermarks: https://apps.apple.com/us/app/saigon-watermark/id6777061197 https://apps.apple.com/us/app/saigon-watermark/id6777061197 for detecting and removing provenence markers in AI. The tool also removes c2pa markers, which google is now linking the device that took the photo with the photo. scary stuff. https://security.googleblog.com/2025/09/pixel-android-trusted-images-c2pa-content-credentials.html https://security.googleblog.com/2025/09/pixel-android-truste...
- RobotToaster 3mo ago
  No android version?
  - itake 3mo ago
    Still working on it... I'm waiting on Apple to approve the MacOS version. After I will either focus on removing SynthID (currently not supported) or releasing android.
- hparadiz 3mo ago
  Another reason to drop both iOS and Android.
  - mort96 3mo ago
    In favour of what, might I ask? Keep in mind that the replacement must be a first class citizen in the society I live in. That means being able to easily transfer money using the app everyone uses where I live. It means public transport ticket apps. It means banking apps. It means iMessage, Facebook Messenger, Snapchat and Signal.
- UltraSane 3mo ago
  When generative AI can create such good fake images a valid c2pa linked to the source camera will become mandatory for an image to be considered authentic.
kamranjon 3mo ago
How common is it for peer reviewed papers like this to be so far off their claimed findings? “According to Google's peer-reviewed and published paper, they claim to have a true positive rate (TPR) above 99.97% -- meaning that they will miss their own watermarks less than 1 in 10,000 times. However, my own empirical testing found that is it much closer to 1 in 20.”
- a34729t 3mo ago
  If there were bounties for invalidating peer reviewed research, I suspect this would be a lot leas common.
f33d5173 3mo ago
The point is to embed a particular signature, which was generated randomly, in the image. The distance function discussed is the same as the popcount of the xor. It's well know that the xor of random data with correlated data is statistically random. Hence, however well correlated the signatures of unwatermarked images may be with each other, they would show no correlation with the signature of a watermarked image. That is, unless the watermark by extremely bad luck happened to be near one of these clusters the author discovered. This does represent a genuine flaw, but an extremely minor one, and one that can be easily mitigated with no changes to the underlying algorithm,
chris_explicare 3mo ago
[dead]

146 points by ementally 3mo ago

flaxxer 3mo ago

also, easily bypassed now: https://twotensors.ai/ https://twotensors.ai/

tarpitt 3mo ago
There is actually an older method for countering steganography and adversarial image generation attacks: https://en.wikipedia.org/wiki/Gaussian_blur https://en.wikipedia.org/wiki/Gaussian_blur
- miohtama 3mo ago
  AFAIK these methods claim to be blur resistant
  - masfuerte 3mo ago
    Yes. These methods don't work reliably. Apply a blur and they still don't work reliably.

miohtama 3mo ago

Related to this, the EU AI Act requires mandatory watermarking that is cannot be removed or is illegal to remove. https://digital-strategy.ec.europa.eu/en/policies/eu-icons-labelling-ai-generated-content https://digital-strategy.ec.europa.eu/en/policies/eu-icons-l... If Facebook already embeds user IDs in images (AI or no AI) I can only drool to think what kind tracking, advertising and mass surveillance opportunities are coming.

N19PEDL2 3mo ago
How can a watermark be unremovable?
- embedding-shape 3mo ago
  The actual rules don't say that I think, it's more about the intention that the watermark is embedded with the image/multimedia itself, so it's persisting even if someone "right-click > save" the image or takes a screenshot, not literally regulated the watermark has to be unremovable. > (Summary) The icon should be directly embedded into the deep fake or published text (except for creative works), unless equivalent alternatives are available such as a user interface overlay. The icon must be visible when content is reshared or downloaded.
  - charcircuit 3mo ago
    It says the icons are optional. So that icon must not be what the other person was talking about.
    - embedding-shape 3mo ago
      > It says the icons are optional. So that icon must not be what the other person was talking about. What "watermark" are they talking about if not the label/icons? The label/icon in question are what the whole "EU Icons for labelling AI-generated content" thing is about, someone correct me if I'm having a big brain fart.
      - Ohentis 3mo ago
        I imagine the goal is for everything to use something like Google's synthid.
        embedding-shape 3mo ago
        That sounds like one possible implementation, not the goal per se. The goal (the explicit/stated one at least) is to give people a heads up what's AI generated vs not, when that's unclear.
- deleted 3mo ago
  [deleted]
InsideOutSanta 3mo ago
I don't see anything about watermarking in the linked article, it's about labelling requirements. It describes situations where you are required to disclose if an image was AI-generated.

richardfey 3mo ago

This is a great statistical analysis and it was a pleasure to read, but I wasn't expecting the claims to be so poorly supported. There's also a reply from one of the Meta authors there, worth checking out.

hackerfactor1 3mo ago
Which claims do you think are poorly supported? I'm the author; I tried to include everything other people need to repeat the same experiments. I've even had two people write in directly to me, stating that they have been able to replicate my findings. Or are you referring to the claims from Meta, Google, and Adobe -- which failed to hold up under independent evaluation. You are also correct that one of the meta authors wrote in a comment. However, he demonstrated a clear lack of understanding regarding what makes the bits "independent" or how to resolve the independence problem.
- richardfey 3mo ago
  > Or are you referring to the claims from Meta, Google, and Adobe -- which failed to hold up under independent evaluation. This. > However, he demonstrated a clear lack of understanding regarding what makes the bits "independent" or how to resolve the independence problem. Yes, I didn’t want to call it out explicitly, but this is exactly the kind of thing that would have made my undergraduate statistics professor lose patience.

itake 3mo ago

A watermark is not just “transparency.” It can reveal what tool someone used, how they work, or that an image came from a stigmatized platform. In sensitive contexts—politics, sexuality, medical issues, protest material, or private expression—that can become surveillance. I am working on Saigon Watermarks: https://apps.apple.com/us/app/saigon-watermark/id6777061197 https://apps.apple.com/us/app/saigon-watermark/id6777061197 for detecting and removing provenence markers in AI. The tool also removes c2pa markers, which google is now linking the device that took the photo with the photo. scary stuff. https://security.googleblog.com/2025/09/pixel-android-trusted-images-c2pa-content-credentials.html https://security.googleblog.com/2025/09/pixel-android-truste...

RobotToaster 3mo ago
No android version?
- itake 3mo ago
  Still working on it... I'm waiting on Apple to approve the MacOS version. After I will either focus on removing SynthID (currently not supported) or releasing android.
hparadiz 3mo ago
Another reason to drop both iOS and Android.
- mort96 3mo ago
  In favour of what, might I ask? Keep in mind that the replacement must be a first class citizen in the society I live in. That means being able to easily transfer money using the app everyone uses where I live. It means public transport ticket apps. It means banking apps. It means iMessage, Facebook Messenger, Snapchat and Signal.
UltraSane 3mo ago
When generative AI can create such good fake images a valid c2pa linked to the source camera will become mandatory for an image to be considered authentic.

kamranjon 3mo ago

How common is it for peer reviewed papers like this to be so far off their claimed findings? “According to Google's peer-reviewed and published paper, they claim to have a true positive rate (TPR) above 99.97% -- meaning that they will miss their own watermarks less than 1 in 10,000 times. However, my own empirical testing found that is it much closer to 1 in 20.”

a34729t 3mo ago
If there were bounties for invalidating peer reviewed research, I suspect this would be a lot leas common.

f33d5173 3mo ago

The point is to embed a particular signature, which was generated randomly, in the image. The distance function discussed is the same as the popcount of the xor. It's well know that the xor of random data with correlated data is statistically random. Hence, however well correlated the signatures of unwatermarked images may be with each other, they would show no correlation with the signature of a watermarked image. That is, unless the watermark by extremely bad luck happened to be near one of these clusters the author discovered. This does represent a genuine flaw, but an extremely minor one, and one that can be easily mitigated with no changes to the underlying algorithm,

chris_explicare 3mo ago

[dead]