4 ms·

I think it's useful even if it can be spoofed. Many people don't even bother to edit visible watermarks out of AI photos/videos. I'm fairly certain this will de

by fwipsy 1mo ago

I think it's useful even if it can be spoofed. Many people don't even bother to edit visible watermarks out of AI photos/videos. I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing. People are concerned that the technology will lend additional credence to the last 0.1%. But anyone who thinks about the technology for 2 minutes will see you can just point the camera at the screen. In cases where it really matters (a court of law, internet arguments between nerds) people will know it's not 100% reliable. Locks can be picked, and signatures can be forged, but that doesn't make them useless. "C2PA Cameras Do Not Survive Contact With Reality" does not survive contact with reality where very, very few users would even think of rooting their phone so they can create signed fake images.

Retr0id 1mo ago
When I search for "C2PA" on the google play store, there are more AI-watermark-removal apps than there are signing apps. Certain types will jump through ridiculous hoops if they think it will affect their algorithmic reach on social media. Malicious users don't need to root their own phones. They just need to go to fakemyimage dot com, and someone else's rooted phone in a clickfarm-type setup signs it for them. I am not operating such a service myself because I thought it was unnecessary in making my point, but perhaps I will have to reconsider.
- fwipsy 1mo ago
  You're arguing that lots of people can spoof this, the other guy is arguing that nobody will know it can be spoofed so it will do more damage. But these are contradictory -- if fakes become common, then they will also become common knowledge. The impact of any given fake is reduced if there are more of them. The technology doesn't need to provide 100% assurance. If it adds even a little friction to the slop mills then that's increasing the signal to noise ratio.
treyd 1mo ago
It's actually worse if it is plausibly trustworthy for "99.9%", since that's enough that naive users will get accustomed to believing the verification badge is authentic. When a motivated malicious user (who doesn't actually need that much resources) will be able to convince people something is authentic because the verification passes when it shouldn't since naive users are primed to believe it by default.
- fwipsy 1mo ago
  Read the rest of my comment please. Is the single motivated malicious user able to do as much damage as all of the blocked attempts put together? Probably not, since if there's really all that much riding on it, people will point out it can be bypassed. Should we also abolish Pangram, because it's not 100% accurate? Someone might be convinced a text is not AI-generated when it actually is! We should get rid of it rather than fool people into thinking it can be determined accurately. What about antivirus? We should abolish it as well rather than fool people into thinking that their software is ever 100% safe. What about HTTPS? We shouldn't call it "secure" shell because the computer you're connecting to could be compromised! I could go on and on and on. The median instance of AI image generation isn't evidence in a court case. It's cyberbullying, or deepfakes, or fake news. It's called "slop" because there's a lot of it being churned out at low effort.
  - hypfer 1mo ago
    You're missing all of the points that there could be by focussing on random people. While it is always an individual tragedy when people treat each other badly (e.g. through deepfakes and all), the real threat does not exist on that level. This is about misinformation and disinformation, so we're talking state actors. And with that, the 99.9% hypothesis does not hold true.
    - fwipsy 1mo ago
      It's funny how people always say something is "a tragedy at the individual level" when they mean "it's not my problem." It's even crazier to dismiss the value of a security feature, just because it might make people feel more secure. That's true of every security feature! Very little of the technology that the web is built on is proof against state actors. I like being contrarian as much as the next guy, but "Actually, having security is worse for security" is taking it a little too far.
      - hypfer 1mo ago
        I am repeating myself, but this is about systems, and not about people. It is however in the interest of the people to keep the systems running in an untainted way. As said, on the individual level it's a tragedy, but one that can be absorbed somewhat. Democracy itself failing otoh is kinda hard to absorb. C2PA is not "having security". It is "having an illusion of security for compliance and CYA reasons, that can be fairly trivially exploited by nation state actors". Banality of evil. Again. ___ Actually, come to think of it, "security" is the wrong term there. Signatures don't secure anything. They attest. Those are different things. Argh and I ran with your term aah
fightfake-ai 1mo ago
I agree with "I'm fairly certain this will defeat 99.9% of malicious users". Also, note that C2PA should have something like Level 3 as well: "The image is mathematically proven to have come from the physical camera sensor." It's somehow difficult to achieve this, but it's possible (although the attacks will always be possible of course).
- dTal 1mo ago
  Difficult and also solves nothing, since you can just point the camera at a screen.
  - fightfake-ai 1mo ago
    There is active research addressing this problem, for example https://www.usenix.org/system/files/usenixsecurity25-park.pdf https://www.usenix.org/system/files/usenixsecurity25-park.pd... But yeah, difficult too :)
  - thaumasiotes 1mo ago
    Have you ever tried pointing a camera at a screen? The screen is doing all kinds of crazy things that are not apparent to your eyes, but that show up clearly on camera.
    - blincoln 1mo ago
      Some screens are like that, but it's not an inherent property of all visual displays. For example, you'll see a sort of barber-pole effect when pointing a video camera at a raster-scan digital display whose refresh rate isn't synced to the camera's frame rate. To avoid that, sync them, or maybe use a colour e-ink display. Alternatively, print a high-resolution version with a decent photograph printer and take a picture of the print with the C2PA camera.
rcxdude 1mo ago
What percentage of users are malicious, do you think? To me the issue is precisely what the product is trying to solve: propaganda using faked footage passed off as real, which generally has no real difficulty with resources available to boost their message. Giving that any kind of stamp of authenticity is a bad idea, IMO, and the fact that it'll work on 99% of the cases that don't matter makes it even worse.
Gormo 1mo ago
> I think it's useful even if it can be spoofed. Many people don't even bother to edit visible watermarks out of AI photos/videos. I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing. Exactly: most people don't bother editing visible watermarks out of AI-generated media, because they have little or no incentive to bother doing so. This will "defeat" the 99.9% of users who are not actually trying to do anything malicious, but will be a minor annoyance to the 0.1% of users who are actively engaging in fraud, fabrication of evidence, etc. The upshot is that not only us this not useful for its intended purpose, it will lure people into a false sense of security by creating expectations that AI-generated media will always be easily identifiable as such, and reduce the level of scrutiny that gets applied to the stuff that actually is malicious.
mistercow 1mo ago
> I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing. Everyone realizing that photos don't prove anything would defeat 100% of malicious users. I don't understand what people incorrectly trusting photos is supposed to achieve at this point, in your view.
- fwipsy 1mo ago
  Good luck with that. People are basically going to believe photos if they're aligned with what they already believe. They're not going to make strong decisions based on the C2PA tag. The idea that people will revise their entire worldview or submit to fraud based on a C2PA tag is typical HN thinking. People don't trust technology that much.
fwipsy 1mo ago
Almost everyone seems to be overindexing on this second-order effect. But second-order effects don't typically negate or surpass the original effect, because they depend on the original effect.

by fwipsy 1mo ago

Retr0id 1mo ago

When I search for "C2PA" on the google play store, there are more AI-watermark-removal apps than there are signing apps. Certain types will jump through ridiculous hoops if they think it will affect their algorithmic reach on social media. Malicious users don't need to root their own phones. They just need to go to fakemyimage dot com, and someone else's rooted phone in a clickfarm-type setup signs it for them. I am not operating such a service myself because I thought it was unnecessary in making my point, but perhaps I will have to reconsider.

fwipsy 1mo ago
You're arguing that lots of people can spoof this, the other guy is arguing that nobody will know it can be spoofed so it will do more damage. But these are contradictory -- if fakes become common, then they will also become common knowledge. The impact of any given fake is reduced if there are more of them. The technology doesn't need to provide 100% assurance. If it adds even a little friction to the slop mills then that's increasing the signal to noise ratio.

treyd 1mo ago

It's actually worse if it is plausibly trustworthy for "99.9%", since that's enough that naive users will get accustomed to believing the verification badge is authentic. When a motivated malicious user (who doesn't actually need that much resources) will be able to convince people something is authentic because the verification passes when it shouldn't since naive users are primed to believe it by default.

fwipsy 1mo ago
Read the rest of my comment please. Is the single motivated malicious user able to do as much damage as all of the blocked attempts put together? Probably not, since if there's really all that much riding on it, people will point out it can be bypassed. Should we also abolish Pangram, because it's not 100% accurate? Someone might be convinced a text is not AI-generated when it actually is! We should get rid of it rather than fool people into thinking it can be determined accurately. What about antivirus? We should abolish it as well rather than fool people into thinking that their software is ever 100% safe. What about HTTPS? We shouldn't call it "secure" shell because the computer you're connecting to could be compromised! I could go on and on and on. The median instance of AI image generation isn't evidence in a court case. It's cyberbullying, or deepfakes, or fake news. It's called "slop" because there's a lot of it being churned out at low effort.
- hypfer 1mo ago
  You're missing all of the points that there could be by focussing on random people. While it is always an individual tragedy when people treat each other badly (e.g. through deepfakes and all), the real threat does not exist on that level. This is about misinformation and disinformation, so we're talking state actors. And with that, the 99.9% hypothesis does not hold true.
  - fwipsy 1mo ago
    It's funny how people always say something is "a tragedy at the individual level" when they mean "it's not my problem." It's even crazier to dismiss the value of a security feature, just because it might make people feel more secure. That's true of every security feature! Very little of the technology that the web is built on is proof against state actors. I like being contrarian as much as the next guy, but "Actually, having security is worse for security" is taking it a little too far.
    - hypfer 1mo ago
      I am repeating myself, but this is about systems, and not about people. It is however in the interest of the people to keep the systems running in an untainted way. As said, on the individual level it's a tragedy, but one that can be absorbed somewhat. Democracy itself failing otoh is kinda hard to absorb. C2PA is not "having security". It is "having an illusion of security for compliance and CYA reasons, that can be fairly trivially exploited by nation state actors". Banality of evil. Again. ___ Actually, come to think of it, "security" is the wrong term there. Signatures don't secure anything. They attest. Those are different things. Argh and I ran with your term aah

fightfake-ai 1mo ago

I agree with "I'm fairly certain this will defeat 99.9% of malicious users". Also, note that C2PA should have something like Level 3 as well: "The image is mathematically proven to have come from the physical camera sensor." It's somehow difficult to achieve this, but it's possible (although the attacks will always be possible of course).

dTal 1mo ago
Difficult and also solves nothing, since you can just point the camera at a screen.
- fightfake-ai 1mo ago
  There is active research addressing this problem, for example https://www.usenix.org/system/files/usenixsecurity25-park.pdf https://www.usenix.org/system/files/usenixsecurity25-park.pd... But yeah, difficult too :)
- thaumasiotes 1mo ago
  Have you ever tried pointing a camera at a screen? The screen is doing all kinds of crazy things that are not apparent to your eyes, but that show up clearly on camera.
  - blincoln 1mo ago
    Some screens are like that, but it's not an inherent property of all visual displays. For example, you'll see a sort of barber-pole effect when pointing a video camera at a raster-scan digital display whose refresh rate isn't synced to the camera's frame rate. To avoid that, sync them, or maybe use a colour e-ink display. Alternatively, print a high-resolution version with a decent photograph printer and take a picture of the print with the C2PA camera.

rcxdude 1mo ago

What percentage of users are malicious, do you think? To me the issue is precisely what the product is trying to solve: propaganda using faked footage passed off as real, which generally has no real difficulty with resources available to boost their message. Giving that any kind of stamp of authenticity is a bad idea, IMO, and the fact that it'll work on 99% of the cases that don't matter makes it even worse.

Gormo 1mo ago

> I think it's useful even if it can be spoofed. Many people don't even bother to edit visible watermarks out of AI photos/videos. I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing. Exactly: most people don't bother editing visible watermarks out of AI-generated media, because they have little or no incentive to bother doing so. This will "defeat" the 99.9% of users who are not actually trying to do anything malicious, but will be a minor annoyance to the 0.1% of users who are actively engaging in fraud, fabrication of evidence, etc. The upshot is that not only us this not useful for its intended purpose, it will lure people into a false sense of security by creating expectations that AI-generated media will always be easily identifiable as such, and reduce the level of scrutiny that gets applied to the stuff that actually is malicious.

mistercow 1mo ago

> I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing. Everyone realizing that photos don't prove anything would defeat 100% of malicious users. I don't understand what people incorrectly trusting photos is supposed to achieve at this point, in your view.

fwipsy 1mo ago
Good luck with that. People are basically going to believe photos if they're aligned with what they already believe. They're not going to make strong decisions based on the C2PA tag. The idea that people will revise their entire worldview or submit to fraud based on a C2PA tag is typical HN thinking. People don't trust technology that much.

fwipsy 1mo ago

Almost everyone seems to be overindexing on this second-order effect. But second-order effects don't typically negate or surpass the original effect, because they depend on the original effect.