7 ms·

New Tech from Camera Makers Tries to Prove Photos Are Not AI Fakes

28 points by webmaven 3y ago

ohlokkru 3y ago
Meh would prefer untrusted digital world. Would act as a forcing function for us to live in the real world, where being there is the only reliable source.
- ben_w 3y ago
  Ah, I see you've not encountered any performance magicians, sleight of hand tricks, etc.
- DFHippie 3y ago
  Receiving and trusting information is essential to being human. We don't have to prove everything to ourselves from sensory evidence and first principles because other people can credibly tell us things.
  - benjiManir 3y ago
    Eh, Language is a mechanical behavior involving interpretation of sound. Triggering sensorial experience is how other people credibly tell us things. We have to verify via prior sensory experience (past experience) if the person speaking in the present is credible. How do we do that with limited sensory experience of their character? Personal ignorances and biases due to lack of experience? We have to understand how to leverage the first principles of math to extend them, and teach those to others to propagate the knowledge. Not really sure it’s possible to remove sensory experience from … well any understanding of the world. Not sure what you wrote makes any sense.
    - nullandvoid 3y ago
      People lie about, or exaggerate their memories - often actually believing their own false interpretation, this is human nature. To have no ability to determine whether a video, or image (or any media) is from a reliable source forever more, seems like a pretty bad place to be in - if this is what you're suggesting?
- jdietrich 3y ago
  Attesting documents is an old and important social technology. Here in the UK, the police have a long tradition of using pocket notebooks to record their observations and activities. It is a professional duty to make a contemporaneous note of anything significant that happens during a shift - attending a crime scene, observing a suspect, taking a statement from a witness. Each notebook has a unique serial number, which is recorded by the officer's supervisor when it is issued to them. Each page of the notebook is numbered sequentially. Each entry in that notebook must be closed with a line, the time and date and the officer's signature. When a police officer gives evidence in court, the defence will ask to see their pocket notebook as a matter of course. If the evidence they give isn't corroborated by their pocket notebook, a jury or magistrate is entitled to draw their own conclusions about the veracity of the testimony. If an entry is obliterated or a page has been torn out, if the entries are out-of-sequence or don't match with the dispatch logs from the day in question, that's obviously a cause for significant scepticism and may constitute an act of professional misconduct. A notebook with numbered pages isn't a particularly clever technical innovation, but the practice surrounding it creates a great deal of value. It doesn't infallibly prove that an officer is telling the truth, but it creates a lot of hurdles and pitfalls for a dishonest officer. Likewise, in-camera signing of images doesn't prove that an image is an accurate reflection of reality, but it does make life more difficult for dishonest photojournalists or insurance assessors or forensic investigators. It gives few firm answers, but it does allow us to ask useful questions. If you're presenting an image as evidence, is it signed? Do the timestamp and geotag fit with your testimony regarding the circumstances in which the image was taken? If not, why?
  - Kon-Peki 3y ago
    Every camera maintains its own blockchain!
- mensetmanusman 3y ago
  Probably humanity will regret the technology that strengthens a post truth world.
h0l0cube 3y ago
Anyone know how resistant this could be to cropping, scaling, or grading? Is this purely for verifying the raw or could this be something that social platforms could verify and tag on posts?
- Retr0id 3y ago
  The idea is that full edit history is preserved - not verbatim in the metadata, but as a tag that can be looked up in a database. More info here: https://c2pa.org/ https://c2pa.org/
  - h0l0cube 3y ago
    As much as I can glean from the video, it's heavily dependent on a central database to hold the edits. If someone alters a video, the original would need to be viewable. Who's storing that, and streaming it? And then who's paying for that? I'm guessing that in the end this will be a paid service for journalistic organizations who buy in.
    - jdietrich 3y ago
      >If someone alters a video, the original would need to be viewable. What exactly do you mean by "need"? This system doesn't stop anyone from distributing edited or staged or otherwise bogus media, because that was never the intention. It does allow someone to demonstrate - with a fairly high level of trustworthiness - that the media they're delivering is raw, unedited footage taken with a particular camera at a particular time in a particular place.
      - Retr0id 3y ago
        Proving that you're delivering the raw unedited footage requires retaining the raw unedited footage (i.e. whatever was originally signed by the camera body), and distributing it to anyone who asks to check your proofs. This is inevitably a different (and much larger) file, to whatever ends up streaming from social media platforms.
      - 3y ago
Retr0id 3y ago
I can't wait to take real pictures of a high-resolution display (or even, interpose the bus connecting the image sensor to the rest of the camera)
- vanviegen 3y ago
  In order to achieve what exactly?
  - Retr0id 3y ago
    To get that shiny authentic signature on arbitrary image data.
  - geraldhh 3y ago
    an audit trail
- Retr0id 3y ago
  I just watched the embedded video, which actually shows a photographer using a digital camera to digitize and "authenticate" analog slides. The whole thing seems like a bit of a charade, to me. I found a service selling 35mm slides of digital image files for £3.25 each.
  - jdietrich 3y ago
    OK, so you've digitised an analog slide. Any experienced photographer could spot that instantly because slide film has very different properties to an imaging sensor, but we'll set that aside for a moment. Your camera generated a C2PA manifest when you created that image. If you include the EXIF metadata in that manifest, your subterfuge is revealed immediately. Even without GPS fields, the lens and camera parameters will tell me that you've taken a photo of a very small object rather than the scene represented in the image. If you don't include that metadata, then I know that you've intentionally concealed it from me. You might have a good reason to conceal that data, but I know that the manifest only "validates" a limited set of claims that do not actually tell me much (if anything) about the image. C2PA is a tool to establish provenance, nothing more. It doesn't prove that an image is "true" in a philosophical sense, it just adds a layer of trust about the technical processes used to create that image. It doesn't stop anyone from misrepresenting an image, it doesn't stop anyone from circulating staged or AI-generated images on social media, but it does allow legitimate reporters, news agencies and other bodies to audit their information flows and show their workings. That isn't a magic bullet against misinformation, but it's still extremely valuable in lots of circumstances.
leereeves 3y ago
How will they keep the private keys secret?
- Retr0id 3y ago
  They can't, ultimately, it's basically just DRM but in reverse. I'm sure they can make it hard, with secure silicon etc., but they'll never make it impossible and that somewhat defeats the purpose.
  - dist-epoch 3y ago
    The private keys in credit cards (the ones with chips) are pretty secure. I'm no aware of them being extractable.
    - Retr0id 3y ago
      I'm sure they're not extractable at scale, otherwise we'd have heard about it, but they're almost certainly extractable on a one-off basis if you have a big enough budget (one that likely outweighs whatever you'd stand to gain from it). It's just not the low hanging fruit of the system as a whole. I also expect it not to be the low-hanging fruit of these cameras, but it is still one possible attack vector.
  - yrxuthst 3y ago
    It still serves a legal purpose, as long as it's difficult enough. If you need to physically modify the device to access the keys, then bringing the unmodified device to court should be enough to verify legitimacy of the signature.
    - Retr0id 3y ago
      Since when are we taking photojournalists and social media users to court to verify each image?
- dist-epoch 3y ago
  With homomorphic encryption you can make them provably unextractable.
ChatGTP 3y ago
Slightly off topic, but has anyone felt they've improved at identifying "AI" images? My instagram feed is increasingly filled with "AI models", for some reason I can really easily identify these images whereas earlier on I really struggled a lot. Now I can more easily recogize these images, it actually turns me off using Instagram as I find it annoying / jarring. A few months back, I'd be looking at the hands etc to try work out if the image was real or not. I'd always fail those "is this person real or not" quizzes where as now, I can smash those things pretty easily and almost always know when an IG post is faked.
- dukeyukey 3y ago
  I _think_ I'm good at identifying AI images, but part of it is you can never be sure.
  - ChatGTP 3y ago
    Yeah, I don’t disagree. But in some sense , when could any photo be trusted ? We’ve had airbrushing, CGI forever now.
    - fortran77 3y ago
      Or just weird lighting and "Myspace Angles" (people photographing themselves from an angle above, etc)
- mensetmanusman 3y ago
  What you are looking at might have a lower compute count as the popularity grows and the GPU/User ratio shrinks. A GPU shortage under popularity growth should result in more obvious AI art over time sans an algorithm breakthrough that reduces compute cost by orders.
iamleppert 3y ago
Can’t wait to crack this. Looking at the specs, it’s hugely complicated for what it is and has an enormous attack surface.
- patcon 3y ago
  It may not be as stupid as it looks. If this is connected to what I understand, the tech has its provenance in Starling Lab: https://www.starlinglab.org/journalism-about/ https://www.starlinglab.org/journalism-about/ Starling Lab involves Dan Boneh.
jillesvangurp 3y ago
Generally, signing digital files is a great way to have some strong form of authenticity. There's a lot of unsigned content on the internet which makes impersonating people quite easy. We have fake news, fake comments, fake reviews, propaganda departments from intelligence agencies from hostile countries meddling with elections and public affairs, companies and lobby organizations spreading FUD, marketing companies specializing in bending public opinion, etc. There's a lot of good reason to start insisting of people using digital signatures for everything they produce. Even without considering AI. People obsess about what AIs can do here. But you might just make the point that at some point it starts rivaling what a determined hostile person could do manually right now. Sure, it adds scale and convenience for those with such intentions; which makes the problem more urgent than it already was. But it was pretty bad before that. Putin did not use AI a few years ago. He simply just put some teams of writers to work to flood the internet with content intended to bend Elections this way or that way or erode public confidence. Web of trust has been tried before of course but IMHO it deserves another chance. The current state is not sustainable when we might get AI generated content dwarfing the amount of genuine content very soon.
- Retr0id 3y ago
  I do think there's something useful here. It's like having photographers PGP sign all their images, but with an actually workable UX, and with keys (presumably) stored in a secure enclave within the camera body. This is good. I just think that marketing teams are inevitably going to oversell it. The moment someone posts an obviously fake image to social media, with all checks from this tech giving the green light, it will destroy the public's confidence in it. The actual usefulness of the tech is nuanced, and it will be a struggle to convey it meaningfully (to technical and non-technical audiences alike)
- loceng 3y ago
  What if what was recorded or photographed, you don't want your identity associated with it due to its contents - e.g. whistleblowing classified government/state documents exposing horrific crimes against humanity? If no digital signature then it's automatically fake? It doesn't seem to matter at the moment though anyhow, the corrupt establishment so far seems to be a good enough job suppressing what they don't want out and maintaining control.
throwawayForMe2 3y ago
Leica has a camera model out implementing this standard. https://leicacamerausa.com/leica-m11-p-in-silver-chrome.html https://leicacamerausa.com/leica-m11-p-in-silver-chrome.html
lemoncookiechip 3y ago
Sorry for the dumb question, but what is stopping anyone from taking a picture of an AI generated image and passing it up as the "real deal"?
- Moldoteck 3y ago
  Noise patterns from the camera will be off compared to what is in the image (I think)
- waynenilsen 3y ago
  Depth of field etc will be hard to fake converting from analog back to digital like this esp if the focal length etc are encoded in the committed data
mikewarot 3y ago
It's not new, they just waited for the Patents[1,2] to run out. It's been a long time since I worked on it. [1] https://patentimages.storage.googleapis.com/f7/e9/46/00ddfea701de89/US8219545.pdf https://patentimages.storage.googleapis.com/f7/e9/46/00ddfea... [2] https://patentimages.storage.googleapis.com/4f/e3/ed/a46504f8cc2998/US6757828.pdf https://patentimages.storage.googleapis.com/4f/e3/ed/a46504f...
- JieJie 3y ago
  Many comments suggest this won't be useful technology in that it will be easily circumvented for nefarious purposes. Would you mind commenting on your perception of the potential of this particular implementation of the technology you helped create?
  - mikewarot 3y ago
    This is a way to sign an image while it's still in a camera. The signature is simply a way to strongly assert that an image is as the camera saw it in the raw. I saw it as a way for Law Enforcement to prove an evidence photo was not doctored.

28 points by webmaven 3y ago

Meh would prefer untrusted digital world. Would act as a forcing function for us to live in the real world, where being there is the only reliable source.

ben_w 3y ago
Ah, I see you've not encountered any performance magicians, sleight of hand tricks, etc.
DFHippie 3y ago
Receiving and trusting information is essential to being human. We don't have to prove everything to ourselves from sensory evidence and first principles because other people can credibly tell us things.
- benjiManir 3y ago
  Eh, Language is a mechanical behavior involving interpretation of sound. Triggering sensorial experience is how other people credibly tell us things. We have to verify via prior sensory experience (past experience) if the person speaking in the present is credible. How do we do that with limited sensory experience of their character? Personal ignorances and biases due to lack of experience? We have to understand how to leverage the first principles of math to extend them, and teach those to others to propagate the knowledge. Not really sure it’s possible to remove sensory experience from … well any understanding of the world. Not sure what you wrote makes any sense.
  - nullandvoid 3y ago
    People lie about, or exaggerate their memories - often actually believing their own false interpretation, this is human nature. To have no ability to determine whether a video, or image (or any media) is from a reliable source forever more, seems like a pretty bad place to be in - if this is what you're suggesting?
jdietrich 3y ago
Attesting documents is an old and important social technology. Here in the UK, the police have a long tradition of using pocket notebooks to record their observations and activities. It is a professional duty to make a contemporaneous note of anything significant that happens during a shift - attending a crime scene, observing a suspect, taking a statement from a witness. Each notebook has a unique serial number, which is recorded by the officer's supervisor when it is issued to them. Each page of the notebook is numbered sequentially. Each entry in that notebook must be closed with a line, the time and date and the officer's signature. When a police officer gives evidence in court, the defence will ask to see their pocket notebook as a matter of course. If the evidence they give isn't corroborated by their pocket notebook, a jury or magistrate is entitled to draw their own conclusions about the veracity of the testimony. If an entry is obliterated or a page has been torn out, if the entries are out-of-sequence or don't match with the dispatch logs from the day in question, that's obviously a cause for significant scepticism and may constitute an act of professional misconduct. A notebook with numbered pages isn't a particularly clever technical innovation, but the practice surrounding it creates a great deal of value. It doesn't infallibly prove that an officer is telling the truth, but it creates a lot of hurdles and pitfalls for a dishonest officer. Likewise, in-camera signing of images doesn't prove that an image is an accurate reflection of reality, but it does make life more difficult for dishonest photojournalists or insurance assessors or forensic investigators. It gives few firm answers, but it does allow us to ask useful questions. If you're presenting an image as evidence, is it signed? Do the timestamp and geotag fit with your testimony regarding the circumstances in which the image was taken? If not, why?
- Kon-Peki 3y ago
  Every camera maintains its own blockchain!
mensetmanusman 3y ago
Probably humanity will regret the technology that strengthens a post truth world.

h0l0cube 3y ago

Anyone know how resistant this could be to cropping, scaling, or grading? Is this purely for verifying the raw or could this be something that social platforms could verify and tag on posts?

Retr0id 3y ago
The idea is that full edit history is preserved - not verbatim in the metadata, but as a tag that can be looked up in a database. More info here: https://c2pa.org/ https://c2pa.org/
- h0l0cube 3y ago
  As much as I can glean from the video, it's heavily dependent on a central database to hold the edits. If someone alters a video, the original would need to be viewable. Who's storing that, and streaming it? And then who's paying for that? I'm guessing that in the end this will be a paid service for journalistic organizations who buy in.
  - jdietrich 3y ago
    >If someone alters a video, the original would need to be viewable. What exactly do you mean by "need"? This system doesn't stop anyone from distributing edited or staged or otherwise bogus media, because that was never the intention. It does allow someone to demonstrate - with a fairly high level of trustworthiness - that the media they're delivering is raw, unedited footage taken with a particular camera at a particular time in a particular place.
    - Retr0id 3y ago
      Proving that you're delivering the raw unedited footage requires retaining the raw unedited footage (i.e. whatever was originally signed by the camera body), and distributing it to anyone who asks to check your proofs. This is inevitably a different (and much larger) file, to whatever ends up streaming from social media platforms.
    - 3y ago

Retr0id 3y ago

I can't wait to take real pictures of a high-resolution display (or even, interpose the bus connecting the image sensor to the rest of the camera)

vanviegen 3y ago
In order to achieve what exactly?
- Retr0id 3y ago
  To get that shiny authentic signature on arbitrary image data.
- geraldhh 3y ago
  an audit trail
Retr0id 3y ago
I just watched the embedded video, which actually shows a photographer using a digital camera to digitize and "authenticate" analog slides. The whole thing seems like a bit of a charade, to me. I found a service selling 35mm slides of digital image files for £3.25 each.
- jdietrich 3y ago
  OK, so you've digitised an analog slide. Any experienced photographer could spot that instantly because slide film has very different properties to an imaging sensor, but we'll set that aside for a moment. Your camera generated a C2PA manifest when you created that image. If you include the EXIF metadata in that manifest, your subterfuge is revealed immediately. Even without GPS fields, the lens and camera parameters will tell me that you've taken a photo of a very small object rather than the scene represented in the image. If you don't include that metadata, then I know that you've intentionally concealed it from me. You might have a good reason to conceal that data, but I know that the manifest only "validates" a limited set of claims that do not actually tell me much (if anything) about the image. C2PA is a tool to establish provenance, nothing more. It doesn't prove that an image is "true" in a philosophical sense, it just adds a layer of trust about the technical processes used to create that image. It doesn't stop anyone from misrepresenting an image, it doesn't stop anyone from circulating staged or AI-generated images on social media, but it does allow legitimate reporters, news agencies and other bodies to audit their information flows and show their workings. That isn't a magic bullet against misinformation, but it's still extremely valuable in lots of circumstances.

leereeves 3y ago

How will they keep the private keys secret?

Retr0id 3y ago
They can't, ultimately, it's basically just DRM but in reverse. I'm sure they can make it hard, with secure silicon etc., but they'll never make it impossible and that somewhat defeats the purpose.
- dist-epoch 3y ago
  The private keys in credit cards (the ones with chips) are pretty secure. I'm no aware of them being extractable.
  - Retr0id 3y ago
    I'm sure they're not extractable at scale, otherwise we'd have heard about it, but they're almost certainly extractable on a one-off basis if you have a big enough budget (one that likely outweighs whatever you'd stand to gain from it). It's just not the low hanging fruit of the system as a whole. I also expect it not to be the low-hanging fruit of these cameras, but it is still one possible attack vector.
- yrxuthst 3y ago
  It still serves a legal purpose, as long as it's difficult enough. If you need to physically modify the device to access the keys, then bringing the unmodified device to court should be enough to verify legitimacy of the signature.
  - Retr0id 3y ago
    Since when are we taking photojournalists and social media users to court to verify each image?
dist-epoch 3y ago
With homomorphic encryption you can make them provably unextractable.

ChatGTP 3y ago

Slightly off topic, but has anyone felt they've improved at identifying "AI" images? My instagram feed is increasingly filled with "AI models", for some reason I can really easily identify these images whereas earlier on I really struggled a lot. Now I can more easily recogize these images, it actually turns me off using Instagram as I find it annoying / jarring. A few months back, I'd be looking at the hands etc to try work out if the image was real or not. I'd always fail those "is this person real or not" quizzes where as now, I can smash those things pretty easily and almost always know when an IG post is faked.

dukeyukey 3y ago
I _think_ I'm good at identifying AI images, but part of it is you can never be sure.
- ChatGTP 3y ago
  Yeah, I don’t disagree. But in some sense , when could any photo be trusted ? We’ve had airbrushing, CGI forever now.
  - fortran77 3y ago
    Or just weird lighting and "Myspace Angles" (people photographing themselves from an angle above, etc)
mensetmanusman 3y ago
What you are looking at might have a lower compute count as the popularity grows and the GPU/User ratio shrinks. A GPU shortage under popularity growth should result in more obvious AI art over time sans an algorithm breakthrough that reduces compute cost by orders.

iamleppert 3y ago

Can’t wait to crack this. Looking at the specs, it’s hugely complicated for what it is and has an enormous attack surface.

patcon 3y ago
It may not be as stupid as it looks. If this is connected to what I understand, the tech has its provenance in Starling Lab: https://www.starlinglab.org/journalism-about/ https://www.starlinglab.org/journalism-about/ Starling Lab involves Dan Boneh.

jillesvangurp 3y ago

Generally, signing digital files is a great way to have some strong form of authenticity. There's a lot of unsigned content on the internet which makes impersonating people quite easy. We have fake news, fake comments, fake reviews, propaganda departments from intelligence agencies from hostile countries meddling with elections and public affairs, companies and lobby organizations spreading FUD, marketing companies specializing in bending public opinion, etc. There's a lot of good reason to start insisting of people using digital signatures for everything they produce. Even without considering AI. People obsess about what AIs can do here. But you might just make the point that at some point it starts rivaling what a determined hostile person could do manually right now. Sure, it adds scale and convenience for those with such intentions; which makes the problem more urgent than it already was. But it was pretty bad before that. Putin did not use AI a few years ago. He simply just put some teams of writers to work to flood the internet with content intended to bend Elections this way or that way or erode public confidence. Web of trust has been tried before of course but IMHO it deserves another chance. The current state is not sustainable when we might get AI generated content dwarfing the amount of genuine content very soon.

Retr0id 3y ago
I do think there's something useful here. It's like having photographers PGP sign all their images, but with an actually workable UX, and with keys (presumably) stored in a secure enclave within the camera body. This is good. I just think that marketing teams are inevitably going to oversell it. The moment someone posts an obviously fake image to social media, with all checks from this tech giving the green light, it will destroy the public's confidence in it. The actual usefulness of the tech is nuanced, and it will be a struggle to convey it meaningfully (to technical and non-technical audiences alike)
loceng 3y ago
What if what was recorded or photographed, you don't want your identity associated with it due to its contents - e.g. whistleblowing classified government/state documents exposing horrific crimes against humanity? If no digital signature then it's automatically fake? It doesn't seem to matter at the moment though anyhow, the corrupt establishment so far seems to be a good enough job suppressing what they don't want out and maintaining control.

throwawayForMe2 3y ago

Leica has a camera model out implementing this standard. https://leicacamerausa.com/leica-m11-p-in-silver-chrome.html https://leicacamerausa.com/leica-m11-p-in-silver-chrome.html

lemoncookiechip 3y ago

Sorry for the dumb question, but what is stopping anyone from taking a picture of an AI generated image and passing it up as the "real deal"?

Moldoteck 3y ago
Noise patterns from the camera will be off compared to what is in the image (I think)
waynenilsen 3y ago
Depth of field etc will be hard to fake converting from analog back to digital like this esp if the focal length etc are encoded in the committed data

mikewarot 3y ago

It's not new, they just waited for the Patents[1,2] to run out. It's been a long time since I worked on it. [1] https://patentimages.storage.googleapis.com/f7/e9/46/00ddfea701de89/US8219545.pdf https://patentimages.storage.googleapis.com/f7/e9/46/00ddfea... [2] https://patentimages.storage.googleapis.com/4f/e3/ed/a46504f8cc2998/US6757828.pdf https://patentimages.storage.googleapis.com/4f/e3/ed/a46504f...

JieJie 3y ago
Many comments suggest this won't be useful technology in that it will be easily circumvented for nefarious purposes. Would you mind commenting on your perception of the potential of this particular implementation of the technology you helped create?
- mikewarot 3y ago
  This is a way to sign an image while it's still in a camera. The signature is simply a way to strongly assert that an image is as the camera saw it in the raw. I saw it as a way for Law Enforcement to prove an evidence photo was not doctored.