6 ms·

> that a tool designed to stop the spread of CSAM is a "bad thing" It's certainly said to be designed to do it, but have you seen concerns raised in the other

> that a tool designed to stop the spread of CSAM is a "bad thing" It's certainly said to be designed to do it, but have you seen concerns raised in the other thread (https://news.ycombinator.com/item?id=28068741 https://news.ycombinator.com/item?id=28068741)? There have been reports from some commenters of the NCMEC database containing unobjectionable photos because they were merely found in a context alongside some CSAM. Who audits these databases? Where is the oversight to guarantee only appropriate content is included? They are famously opaque because the very viewing of the content is illegal. So how can we know that they contain what they are purported to contain? This is overreach.

Shank 5y ago
> Who audits these databases? Where is the oversight to guarantee only appropriate content is included? They are famously opaque because the very viewing of the content is illegal. So how can we know that they contain what they are purported to contain? I wholeheartedly agree: there is an audit question here too. The contents of the database are by far the most dangerous part of this equation, malicious or not, targeted or not. I don't like the privacy implications about this, nor the potential for abuse. I would love to see some kind of way to audit the database, or ensure that it's only used "for good." I just don't know what that system is, and I know that PhotoDNA is already in use on other cloud providers. Matthew Green's ongoing analysis [0] is really worth keeping an eye on. For example, there's a good question: can you just scan against a different database for different people? These are the right questions given what we have right now. [0]: https://twitter.com/matthew_d_green/status/1423378285468209153 https://twitter.com/matthew_d_green/status/14233782854682091...
- shuckles 5y ago
  Matt should read the release before live tweeting FUD. The database is shipped in the iOS image, per the overview, so targeting users is not an issue (roughly).
  - pseudalopex 5y ago
    You should understand the difference between a protocol and an easy to change implementation detail before throwing around words like FUD.
    - shuckles 5y ago
      Has Matt redacted any of the FUD from his tweets last night which aren’t true given the published details from today? For example, his claim that the method is vulnerable to black box attacks from GANs isn’t applicable to the protocol because the attacker can’t access model outputs. Furthermore, if “an easy to change implementation detail” in your threat model is anything which could be changed by iOS update, you should’ve stopped using iPhone about 14 years ago.
      - deleted 5y ago
        [deleted]
  - pushrax 5y ago
    Is the database frozen or can they push out updates independently of iOS updates? If not, targeting individual users definitely doesn't seem possible unless you control OS signing.
    - shuckles 5y ago
      The database is shipped in the iOS image.
      - pushrax 5y ago
        That's what you wrote originally - and to me it doesn't indicate whether it can also be updated from other sources or not. Lots of content is shipped in the iOS image but can update independently.
        shuckles 5y ago
- cwkoss 5y ago
  Do the people authorized to query the database have access to view its contents? How many people in the world can certify that it only contains CSAM? I wonder if there are images of US troops committing warcrimes, or politicians doing cocaine, or honeypot intel op blackmail images in there too. Lots of powerful people would love to have an early warning system for leaks of embarrassing non-CSAM images.
shuckles 5y ago
That’s a problem with NCMEC, not Apple’s proposal today. Furthermore, if it were an actual problem, it would’ve already manifested with the numerous current users of PhotoDNA which includes Facebook and Google. I don’t think the database of known CSAM content includes photos that cannot be visually recognized as child abuse.
- tsimionescu 5y ago
  Why do you not think that? As far as I understand, there is no procedure for reviewing the contents, it is simply a database that law enforcement vouches is full of bad images.
  - shuckles 5y ago
    NCMEC, not law enforcement, produces a list of embeddings of known images of child abuse. Facebook and Google run all photos uploaded to their platforms against this list. Those which match are manually reviewed and if confirmed to depict such scenes, are reported to CyberTip. If the list had a ton of false positives, you think they wouldn’t notice that their human reviewers were spending a lot of time looking at pictures of the sky?
    - suizi 5y ago
      It's well-known that this algorithm doesn't have a perfect matching rate. It'd be easy to presume that any false positives are not erroneously tagged images, but the error rate of the underlying algorithm, if all the images were tagged correctly. Who would know? IIRC Wired reported the algorithm "PhotoDNA" worked around 99% of the time a number of years ago, however newer algorithms may be fuzzier. This is not the same algorithm. And even "PhotoDNA" appears to change over time. I doubt reviewers of such content are at a liberty to discuss what they see or don't with anyone here. Standard confidentiality agreements.
- cwkoss 5y ago
  > it would’ve already manifested How do we know it hasn't? Maybe the people who have seen these situations have received to national security letters that claim to prevent them from even talking to a lawyer about the non-CSAM classified images they've seen in the course of the investigation?
oh_sigh 5y ago
"Reports from commenters" = unsubstantiated speculation. Weird how no one was able to specifically state any information about these unobjectionable photos except for a theoretical mechanism for them to find their way into the database.
- feanaro 5y ago
  Some have been able to give specific information about these photos: https://news.ycombinator.com/item?id=28071047 https://news.ycombinator.com/item?id=28071047 Yes, of course, it's "unsubstantiated speculation", but how do you suppose the speculation be substantiated if the entire process and database are not available for audit? That's exactly the problem with this.
  - oh_sigh 5y ago
    That isn't specific information - that is more speculation. I expect people who make positive claims (the database contains innocuous images) to be able to back up those claims. Saying that there are innocuous images in the database is completely different from saying there may be innocuous images in the database. It would be like if I claimed that there is a burglar in your house - and how do I know? Well all a burglar would have to do is break your window and crawl in.
neop1x 5y ago
>> Who audits these databases? Maybe pedophiles working for those companies.

Shank 5y ago

> Who audits these databases? Where is the oversight to guarantee only appropriate content is included? They are famously opaque because the very viewing of the content is illegal. So how can we know that they contain what they are purported to contain? I wholeheartedly agree: there is an audit question here too. The contents of the database are by far the most dangerous part of this equation, malicious or not, targeted or not. I don't like the privacy implications about this, nor the potential for abuse. I would love to see some kind of way to audit the database, or ensure that it's only used "for good." I just don't know what that system is, and I know that PhotoDNA is already in use on other cloud providers. Matthew Green's ongoing analysis [0] is really worth keeping an eye on. For example, there's a good question: can you just scan against a different database for different people? These are the right questions given what we have right now. [0]: https://twitter.com/matthew_d_green/status/1423378285468209153 https://twitter.com/matthew_d_green/status/14233782854682091...

shuckles 5y ago
Matt should read the release before live tweeting FUD. The database is shipped in the iOS image, per the overview, so targeting users is not an issue (roughly).
- pseudalopex 5y ago
  You should understand the difference between a protocol and an easy to change implementation detail before throwing around words like FUD.
  - shuckles 5y ago
    Has Matt redacted any of the FUD from his tweets last night which aren’t true given the published details from today? For example, his claim that the method is vulnerable to black box attacks from GANs isn’t applicable to the protocol because the attacker can’t access model outputs. Furthermore, if “an easy to change implementation detail” in your threat model is anything which could be changed by iOS update, you should’ve stopped using iPhone about 14 years ago.
    - deleted 5y ago
      [deleted]
- pushrax 5y ago
  Is the database frozen or can they push out updates independently of iOS updates? If not, targeting individual users definitely doesn't seem possible unless you control OS signing.
  - shuckles 5y ago
    The database is shipped in the iOS image.
    - pushrax 5y ago
      That's what you wrote originally - and to me it doesn't indicate whether it can also be updated from other sources or not. Lots of content is shipped in the iOS image but can update independently.
      - shuckles 5y ago
cwkoss 5y ago
Do the people authorized to query the database have access to view its contents? How many people in the world can certify that it only contains CSAM? I wonder if there are images of US troops committing warcrimes, or politicians doing cocaine, or honeypot intel op blackmail images in there too. Lots of powerful people would love to have an early warning system for leaks of embarrassing non-CSAM images.

shuckles 5y ago

That’s a problem with NCMEC, not Apple’s proposal today. Furthermore, if it were an actual problem, it would’ve already manifested with the numerous current users of PhotoDNA which includes Facebook and Google. I don’t think the database of known CSAM content includes photos that cannot be visually recognized as child abuse.

tsimionescu 5y ago
Why do you not think that? As far as I understand, there is no procedure for reviewing the contents, it is simply a database that law enforcement vouches is full of bad images.
- shuckles 5y ago
  NCMEC, not law enforcement, produces a list of embeddings of known images of child abuse. Facebook and Google run all photos uploaded to their platforms against this list. Those which match are manually reviewed and if confirmed to depict such scenes, are reported to CyberTip. If the list had a ton of false positives, you think they wouldn’t notice that their human reviewers were spending a lot of time looking at pictures of the sky?
  - suizi 5y ago
    It's well-known that this algorithm doesn't have a perfect matching rate. It'd be easy to presume that any false positives are not erroneously tagged images, but the error rate of the underlying algorithm, if all the images were tagged correctly. Who would know? IIRC Wired reported the algorithm "PhotoDNA" worked around 99% of the time a number of years ago, however newer algorithms may be fuzzier. This is not the same algorithm. And even "PhotoDNA" appears to change over time. I doubt reviewers of such content are at a liberty to discuss what they see or don't with anyone here. Standard confidentiality agreements.
cwkoss 5y ago
> it would’ve already manifested How do we know it hasn't? Maybe the people who have seen these situations have received to national security letters that claim to prevent them from even talking to a lawyer about the non-CSAM classified images they've seen in the course of the investigation?

oh_sigh 5y ago

"Reports from commenters" = unsubstantiated speculation. Weird how no one was able to specifically state any information about these unobjectionable photos except for a theoretical mechanism for them to find their way into the database.

feanaro 5y ago
Some have been able to give specific information about these photos: https://news.ycombinator.com/item?id=28071047 https://news.ycombinator.com/item?id=28071047 Yes, of course, it's "unsubstantiated speculation", but how do you suppose the speculation be substantiated if the entire process and database are not available for audit? That's exactly the problem with this.
- oh_sigh 5y ago
  That isn't specific information - that is more speculation. I expect people who make positive claims (the database contains innocuous images) to be able to back up those claims. Saying that there are innocuous images in the database is completely different from saying there may be innocuous images in the database. It would be like if I claimed that there is a burglar in your house - and how do I know? Well all a burglar would have to do is break your window and crawl in.

neop1x 5y ago

>> Who audits these databases? Maybe pedophiles working for those companies.