3 ms·

It didn't even last a week. Introducing the people asking us to trust them with the responsibility of mass, automated crime accusations.

hannasanarion 5y ago
Why do you think somebody will accuse you of a crime because you have a photo of a grey blob? The real world isn't as stupid as the computer one. The justice system is not deterministic and automatic. Nobody is going to look at this grey blob and go "welp, we have no choice but to throw you in prison forever"
- argvargc 5y ago
  The real world and the computer world intersect. This is precisely typified by what is being discussed, surely? Apple is trying to automate and computerise a process that was not automated previously, apply it to a huge number of people, and with disastrous potential consequences should their wonderful design be found lacking. And within days, they have already utterly failed to provide one of their own self-stated and incredibly obvious requirements. What else could possibly go wrong? Further, if you think that because the first pre-image failure found (in days...) was a grey blob, that this means all future possible cultivated collisions will only ever be grey blobs, well - good luck with that.
  - hannasanarion 5y ago
    The process of making accusations and arrests is not automatic. The Apple system sends flagged photos to reviewers, and if the reviewers find them suspicious, they send them to the poor souls at NCMEC, who will compare the flagged photo with the original illegal photo that's supposedly a match, and inform law enforcement if they are in fact a match. Nobody will get cops at their door because somebody sent them a grey blob image. The process that's being "automated" is merely an automatic flag that initiates a several-step process of human review. Apple isn't rolling out robocops. Seriously, what are the "disasterous consequences" that you envision? What is the sequence of events where a hash collision leads to any inconvenience whatsoever for a user?
    - argvargc 5y ago
      What part of "they already totally fucked up how their own process is supposed to work" don't you understand?
      - hannasanarion 5y ago
        What part of it is fucked up? They never promised that their hashing algorithm was uncollidable. The process is specifically designed to be tolerant of hash collisions (and in fact wouldn't work otherwise, because the system needs to ignore small differences between copies of the illegal images, like one-pixel edits or color temperature differences or photocopies). There are multiple stages of human review in the process and a high threshold for activation of the proccess because collisions are inevitable. The fact that collisions exist doesn't impact the safety of the product whatsoever. I ask again, what is the sequence of events where a hash collision from a benign image can lead to any consequence at all for the user?
        argvargc 5y ago
        "Apple said that there is a one in one trillion chance of a false positive" https://techcrunch.com/2021/08/05/apple-icloud-photos-scanning/ https://techcrunch.com/2021/08/05/apple-icloud-photos-scanni... Versus: https://blog.roboflow.com/nerualhash-collision/ https://blog.roboflow.com/nerualhash-collision/ Along with: https://github.com/anishathalye/neural-hash-collider https://github.com/anishathalye/neural-hash-collider Demonstrates blatant incompetence on the part of Apple's design. At this point, trust is broken. That's the end of credibility regarding any other claim made about the security or base function of the system. As for a simple example of one of many potential attack vectors (search HN for "neuralhash" and read the comments for countless examples), here is a basic attack: As Apple's hash algorithm is entirely broken and useless, a malicious actor can easily craft a NSFW, but otherwise legal image, that has a hash matching one in the CSAM database, that is visually ambiguous in that context (for example, a close-up). Targeted attacks could be launched on unsuspecting victims as simply as sending them an image. If their app is configured to auto-save to iCloud - as some are by default, and as many are configurable to be - the image will then get automatically flagged as a match, and may well potentially pass a human review - because the image may indeed look just like CP - and an innocent person, who may not even be aware of the presence of the image in their iCloud library at this point, may get visited by the police. That in itself is bad enough. However it gets worse - do bear in mind that "the police" is not an equivalent concept across nations, yet iPhones are ubiquitously used across the world. In certain countries, a "visit from the police" over such automatically-flagged content could well result in presumption of guilt to the point of immediately administered punishment and a destroyed innocent life. In China for example, conviction rates are routinely 99.99% - being accused of a crime there is equivalent to being convicted, and that's just one major country containing well over 100 million iphone users. Apple is demonstrating not only gross incompetence in this design of this "system", but vagrant disregard for human rights, along with utterly destroying their own long-cultivated pro-privacy stance. It's a disaster for them and their customers in every possible way it could be.

hannasanarion 5y ago

Why do you think somebody will accuse you of a crime because you have a photo of a grey blob? The real world isn't as stupid as the computer one. The justice system is not deterministic and automatic. Nobody is going to look at this grey blob and go "welp, we have no choice but to throw you in prison forever"

argvargc 5y ago
The real world and the computer world intersect. This is precisely typified by what is being discussed, surely? Apple is trying to automate and computerise a process that was not automated previously, apply it to a huge number of people, and with disastrous potential consequences should their wonderful design be found lacking. And within days, they have already utterly failed to provide one of their own self-stated and incredibly obvious requirements. What else could possibly go wrong? Further, if you think that because the first pre-image failure found (in days...) was a grey blob, that this means all future possible cultivated collisions will only ever be grey blobs, well - good luck with that.
- hannasanarion 5y ago
  The process of making accusations and arrests is not automatic. The Apple system sends flagged photos to reviewers, and if the reviewers find them suspicious, they send them to the poor souls at NCMEC, who will compare the flagged photo with the original illegal photo that's supposedly a match, and inform law enforcement if they are in fact a match. Nobody will get cops at their door because somebody sent them a grey blob image. The process that's being "automated" is merely an automatic flag that initiates a several-step process of human review. Apple isn't rolling out robocops. Seriously, what are the "disasterous consequences" that you envision? What is the sequence of events where a hash collision leads to any inconvenience whatsoever for a user?
  - argvargc 5y ago
    What part of "they already totally fucked up how their own process is supposed to work" don't you understand?
    - hannasanarion 5y ago
      What part of it is fucked up? They never promised that their hashing algorithm was uncollidable. The process is specifically designed to be tolerant of hash collisions (and in fact wouldn't work otherwise, because the system needs to ignore small differences between copies of the illegal images, like one-pixel edits or color temperature differences or photocopies). There are multiple stages of human review in the process and a high threshold for activation of the proccess because collisions are inevitable. The fact that collisions exist doesn't impact the safety of the product whatsoever. I ask again, what is the sequence of events where a hash collision from a benign image can lead to any consequence at all for the user?
      - argvargc 5y ago
        "Apple said that there is a one in one trillion chance of a false positive" https://techcrunch.com/2021/08/05/apple-icloud-photos-scanning/ https://techcrunch.com/2021/08/05/apple-icloud-photos-scanni... Versus: https://blog.roboflow.com/nerualhash-collision/ https://blog.roboflow.com/nerualhash-collision/ Along with: https://github.com/anishathalye/neural-hash-collider https://github.com/anishathalye/neural-hash-collider Demonstrates blatant incompetence on the part of Apple's design. At this point, trust is broken. That's the end of credibility regarding any other claim made about the security or base function of the system. As for a simple example of one of many potential attack vectors (search HN for "neuralhash" and read the comments for countless examples), here is a basic attack: As Apple's hash algorithm is entirely broken and useless, a malicious actor can easily craft a NSFW, but otherwise legal image, that has a hash matching one in the CSAM database, that is visually ambiguous in that context (for example, a close-up). Targeted attacks could be launched on unsuspecting victims as simply as sending them an image. If their app is configured to auto-save to iCloud - as some are by default, and as many are configurable to be - the image will then get automatically flagged as a match, and may well potentially pass a human review - because the image may indeed look just like CP - and an innocent person, who may not even be aware of the presence of the image in their iCloud library at this point, may get visited by the police. That in itself is bad enough. However it gets worse - do bear in mind that "the police" is not an equivalent concept across nations, yet iPhones are ubiquitously used across the world. In certain countries, a "visit from the police" over such automatically-flagged content could well result in presumption of guilt to the point of immediately administered punishment and a destroyed innocent life. In China for example, conviction rates are routinely 99.99% - being accused of a crime there is equivalent to being convicted, and that's just one major country containing well over 100 million iphone users. Apple is demonstrating not only gross incompetence in this design of this "system", but vagrant disregard for human rights, along with utterly destroying their own long-cultivated pro-privacy stance. It's a disaster for them and their customers in every possible way it could be.