48 ms·

I have a newborn at home, and like every other parent, we take thousands of pictures and videos of our newest family member. We took pictures of the very first

• deleted 5y ago
  [deleted]
• baal80spam 5y ago
  No, it doesn't work like that.
  • zionic 5y ago
    Yes it does, it uses fuzzy perceptual hashes not crypto hashes. So if your innocent baby pic looks similar enough to a previously tagged child abuse image then YES, it will flag you and send a copy to the feds. And before you correct me, the Apple employee will see a picture of your naked baby and hit “forward to NCMEC”, which… upon investigation is actually just the feds
    • outworlder 5y ago
      IF there are multiple matches, IF it's going to icloud, THEN a 'derivative image' will be show for screening and IF deemed to be warranted, sent to NCMEC.
    • nicce 5y ago
      Perceptual hashes are extremly accurate. You might need a twin and top of that somehow identical environment almost in pixel level. Are news filled with false-positive accusations by PhotoDNA, flagging wrong images in Google, Facebook, Instagram etc.?
      • heavyset_go 5y ago
        > Perceptual hashes are extremly accurate. No, they are not. Ask anyone who has worked in this space[1][2], including myself. They are incredibly common. Two images that kind of look like one another will have similar or the same hashes. That is the point of perceptual hashing. [1] https://news.ycombinator.com/item?id=28091750 https://news.ycombinator.com/item?id=28091750 [2] https://news.ycombinator.com/item?id=28110159 https://news.ycombinator.com/item?id=28110159
• slownews45 5y ago
  If you don't choose upload to icloud, no upload to apple at all. If you do choose icloud upload (most do), they were being uploaded already and stored and may be available to law enforcement. If you do upload to icloud, NOW they will be screened for matches with "known" images in a database, and if you have more than a threshold number of hits, you may be reported. This will happen on device. Apple will also scan photos in their cloud system as well from what I can tell (though once on device is working less should land in cloud). Note that it is HIGHLY likely that google photos / facebook / instagram and others will or are already doing similar scanning and reporting. I've heard millions of reports go in a year.
  • cwkoss 5y ago
    Arent the perceptual hashes based on a chunk of the image? I wonder what the false positive rates are for: - A random image against the DB of perceptual hashes - Images of a baby's skin against the DB of perceptual hashes It seems like the second would necessarily have a higher false positive rate: similar compositions (contains baby's skin) would more likely have similar chunks. Is it just a little higher or several orders of magnitude higher? I know hash collisions are rare, but wonder how much rarity of collisions decreases with perceptual hashes.
    • fossuser 5y ago
      It's two factors, both the match on an image hash and an unknown threshold of matches at which point the data gets sent up. If the threshold is not met then nothing gets notified (even if there is a match). Arguably this is why this approach is better for privacy. Cloud matches would not be able to have this extra threshold (in addition to this model allowing e2ee on the cloud in the future). I'd also like to know more about the specifics here, my guess is that threshold value is pretty high (their 'one in a trillion' comment not withstanding). It's probably targeting large CSAM dumps of matches which would not get flagged by different images.
    • slownews45 5y ago
      Absolute - I think this is one of two key questions for me. That is why I put "known" in quotes. It can't be an exact match because it has to handle cropping, rotation, resize etc. Images then do get a manual review before a report is made which is good and may help provide feedback on alogs being used. Going to be hard though for apple to set the second factor to high - I'd say 5 maybe? It's hard to say you had matches on potential CASM and ignored them I'd think.
• fossuser 5y ago
  It's worth reading this, which is basically the only good reporting I've seen on this topic: https://daringfireball.net/2021/08/apple_child_safety_initiatives_slippery_slope https://daringfireball.net/2021/08/apple_child_safety_initia... There are legitimate things to be concerned about, but 99% of internet discussion on this topic is junk.
  • Spooky23 5y ago
    The EFF wrote a really shitty hit piece deliberately confused the parental management function with the matching against hashes of illegal images. Two different things. From there, a bazillion hot takes followed.
    • dathinab 5y ago
      Two different things which are sold as one package by Apple. Two different things which both are known to be prone to all kind of miss-detection.
    • rootusrootus 5y ago
      Yeah I found the EFF's piece to be really disappointing, coming from an organization I'm otherwise aligned with nearly 100% of the time.
      • shapefrog 5y ago
        EFF today is really not the organisation it was just a few years ago. I dont know who they hired badly, but the reasoned takedowns have been replaced with hysterical screaming.
        • matwood 5y ago
          > hysterical screaming Given the political/societal climate, it probably gets them more donations.
        • xdennis 5y ago
          > hysterical screaming Ah, I see you work for Apple, given the language ("the screeching voices of the minority").
• samename 5y ago
  Unless those pictures are also in the NCMEC database, there won’t be a match.* * As addressed in the comments below, this isn’t entirely true: the hash looks for visually similar picture and there may be false positives.
  • gambiting 5y ago
    Absolutely not true. Apple is using a similarity based hash, so if the NCMEC database contains a picture that's similar to one that you have, it could produce a match even if it's not the same. Apple says this isn't an issue, because a person will look at your picture(yes, a random person somewhere will look at the pictures of your newborn) and judge whether they are pictures of child abuse or not. If this unknown person thinks your picture shows child abuse, you will be reported to NCMEC and then what happens is unknown - but likely that it would result in some legal action against you.
    • lawkwok 5y ago
      Keep in mind, this manual review only happens after Apple’s system detects multiple occurrences of matches. Until that point, no human is alerted of matches nor does anyone see how many matches there have been. In a TechCrunch interview Apple said that they are going after larger targets that are worth NCMEC’s time.
      • zionic 5y ago
        Parents take a lot of photos of their kid. Like, lots.
        • simondotau 5y ago
          How many of them include erect adult penises and active participation in sex acts? Apple's on-device list of hashes only includes images which have been classified "A1" under the CSAM categorisation scale. If any other photographs are accidental hash collisions to these images, it's going to be pretty damn obvious to the human reviewer.
• deleted 5y ago
  [deleted]
• stevenicr 5y ago
  from my current understanding - that does occur with m-soft one-drive which is a default in many systems), but not the hash-looking thing apple is currently proposing.
• dathinab 5y ago
  Unlikely except if you send them to a iphone which is registered with a "child" account. Apple uses two different approaches: 1. Some way to try to detect _known_ child pornographic material, but it's fuzzy and there is no guarantee that it doesn't make mistakes like detecting a flower pot as child porn. But the chance that your photos get "miss detected" as _known_ child pornographic material shouldn't be too high. BUT given how many parents have IPhones it's basically guaranteed to happen from time to time! 2. Some KI child porn detection on child accounts, which is not unlikely to labile such innocent photos as child porn.
  • fossuser 5y ago
    Even in the child account case it's not sent to Apple - it alerts parent accounts in the family. It's also just nudity generally, more akin to garden variety parental control content filtering. The child account iMessage thing is really entirely separate from the CSAM related iCloud announcement. It's unfortunate people keep confusing them.
• s5300 5y ago
  So... this is the way I understand it, which the general public will never have the attention span to understand, so it doesn't fucking matter one bit. LEO's/FBI/every other institution/group that deals with child pornography and abuse have teams that go through a near infinite amount of pictures and videos of CP/etc. These are then marked by said people as either - yes, CP/Abuse/etc - or marked false positive. Once marked as what they're after, they're uploaded to a shared database between all groups involved. Only what is in these worldwide national databases is what's going to be checked against. Your new pictures of your children will have obviously never made their way to any of these groups as they've never been shared/distributed in any areas of the internet/etc these people work in to track down trafficking rings (well, I'd hope you're not selling pictures of your children to them). This is the way I understand it. I admit I haven't looked into it that much. If it's anything different than what I've said, then yeah, it's probably fucked. I don't get what people don't understand about checking against a database though. No, your new pictures of whatever are not in this pre-existing database
• bouncycastle 5y ago
  My understanding is that you should not upload these photos to the cloud anyway. The cloud is not your computer and who knows, maybe apple engineers might be snooping on them, or there could be a hack and so on..Putting on the cloud is like sharing with Apple.
• dev_tty01 5y ago
  No. The CSAM (Child Sexual Abuse Material) scanning is comparing hashes of photos about to be uploaded to iCloud against a specific set of images at NCMEC (National Center for Missing and Exploited Children) which are specific to missing and exploited children. It is not machine learning models looking for nudes or similar. It is not a generalized screening. If enough matched images are found, the images are flagged for manual verification. If the manual verification confirms that the images match specific images in the NCMEC database, law enforcement is informed. Be aware that almost all cloud providers screen photos. Facebook reported 20 million images in 2020, Google reported half a million. Dropbox, Box, and many, many others report images. See https://www.missingkids.org/content/dam/missingkids/gethelp/2020-reports-by-esp.pdf https://www.missingkids.org/content/dam/missingkids/gethelp/... to see a complete list of companies that screen and report images. The other thing Apple announced which is completely separate from the CSAM photo scanning is additional parental controls for the Messages app. If a parent opts in for their under-13 children, a machine learning model will look for inappropriate material and warn the child prior to showing the image. The child is also told that their parent will be flagged if the child looks at it anyway. For 13-18 year olds whose parents opted in, the teen is warned first about the content. If the teen continues past the warning the image is shown and no further action is taken. Parents are not flagged for children 13 and over. As I said, this is a parental control for pre-adult kids. It requires opt-in from the parents and has no law enforcement implications.
  • ok123456 5y ago
    The correct answer is a well qualified "Maybe." The hashes are fuzzy AI generated weights. It's impossible to know what will cause a false-positive.
  • aczerepinski 5y ago
    Comparing hashes reminds me of this announcement from a few years ago that Google had produced a SHA1 collision: https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html https://security.googleblog.com/2017/02/announcing-first-sha... Can you imagine the chaos of a successful collision matching some explicit material being sent as a prank or targeted attack?
• sandworm101 5y ago
  Yes, if they wind up part of a child porn investigation. Your cloud account gets hacked. Some perv gets your images. He is then arrested and his "collection" added to the hash database... including your family photos. Context often matters more than the nature of the actual content. Police aquire thousands of images with little hope of ever knowing where they originated. If they are collected by pervs, and could be construed as illegal in the hands of pervs, the images become child porn and can be added to the databases.
  • snowwrestler 5y ago
    It’s worth pointing out that this could happen with any Internet-attached photo storage, and pre-dates Apple’s announcement. What Apple announced is a new system for reading the existing hash lists of known CSAM images and doing the comparison on the device as part of the iCloud upload, rather than on the server after upload.
  • nicce 5y ago
    Actually, we don’t know yet whether you can access your photos from the web anymore after this update, because E2EE ”like” implementation. Protocol is rather device specific (while allowing multi-device), so it might not be enough to access or hack iCloud account to access the photos. So, things get complicated.
    • vineyardmike 5y ago
      > because E2EE ”like” implementation. Did apple actually say photos would be e2ee or are we just assuming?
      • nicce 5y ago
        Did you read the spec? This is why the scanning happens on the device.
        • vineyardmike 5y ago
          I don't recall them explicitly saying anything was e2ee except imeessage which is not relevant for this discussion.
• fortenforge 5y ago
  Lots of people responding to this seem to not understand how perceptual hashing / PhotoDNA works. It's true that they're not cryptographic hashes, but the false positive rate is vanishingly small. Apple claims it's 1 in a trillion [1], but suppose that you don't believe them. Google and Facebook and Microsoft are all using PhotoDNA (or equivalent perceptual hashing schemes) right now. Have you heard of some massive issue with false positives? The fact of the matter is that unless you possess a photo that exists in the NCMEC database, your photos simply will not be flagged to Apple. Photos of your own kids won't trigger it, nude photos of adults won't trigger it; only photos of already known CSAM content will trigger (and that too, Apple requires a specific threshold of matches before a report is triggered). [1] "The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account." Page 4 of https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
  • deleted 5y ago
    [deleted]
  • deleted 5y ago
    [deleted]
  • shawnz 5y ago
    The 1 trillion figure is only after factoring in that you would need multiple false positives to trigger the feature. It's not descriptive of the actual false positive rate of the hashing itself.
  • FabHK 5y ago
    Probability of a false positive for a given image = p Probability of N false positives (assuming independence) = p^N Threshold N is chosen by Apple such that p^N < 10^-12, or N log p < -12 log 10, or N > -12 log(10)/log(p) [since log(p) < 0, since p < 1]. ETA: Suppose, just for the sake of the argument, that p = 10^-3 (one false positive in 1000, so really quite bad). Then log(p) = -3 log(10), so N > -12 log(10)/(-3 log(10)) = 12/3 = 4. Similarly, if p is one in a million (10^6), then N would be required to be > 12/6 = 2. In practice, I'd expect N to be larger than 4, in other words, Apple being very conservative here. ETA: The above doesn't take into account how many images M you have. The analysis gets more complicated, but N needs to be way larger than 4. I'll think about it some more.
• 908B64B197 5y ago
  Wait until someone manages to create an image (white noise) that's a hash collision for anything in that database. And then starts spamming random strangers via airdrop. Enjoy explaining why your mugshot and arrest record had these charges attached to it! (Actually, in this case the prosecution would probably use the other pictures on the phone that were not detected by the scanning tool as a way to get a guilty plea deal!)
  • FabHK 5y ago
    It would have to be a number of pictures that are flagged, and after that threshold is exceeded, they (more precisely, their "visual derivative") are reviewed by a human. So, no mugshot and no arrest record, even if you choose to accept any number of pictures sent from random strangers via airdrop.
  • GuB-42 5y ago
    Assuming it is possible (I think it is), there is a manual verification process if you have a match. And obviously, the white noise will be rejected, like all pictures that do not look remotely like the original. But it can be a form of denial of service: saturate the system with hash collisions so that people can't keep up.
• migueldeicaza 5y ago
  No, this will never be caught. This only catches ownership of illegal photos.
• darkhorn 5y ago
  Just don't critisize your government in any way. Otherwise they will find anything illegal to arrest you like from crossing street in red to I don't know what. You will be fine becouse there is a legal system and no one can put you into jail for crimes you have not commited. Just look at Julian Assange or random Joe in Belarus who was arrested for wearing red white hat. The justice system always is in the innocent people's side, without exception.
• devwastaken 5y ago
  Eventually, yes. As we work more towards AI's and "smart" image recognition eventually there will be a system that will have these false positives on innocent images. Current CSAM depends on humans to "verify" the imagery, this is something companies desperately want to get rid of, and so do the employees understandbly so. Nobody wants a job (99.99%) of comparing CSAM. It costs companies money in labor costs, and draws them bad PR when those employees inevitably develop permanent/semi-permanent mental health issues from it. The only reason it hasn't happened yet is because a startup can't just start scanning CSAM. They need the blessing of the feds to do that, which requires political connections, and of course requires competing with companies that already have that blessing - something that politics prevents. PhotoDNA and current CSAM scanning only gets known CSAM, but not new CSAM. The end goal is to detect CSAM before it's ever even distributed, to be "closer to the victim", rather than just those consuming it. Even with current PhotoDNA you can generate hash collisions, which flag the image for review, and a real human compares the material. This is of course subject to change for the reasons stated above. Secondarily, automatic scanning and ID'ing of imagery is how you can easily throw an FBI raid at someone. Apps like Telegram automatically download every image/video in the thread. Ontop of that you can create images that appear different at differing resolutions. At one resolution a harmless meme, at another, CSAM. Meaning that you can again throw an FBI raid at someone using simple tricks.
• bryanrasmussen 5y ago
  no, because it only catches registered CSAM, however if you sent your pictures to relatives etc. and somehow someone who was into CSAM got one of your pictures and later gets arrested, your photo in their collection could theoretically be registered in the official archives - then you might have something that matches one of the hashes of a known CSAM image in your collection of images (maybe enough matches to have the police come talk to you) on edit: later on of course this will make a great article in some place like the Atlantic with a stolid monochromatic picture of your family in the lead-in and we will all read about it on HN and talk about how this was an obvious problem with the whole system (if it gets posted at the right time and gets enough upvotes).