20 ms·

Apple plans to scan US iPhones for child abuse imagery

372 points by alexanderklein 5y ago

TerntUp 5y ago
https://archive.ph/ys5Q5#selection-1501.0-1501.54 https://archive.ph/ys5Q5#selection-1501.0-1501.54
1-6 5y ago
Apple should perhaps dogfood their system on their own employees first: https://www.losaltosonline.com/news/hills-man-arrested-on-child-porn-charges/article_9bbe51f7-aaa4-5814-98eb-a9a6b0f51a42.html https://www.losaltosonline.com/news/hills-man-arrested-on-ch...
- kwhitefoot 5y ago
  Not employees but all of their board and all of their investors.
  - maerF0x0 5y ago
    Also lets draft on all the children of senate and congress https://quotepark.com/quotes/1911588-immortal-technique-the-first-payback-that-i-would-accomplish-id-dra/ https://quotepark.com/quotes/1911588-immortal-technique-the-...
laputan_machine 5y ago
Ah, the classic 'think of the children' approach to increasing surveillance. If you're opposed to it, what are you, a nonce?
- Markoff 5y ago
  It worked pretty well for anyone opposing vaccination with technologies first time approved for use year (mRNA) or two years (viral vector) ago. Anyone not buying into these proven 1-2 years old technologies is crazy anti-vaxxer. Same reasoning can be used with anyone opposing scanning your private images, after all you have nothing to hide.
  - vimacs2 5y ago
    Except the difference is nudity is natural and sometimes instrumental - for washing, bathroom use, and medical care for example - and denying basic healthcare to your children is not, unless you plan to be instrumental in your child's demise and that comes natural to you.
    - skinkestek 5y ago
      I've taken all vaccines I've been offered including BioNTech/Pfizer but: carefulness regarding new medical procedures is also completely natural. We aren't robots and we need to cut one another some slack. Different persons being mire or less risk averse is even a strength for society.
      - vimacs2 5y ago
        I don't disagree there, I'm just pointing out that there is a difference between the two cases.
        leereeves 5y ago
        Your argument was wrong, though, because those vaccines aren't being given to children anyway.
        vimacs2 5y ago
        Reading back now, I realize my comment was indeed just dumb and borne from not properly reading the OP's comment and thinking of the anti-vaxxer movement in general, not the actually rational aversion to untested ones during the Covid crisis. I would have editing a disclaimer on it but it's too late.
        skinkestek 5y ago
        Here is what you wrote: > and denying basic healthcare to your children is not, unless you plan to be instrumental in your child's demise and that comes natural to you. You wrote that in reply to someone clearly not talking about vaccines generally but the new vaccines specifically.
- CiaTransMatters 5y ago
  Excuse me, sweetie, but "think of the trans" is a very important movement. It prevents suicides, helps bring gender equality, raises awareness, and creates tech jobs for trans activists to assist in stopping online transphobia.
- mhh__ 5y ago
  Well I was in a Pizza express in woking, move along.
imroot 5y ago
https://towardsdatascience.com/black-box-attacks-on-perceptual-image-hashes-with-gans-cc1be11f277?gi=cfb3a66600e5 https://towardsdatascience.com/black-box-attacks-on-perceptu... I mean, who's to say that my hash is actually what they think it is... ...and if it's not uploaded to iCloud for someone to manually review what the image is, is that going to be enough to get a search warrant for my phone? "AUSA SOMEONE: But, Your Honor, The Defendant's phone has an image that matched a hash of a CSAM picture!" "DEFENDANT: I'm not going to give you my phone's password." "THE COURT: Bonk: Stay in jail for contempt." I understand why they're doing it. I don't agree with it. At all, and I say that as a survivor of childhood sexual abuse.
- Crontab 5y ago
  I have always been concerned that this system could be weaponized as a way gain access to someone's account. For example: - Add the hash of a non-pornographic image to the database - Using a burner email address, email the non-pornographic image to the target's Gmail address. The target wouldn't think anything of it. - The innocent image would trigger a CP alert, giving law enforcement the pretense it needs to access the account
  - smsm42 5y ago
    This is how the police gets warrantless searches "for drugs" - having identified the subject, call for a dog and have it "alert" on the subject (or the vehicle belonging to the subject), after which there's a probable cause for a search and everything that is found can be used against the subject. The fact that there was no drugs is irrelevant. Now let's translate it to digital world and eliminate the non-scalable components - and you get AI "alerting" on the content of your account and it's a probable cause for searching you. But the best part is of course the Apple doesn't even need a probable cause to search you - they aren't the police, so they are free to search anything at their pleasure and then report to the police. And this report, by itself, would be the probable cause, of course.
- zug_zug 5y ago
  EDIT: It appears they are using perceptual hashes, which are likely much more prone to collisions. The following math does NOT apply to the tech Apple is using. >> I mean, who's to say that my hash is actually what they think it is... Let's do the math. assume they are using a UUID for the hash. "For example, the number of random version-4 UUIDs which need to be generated in order to have a 50% probability of at least one collision is 2.71 quintillion" [1] "The odds of This number is equivalent to generating 1 billion UUIDs per second for about 85 years." And that's for just 1 collision between two random photos. Considering that the pool of bad hashes is not going to be 2.7 quintillion but a fraction of that, this is more than adequate. And of course they could use a longer checksum if that's the concern. (Not to take a stance on the parent issue, just to explain that hash's are probably a higher standard of proof than DNA matches) 1. https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions https://en.wikipedia.org/wiki/Universally_unique_identifier#...
MeinBlutIstBlau 5y ago
Whose to say what is abuse? Are Caesar Milan tactics on a child abuse? This is outright dangerous beyond belief. Nobody is gonna wanna be a parent if you have to worry about your kid being mad at you and then posts something like that. IIRC there was a teenage girl that lied about her father molesting her back in the 2000s. No evidence, just pure hearsay. Dude was locked up for like 10 years. Now imagine that on a greater scale with several videos and photos taken out of context.
- HWR_14 5y ago
  That would be false testimony, not hearsay. Hearsay ("John Smith told me he heard MeinBlut kidnapped the Lindbergh baby") is not allowed. Witnesses lying about what they saw and or participated in is allowed, although it can be lies. Sorry, and as for "what is abuse", these are looking for exploited children. There are a lot of questions here, the morality of mass surveillance, the risk factors due to hash collisions (intentional or otherwise), etc, the one thing I doubt is an issue is a fuzzy line on "is this abuse"
  - cdolan 5y ago
    I don't think society can function without harsh penalties for child abuse, and I think thats a widely held belief everyone agrees with. I do think your comment is conflicting though. The fuzziness of 'is this abuse' is real because collisions can occur, mass surveillance never goes in reverse, etc.
    - codr7 5y ago
      Doesn't seem to be working very well from here; risk of penalty isn't going to stop most of these people, more likely make their game even more exciting. Of course they have to be prevented from hurting people, to the best of our ability to do so, but punishment isn't going anywhere.
    - HWR_14 5y ago
      Huh, I specifically said "concerns about collisions" is a real concern. I said that "mass surveillance" is a real concern. What is not a real concern is fuzziness in the question "is this image of child abuse". And I really don't want to continue a discussion where you try to come up with gray areas. A computer may not be able to tell the difference, but humans can.
xkcd-sucks 5y ago
So is the phash code actually going to run locally (I.e. can dump it and figure out the function) or is it only running on icloud?
- Rebelgecko 5y ago
  IIRC they've already been running it on icloud photos for a year or two. Hashing local files is the step they're taking now
mikece 5y ago
Scan for child abuse today; scan for wrong-think tomorrow. I'm liking the idea of Graphene OS more and more each day... or just going phone-free entirely.
- alexfromapex 5y ago
  We will not be able to have nice things until we get better privacy laws or come up with novel ways to prevent unwarranted snooping. It's sad that it's come to this.
tibbydudeza 5y ago
The dumbest idea ever -so glad that I am an Android user.
partiallypro 5y ago
I really don't see how this is going to end well, there could be perfectly innocent photos on someone's phone of their own children doing perfectly normal things that kids do. Like a kid running butt naked around the house, or a photo of something like a rash that is sent to a nurse friend for advice on what it is etc. I'm all for protecting children from being abused, but how are they going to filter what is normal and what is abuse without human intervention? And at that point isn't that a vast invasion of privacy to the perfectly innocent? It's different if it's online because that puts it in the public realm or on devices/servers owned by Apple, etc.
- cm2187 5y ago
  What I don't understand is how is it even constitutional in the US. Isn't it a warrant-less search? It's not like scanning content on a cloud. The cloud owns the device on which you store your data. But you own your mobile.
  - thrill 5y ago
    Not if you give permission, which you do by your purchase agreement.
    - BitwiseFool 5y ago
      You're correct, but I would love to see a TOS that gives Apple permission to exercise cruel and unusual punishment for violating their terms. The eighth amendment only applies to the government, after all! /s?
  - shuckles 5y ago
    The constitution constrains what the Federal government can do, not what private companies can do.
    - cm2187 5y ago
      Not that I am qualified to hold a legal debate, but as far as I know, if Apple implements this under the pressure from the state, and that it reports what it findings to the state, it is effectively an agent of the state and it should apply.
forgingahead 5y ago
https://archive.ph/ys5Q5 https://archive.ph/ys5Q5
ohgreatwtf 5y ago
The key item in the article is that if enough content is marked contentious apple will automatically decrypt your phone for you and disable your encryption. So apple's not merely helping law enforcement find contentious content, they are enabling a back door that makes phone encryption essentially swiss cheese. In 5 years: Apple and Google to scan your phones for politically contentious content and automatically remove it for you/report it to the authorities so you can get the proper help you need to understand what you should believe. I've loved my apple phones for many years now, but it seems this is the end of my relationship with apple. I will be using an open source phone next.
julkali 5y ago
https://news.ycombinator.com/item?id=28068741 https://news.ycombinator.com/item?id=28068741 This topic already made it to the current frontpage.
mkoryak 5y ago
Hopefully they are using phash to match the phone's images to known child abuse image hashes, because if they are using ML, I can only imagine what a false positive will do to someone's life.
- heavyset_go 5y ago
  Phashes can have collisions, and the false positive rate increases exponentially when you use a fuzzy metric like hamming distance to find derivatives of a source image.
KingMachiavelli 5y ago
This could backfire in a huge way. I think a lot of people, even EFF/privacy focused people are fine with a private company doing CSAM detection (or anything really) on uploaded user content since it is actually voluntary. Now that CSAM detection is being deployed to user devices including those previously sold without it, there's a motivation to circumvent/block it. Unless Apple is doing CSAM scanning 100% inside a 100% secure processor, eventually someone will figure out the specifics of the current perceptual hash algorithm. There are already methods to make perceptual hash collisions. All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless. Sure eventually they will improve the model/dataset but it just starts a cat & mouse game between Apple/Governments & honest citizens/users which actually benefits the people committing these crimes.
- luckylion 5y ago
  > All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless. And even better: someone could create false-positives and just send them to people, or put them on websites. While you're watching funny kittens tumbling around, your iphone is calling the cops on you because it thinks you're watching child porn.
  - slumdev 5y ago
    Like swatting, but worse.
  - mataug 5y ago
    A high rate of false positives will create distrust in the system, and ultimately make it less effective since no one will take it seriously.
    - dontblink 5y ago
      Could be used to target specific individuals and discredit them (e.g. critics of government).
chaganated 5y ago
as of noon, looks like most commenters disagree with apple's plan. apple will do it anyway. how is plantation life agreeing with everyone here today?
itake 5y ago
WhatsApp automatically saves photos sent to the phone to the Photo Album. Does this mean they will get a knock on their door if someone sent a bad image?
- seanicus 5y ago
  First thing I thought of too. A 'prank' that has far greater consequences for the target than swatting.
bastawhiz 5y ago
> Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system. Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse. It's weird and invasive that they'd search users devices, but this makes it sound like they're just searching for files with certain hashes, which (to me) is fairly uninteresting. What I don't understand is why this is called "neuralMatch". Is it some sort of perceptual hashing (versus cryptographic hashing)? What exactly makes it "neural"? If neuralMatch is to Google's reverse image search as Apple Maps was to Google Maps when it launched, this is going to have so many false positives. > According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities Is this separate? Or is the article just confusing iCloud storage for on-device storage? Regardless, there's some very lazy journalism going on here.
- alfalfasprout 5y ago
  The name is certainly concerning. It's one thing to check for specific hashes. It's another entirely to do approximate hashing. Locality sensitive hashing even in its best implementations has tons of false positives. It's also worrying that eg; someone could stealthily airdrop or iMessage a suspect photo to frame someone.
sandworm101 5y ago
As any of the lawyers here can attest, the legal definitions here open to huge debates. Some material is unambiguous, other images reside extensive grey areas. The legality of material can even change based on context. Something that is perfectly legal in one area can be absolutely illegal in another. Example: If I had to list the most iconic photographs in world history, one of them is of a totally nude child (Vietnam). That isn't an illegal image, but try explaining that to a computer. At the moment these nuances are handled by cops, prosecutors and in extreme cases the courts. Is Apple going to put a marker down on a particular universal definition? Will that definition become the de facto world standard?
- alkonaut 5y ago
  Who would add “gray area” pictures to the database of CP images that are matched?
  - sandworm101 5y ago
    Material that is deemed illegal in jurisdiction A but not in B. Material for which someone was once convicted but if found on someone else's device would not be illegal. Material isn't added to these databases after sober legal deliberation. Police agencies can add basically anything they see fit.
    - alkonaut 5y ago
      This is a system for just one jurisdiction though? Is there talk about doing this outside the US? For multiple jurisdictions you’d need different sets of known hashes.
mrkramer 5y ago
Imagine if Microsoft does this with Windows, lots of people would get arrested rightly so. But like some people say this can be dangerous because evidence can be planted on your device. I remember reading articles how through torrent clients files could be planted but I never actually saw it in the wild.
- TuringNYC 5y ago
  A good reminder to turn off AirDrop "Auto Accept from Everyone"
  - rudian 5y ago
    There is no such thing. AirDrop can "appear" to everyone, but not accept. At most you get a popup.
- em500 5y ago
  No need to imagine, it's already done in OneDrive which comes with Windows 10 by default: https://en.wikipedia.org/wiki/PhotoDNA https://en.wikipedia.org/wiki/PhotoDNA
Rebelgecko 5y ago
In a world where all police were incorruptible and all laws were just, maybe I'd be ok with this. But since we live in this world, I feel like it's only a matter of time until the "think of the children" technology is coopted to scan for other "bad" images. I wonder how long it'll take for China to compel Apple to add winnie_the_pooh.jpg to the set of naughty images for users in China and HK? And like we saw when bing altered the search results for "tank man" even in the countries like the US, at that point we'd be one configuration error away from those hashes leaking into the US dataset.
- r00fus 5y ago
  Forget just the government. Imagine if corporations could abuse regulatory structure to enforce copyright on private images?
- ksec 5y ago
  >I wonder how long it'll take for China... All iCloud Data in China are already within their control. The whole DataCenter are done with "partnership" agreement. HK user have their Data in US but I believe there could be a temporary server in HK DC before it is sent to US.
r00fus 5y ago
I bet Apple understands this is a bad idea (dilutes their "privacy" brand image) I wonder what forces have pushed Apple to this point.
- bitwize 5y ago
  Privacy was only a talking point to get them some advantage over the Eye of Sauron (Google). I have no doubt that they were ready to jettison their privacy protection marketing angle at a moment's notice. You can't have access to people's data in the cloud and on their phones, and not use it in certain ways. To do so would be to leave money, and the good will of the government, on the table.
- maerF0x0 5y ago
  perhaps they're being required by law and gagged for indicating as such https://theintercept.com/2018/06/25/att-internet-nsa-spy-hubs/#:~:text=Jim%20Greer%2C%20an,with%20the%20law.%E2%80%9D https://theintercept.com/2018/06/25/att-internet-nsa-spy-hub...
- ksec 5y ago
  >I bet Apple understands this is a bad idea That would be Steve Job's Apple. Tim Cook's Apple will think they are doing it for the greater good. The road to hell is paved with good intentions
throwaway287391 5y ago
> Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities. Haven't Apple previously claimed that their encryption works such that Apple themselves can't decrypt it no matter what? (IIRC this came up during the San Bernadino attack when the FBI requested that Apple decrypt/unlock a suspect's iPhone and Apple said they couldn't.) Is this not a change? Maybe I'm confusing iCloud encryption vs. iPhone encryption and the former doesn't have any of those guarantees?
- WrtCdEvrydy 5y ago
  iCloud backups are not encrypted, this is a major knowledge point.
- perryizgr8 5y ago
  > Apple themselves can't decrypt it no matter what? Apple has root on every iPhone. They can do whatever they want by pushing whatever code they like to any iphone.
runawaybottle 5y ago
It’s 2035, a major terrorist attack occurs. Apple scans all phones for location and images near the area over the last 12 months, sends the list over since the government declared Marshall law. You are now suspicious until proven not suspicious. The real terrorists don’t even use iPhones. But here we are. Edit: Can any legal-heads explain how this does not constitute unlawful search? Apple is not the government, but once they hand the information over, isn’t the government indirectly committing unlawful search? Don’t you need a warrant for this? ____________ Second Edit: Wanted to reply to another comment but being rate limited: As of June 2016, the Terrorist Watch List [No-Fly-List] is estimated to contain over 2,484,442 records, consisting of 1,877,133 individual identities. The number of Muslims in America: A 2017 study estimated that 3.35 million Muslims were living in the United States, about 1.1 percent of the total U.S. population. Lol. Jesus Christ, you do the math.
- kevin_thibedeau 5y ago
  > isn’t the government indirectly committing unlawful search? No. That's the beauty of the third-party doctrine. Your rights disappear in a puff of twisted logic because you agreed to the fine print.
kwhitefoot 5y ago
Better not take any more pictures of the kids splashing in the bath for granny then.
dehrmann 5y ago
> ...ongoing demands from governments, law enforcement agencies and child safety campaigners for more assistance in criminal investigations, including terrorism and child pornography. I get why they're making these demands, but this is in the US. This would clearly be an unreasonable search by Fourth Amendment standards if it were done by the government, so I'm not sure why government agencies think they can make demands that they, themelves can't legally execute of a private company. To be clear, it's legal for Apple to do this, and legal for the government to ask them to. The government is just laundering its door-to-door search.
- nullifidian 5y ago
  It's also indicative of the unhealthy relationship these giant semi-monopolistic companies have with the government(s), that they implement the government's supposed will out of the fear of anti-trust actions/legislation, without any need for the government to pass legislation. Same goes for the demands to curb "misinformation" of various kinds, which basically constitutes government mandated speech regulation, even though the standing jurisprudence clearly forbids it.
cptskippy 5y ago
Does this paywalled article have any further details? Aside from a cryptic twitter post that mentions CSAM, are there any further details? CSAM also means Cyber Security Asset Management.
kwhitefoot 5y ago
Are they going to compensate the owners for the CPU time, electricity and bandwidth this will cost?
- wayneftw 5y ago
  I asked a similar question in both threads here and both times I got downvoted by some courageous folks without an answer. I also would like to know if I already agreed to this and where can I read the agreement that I signed?
Someone1234 5y ago
If it matches, am I guilty until found innocent? I have no illicit images, but false positives are always going to be a problem and even at sub-1% rates if you're scanning literally every image stored on an iOS devices that could still be thousands of wrong matches. If I lose the false-positive lottery, am I going to have the police calling and have my mugshot in the evening news for "CP on their device" in particular in my state's months long backlog for digital forensics. Law enforcement aren't exactly known for their understanding of technical evidence like hash-matching. I'm very sympathetic/supportive of anti-CP initiatives in general, but this still seems problematic. PS - And before someone replies with "they aren't contacting the police," don't forget the "yet." Once this is in place they'll immediately get pressured by politicians and well-intentioned people alike to flag any positive detections to authorities who will overreact (as law enforcement has a history of doing, particularly around this topic).
- shuckles 5y ago
  In the United States, the process of reporting potential images of CSAM is defined by the government, and a technology company’s active responsibility ends with a report to the National Center for Missing and Exploited Children’s CyberTipline: https://www.missingkids.org/gethelpnow/cybertipline#whathappenstoinformationinacybertip https://www.missingkids.org/gethelpnow/cybertipline#whathapp...
  - __blockcipher__ 5y ago
    From the article: > The automated system would proactively alert a team of human reviewers if it believes illegal imagery is detected, who would then contact law enforcement if the material can be verified. The scheme will initially roll out only in the US. This is dystopian and needs to be opposed, not applauded.
    - shuckles 5y ago
      This is how every single cloud service has worked for a decade. You’ll need to elaborate on how, exactly, you believe it’s dystopian. The article is clearly wrong, because Apple cannot know which jurisdiction to forward a report to. That is NCMEC’s responsibility.
bitwize 5y ago
This just means child molesters will seek out PinePhones, Purism Librems, etc. And those phones will acquire reputations as "pedophones", and anyone buying or using one will be immediately considered sus. People with nothing to hide won't mind an OS with surveillance features.
- exo762 5y ago
  Until few high profile cases of people with nothing to hide getting swatted because of false positive / image crafted by a GAN to create false positive.
  - bitwize 5y ago
    That is most likely to happen when the FBI "discovers" child abuse material on the phone of an uncooperative potential informant. In which case they are going to supermax prison where they will be ventilated by the inmates and no one will miss them.
  - eunos 5y ago
    New malware wont be inserting CP, but running GAN to create CP inside the target.
smnrchrds 5y ago
When it comes to Apple and privacy, it's important to differentiate two different types of privacy: (1) privacy against snooping by the state, and (2) privacy against snooping by all others. As we had seen with Apple and China, it's only the second type of privacy that Apple cares about. Sharing your information with the state is not out of character for Apple. In fact, had they done the opposite, it would have been out of character.
- macintux 5y ago
  Apple cares about #1 when they feel the requests are not legal. https://en.wikipedia.org/wiki/FBI–Apple_encryption_dispute https://en.wikipedia.org/wiki/FBI–Apple_encryption_dispute Apple has to comply with the law, however.
  - smnrchrds 5y ago
    What law prompted this new feature we are discussing here?
    - macintux 5y ago
      I don’t know, and I’m certainly troubled by the story. There is a federal law requiring electronic service providers to notify if they spot such material. Apple may be subject to that. See the “Federal CSAM Law” section here: https://cyberlaw.stanford.edu/blog/2020/01/earn-it-act-how-ban-end-end-encryption-without-actually-banning-it https://cyberlaw.stanford.edu/blog/2020/01/earn-it-act-how-b...
      - diebeforei485 5y ago
        Wow. Okay, so my theory about them trying to stave off future legislation is likely correct.
    - diebeforei485 5y ago
      I assume 18 USC 2258A, but they seem to be going above and beyond. Under current law, they are only required to report, they are not required to prevent[1]. Presumably, they are doing this voluntarily now to stave off future legislation around this that might require more invasive scanning. 1. 18 USC 2258C(c)
marricks 5y ago
Isn't Apple going to start auto scanning the text in your images too for searching/easy copying? How many years until the hash of text in your images is compared to a database of... god knows what. Sensitive information? Dangerous text? I honestly can't think where it's most likely headed but I don't like it.
mistercool 5y ago
> Apple intends to install software on American iPhones to scan for child abuse imagery > Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system Why is there any need for Apple to install software on the iPhone if they are isolating the algorithm to run only on cloud storage, not local images? Not a programmer, so maybe a simple explanation. Also, if this is isolated to iCloud, won't criminals just start using a different cloud backup provider?
- ec109685 5y ago
  Doesn't really answer your question, but the article says this: > According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities.
- rubyist5eva 5y ago
  They are scanning client and server-side photos.
agilob 5y ago
Will Edrogan use this to remove pictures where he is presented in bad light?
- anticensor 5y ago
  Sure.
simion314 5y ago
Do we know why did Apple put the effort to create this? Who forced them to do it? Like it is a move that is bad for PR and also does not make them money, I suspect some big government is demanding it but maybe I am to simple in my thinking and this makes sense.
morpheuskafka 5y ago
> Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system. Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse. This doesn't really make any sense. So it's not a Neural Engine thing but merely hash matching, and it also only applies to iCloud Backup? That is already non-e2e encrypted, so Apple can already access those files whenever it wants.
- rudian 5y ago
  The article is a mess, it could be anything. Locally? On iCloud? Hashed? Content detection? Who knows, the article has all of these keywords.
verifex 5y ago
Non pay-walled site. https://www.msn.com/en-gb/money/technology/apple-plans-to-scan-us-iphones-for-child-abuse-imagery-ft/ar-AAMYKxe https://www.msn.com/en-gb/money/technology/apple-plans-to-sc...
coldacid 5y ago
Today kiddy porn, tomorrow wrongthink.
- 1-6 5y ago
  Like having pictures of Tim Cook eating ice cream?
- BitwiseFool 5y ago
  Isn't PayPal already investigating users' purchases to help the ADL put pressure on transactions which they claim to be funding hate speech?
  - coldacid 5y ago
    Yes they are!
- yepthatsreality 5y ago
  Wrongthink different.
smlss_sftwr 5y ago
I have yet to deep-dive into the techical details of Pegasus so this is pure speculation, but if Pegasus demonstrates the feasibility of deploying and executing arbitrary payloads on remote devices without the owner's knowledge or consent wouldn't that raise a blackmail vector with this new program? Again I'm not familiar with the specific exploits used by Pegasus, but is there anything stopping another malicious actor from relying on the same exploits to say install a bootloader that downloads images that would trigger a positive scan?
nullifidian 5y ago
While they are at it they should up the ante and also add COVID misinformation filters, so that a phone refuses to save it and/or doesn't send it anywhere. Would be very topical. There is also racism and other almost universally recognized wrongthink. If you decide to police offline activity as if it's a cloud platform, why not implement all the online platform's automatic content filtering/censorship? /s
- __blockcipher__ 5y ago
  COVID "misinformation", anti-lockdown protests, any movements that are commonly labelled as domestic terrorism (boog bois, antifa, maybe even BLM; in China it will be anything that speaks out against the CCP, etc) - the potential for abuse is massive. Anyone that trusts the feds to do the right thing here is incredibly naive. They get to operate a black-box database; if your phone has any photos that match an entry in the db you will be reported to the authorities. I can't believe on HN of all places there's not 10x more of an outcry.
  - nullifidian 5y ago
    >if your phone has any photos that match an entry in the db I believe it's not "an entry in the db" but a neural classifier. The article calls it "neuralMatch". So it's much more general than a limited number of known images.
micah94 5y ago
So everyone here just has an FT subscription? I don't see anything but a paywall.
seph-reed 5y ago
I'm too old for this, but I really wonder what it's going to be like for high-school kids and nudes 10 years on. I mean, if you get a really good nude from someone you're going to want to keep it. But for how long? Is there now a social rule of how long to keep nudes for? Is 21 too old, and you should delete them? What if you're still with the person? Personally, I'm glad I don't have to figure it out, but teens sending each-other nudes just has all sorts of confusing ethical quandries to it.
flibble12345 5y ago
so you’ll be able to ruin someone’s life by sending them a live photo where one frame has child abuse. i find the law enforcement part of these well intentioned anti child abuse initiatives to be absolutely terrifying because the standard is not that you were proven to have been a party to child abuse but that an offending file was found on your device.
omgwtfbbq 5y ago
For Apple, who has been trying to highlight their pro-privacy stances, to go through with this is pretty astonishing.
yepthatsreality 5y ago
This will enable Apple to send hashes of matching fingerprint and face scans to their online database. This also completely disabled their argument that their product is safe because the data remains encrypted on your device. Instead of passing the data to the spy service they are sending a report on data to the cloud which will allowed to be much more opaque and misleading. I just ordered a Iphone 12 Mini to replace an Iphone SE, but now I’m going to return it. No amount of cool tech in a product is worth enabling this slippery slope against human privacy rights.
Cycl0ps 5y ago
> The proposals are Apple’s attempt to find a compromise between its own promise to protect customers’ privacy and ongoing demands from governments, law enforcement agencies and child safety campaigners for more assistance in criminal investigations, including terrorism and child pornography. The 1st and 4th amendments tell the fed to go pound sand, Apple should follow suit.
throwaway97904 5y ago
This article raises serious privacy concerns. Just building the infrastructure for on-device scanning and reporting is extremely troubling. A slippery slope, or a break in the dam, as others have said. My view is that we've been on that slope for a long time, and this changes little. We just have to trust Apple, as has always been the case. When I look at the technical details, it seems to me to be a reasonable compromise. It allows Apple and government to do something about the worst offenders, whereas it has no impact on anyone else. Two technical reasons for this: - The "neuralMatch" algorithm suggests some sort of CV, whereas the article talks about matching against a hash of know images. My guess is that the actual technology is something like Microsoft's PhotoDNA (https://news.microsoft.com/on-the-issues/2018/09/12/how-photodna-for-video-is-being-used-to-fight-online-child-exploitation/ https://news.microsoft.com/on-the-issues/2018/09/12/how-phot...). Hash collisions aside, this should only produce matches against images that are already in a government database. It will match manipulated images (e.g., rotated or cropped), but it won't match new images. - As described here, the scanning only applies to images also uploaded to iCloud ("[the] algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system"). While we don't know whether this description is accurate, it suggests that if you don't back up images to iCloud your device won't do any scanning locally. Apple already has the keys to your iCloud backups, so if you value privacy you're probably not backing up to iCloud anyhow. - It sounds like it doesn't flag a single image, but requires multiple hash hits. So, if you want to feel better about this, understand that it is a system that will flag people who are downloading storing known child exploitation images on their devices and naively backing those up to iCloud. This is the sort of privacy compromise that works in practice. Serious offenders are either caught or diverted to other channels. Minor offenders are probably not caught. The risk to non-offenders is zero or close to it. Apple has the control to do all sorts of invasive things to our privacy. They could be scanning and reporting all kinds of things already, and we might not even know. Or they could start doing so tomorrow. From this point of view we're already trusting them to do right by us as their customers, and this feature doesn't change that.
- __blockcipher__ 5y ago
  > As described here, the scanning only applies to images also uploaded to iCloud ("[the] algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system"). While we don't know whether this description is accurate, it suggests that if you don't back up images to iCloud your device won't do any scanning locally. Apple already has the keys to your iCloud backups, so if you value privacy you're probably not backing up to iCloud anyhow. If we had a way to guarantee that it would stay that way, it'd be less concerning. As it stands, clearly there are plans to scan on the device itself, otherwise there would be no need to roll out software on iPhones at all. > Hash collisions aside, this should only produce matches against images that are already in a government database. It will match manipulated images (e.g., rotated or cropped), but it won't match new images. Agreed, but the issue is the government can stick whatever photos in the DB they want. And we have no way to verify what's in there, since they're not gonna release thousands of CP images for us to audit them. > Apple has the control to do all sorts of invasive things to our privacy. They could be scanning and reporting all kinds of things already, and we might not even know. Or they could start doing so tomorrow. From this point of view we're already trusting them to do right by us as their customers, and this feature doesn't change that. It changes plenty. Currently Apple more or less promises not to spy on you, and so they have to actually be lying to do so. Is lying an option for them? Sure, but that's different than them just announcing in advance "hey we're gonna start scanning every photo you have and crossreferencing against a somewhat-transformation-invariant secret government database". I'd prefer that they have to lie to me to spy on me, since that at least has the potential for backlash if discovered. If they get this feature rolled out we're all fucked.
devwastaken 5y ago
Reminder that even accusations of abuse imagery will get you in cuffs and your children taken away (see article at the end) Linux truly is the last bastion of sanity. I am so glad that linux desktops took off and work as well as they do. Last thing I should be worrying about is if an employee at Microsoft or Apple decides their "automated system with no errors" decides that a hash collision or a picture of a baby is close enough to actual abuse imagery. There are plenty of people that believe even a picture of a naked baby in a bathtub is abuse imagery. Anyone remember the case of Walmart? They took their kids away "pending investigation". https://jonathanturley.org/2009/09/18/arizona-couple-sues-wal-mart-after-store-calls-the-police-on-them-for-developing-pictures-of-this-children-in-a-bathtub-and-the-children-are-taken-by-the-state/ https://jonathanturley.org/2009/09/18/arizona-couple-sues-wa...
- deleted 5y ago
  [deleted]
greatgib 5y ago
To laugh, I just imagine the society of 30 years ago and what would have happened if someone would have tried to suggest to do something like that: ie, have private company give themselves the right to go have a look through your belonging and in your home because you might be a criminal. It is now well known that most citizens of most countries are criminals, so no one deserve privacy anymore... But sadly, it looks like that intelligence and common sense has generally decreased in the population. Btw, the appleTV and iphone, that are always listening to your conversations in your home to detect some bad keywords and then denounce you to the police is not yet ready? Nazi collaborators, from their tombs, are still waiting for this nice feature! Anyway, iphone owners, let me tell you that you did a good job by stupidly giving your money and power to Apple!
analyte123 5y ago
Things that would reduce child sexual abuse more than scanning everyone's phones on a daily basis: - Encourage marriage and remove benefits penalties for two-parent households, as children are substantially more likely to be abused when their biological father is not in the household - Fund undercover police work and informants, which is how most large busts already happen now. Ideally highly independent from existing courts and law enforcement to deal with Epstein- or Dutroux-type situations. The worst offenders are not swapping child porn on Facebook or Dropbox (or now iMessage and Apple Photos) - Enforce obscenity laws against incest and other extreme forms of pornography - Institute the death penalty for child sexual abuse and make sure it is performed swiftly and publicly Since these aren't under consideration, we can tell this isn't about reducing child abuse. It's about setting up a permanent and ever-expanding digital system for political control.
- anigbrowl 5y ago
  Enforce obscenity laws against incest and other extreme forms of pornography How would you go about enforcing this, and how does that differ from existing proposals about child porn? Institute the death penalty for child sexual abuse and make sure it is performed swiftly and publicly All the worries about people being framed or wrongly convicted apply equally well to that proposal.
josh_today 5y ago
I’ve been meaning to start a home cloud for pictures on top of Raspberry Pi. Looks like that will get done sooner than I planned
therealjumbo 5y ago
Along with this: https://www.reuters.com/technology/exclusive-facebook-tech-giants-target-manifestos-militias-database-2021-07-26/ https://www.reuters.com/technology/exclusive-facebook-tech-g... Just like the Patriot Act and the no fly list, it's only a very short matter of time until this is expanded to Android and more importantly, non-CP content. The only difference is its the other party doing it this time, in both cases its "to catch bad guys" and "for your own good".
kingsloi 5y ago
I had a very sick 8 month old baby who I obviously had a million photos of. Unfortunately, my lil girl had 9 surgeries and her little body had gone through more than people can think of. She passed in May, and it's really difficult to look back at old pictures because of how much stuff was going on, various levels of critical ICU care, pre/post surgery, etc. But it wasn't until a few months after she was born I had gone to do something in Google and realised my little girl's very severe diaper rash, open sternum, chest tubes, etc, etc, etc photos were backed up and tagged in Photos. This is sort of the premise to why I've de-Googled/de-Cloud myself because I didn't want my little girls pain and suffering to train some computer's image detection AI. How would this work for a parent of a sick child who sends their doctor, S/O, possibly graphic in nature and personal photos of their own child? I was in close contact with my daughter's doctor, and would send him all sorts, I can only imagine the same is true for most parents of chronic children.
- judge2020 5y ago
  This isn't scanning for naked children, it's known CSAM photos that are already circulating online.
  - kingsloi 5y ago
    Ah, I should have paid more attention to the article. I appreciate the clarification.
  - rantwasp 5y ago
    i feel better already /s think about the children!! also, think about poor law enforcement and their mass surveillance attempts. throw them a bone!
wayneftw 5y ago
> the screening is done on the phone. Where is my right to refuse to dedicate my devices cpu, battery time and network bandwidth for this activity? Did I agree to this already? (Sorry for asking a question! I'll try to simply trust Apple at all times from now on...)
InTheArena 5y ago
On one hand, I applaud this - CASM is a massive issue, and the internet community has never taken it as seriously as it should have. As many others have noted - the potential for misuse is rife. Also, realize that Apple has a single set of infrastructure for their phones and macs, especially now that M1 exists. I've read accounts in the person where disaffected people "planted" porn on other peoples system, then called the police. Imagine a web server with a page and a hash map matching image, with a frame-size of 1px. It's now in your cache. Also, it's almost inevitable that this is a slippery slope. Today is may be pre-hashed images. Tommorow it may be AI detection. The day after it may be vaccine falsehoods. The day after it may be other bad government speach. This is needed. This is a major major problem. But I hope someone is thinking of how to mitigate the potential for abuse.
bxjaii 5y ago
Related to this, I love Canva.com but then I noticed they were scanning my designs. Some warning popped up. This idea of perpetually renting services than monitoring the use is something that should be criticized.
rajacombinator 5y ago
Privacy, except for thoughtcrimals.
antisol 5y ago
I'd just like to say thanks for wasting my time by linking to a paywall.
olliej 5y ago
Standard "think of the children" logic. Plenty of countries will put up hashes of LGBT content, "incorrect" religious content, etc
underseacables 5y ago
This is extremely alarming, I don’t care what the reason, I don’t want apple or anyone looking through my personal photos. It’s none of their business, and the sheer chance of false positives and ruining peoples lives is far too great to let this go forward.
- csw-001 5y ago
  Not disagreeing with you - but this reminds me of my college job spending all day looking through personal photos. I put myself through school working at a one-hour photo shop in a drug store. It was my job to feed the machine, keep the chemicals balanced, paper loaded and perform white-balancing and cropping to make sure the end result matched customers' expectations. It was a relatively small town, and I lived in the same neighborhood that I worked in - so I basically saw 80% of my neighbor's photos during the development, cropping, and packing process. I saw hundreds of baby's in bath tubs, and bare a* toddlers on the run, in the mud, etc. Also saw a lot more bedroom photos than you (or at least 17yo me) might expect. In the three years of work, I only had one roll of film for which I called the police. It was obvious child abuse. I stopped the machine, called the police and they collected the film directly into evidence and made the arrest when the person came to pick it up. The guy in question thought that the 1hour process was "all automated" and he was pissed that somebody was looking at all his photos. Not sure that has anything to do with the iPhone AI scanning - but it might be an anecdote of interest to those that don't remember a time when somebody looked at all your photos as part of the creation process.
Animats 5y ago
Next, something that scans for pictures of cops, to detect people taking pictures of law enforcement.
JohnFen 5y ago
Wow, this seems like a terrible, terrible idea on so many counts. It would certainly keep me from buying an iPhone -- the last thing I need is to get a visit from the cops because of false positives. It also means that you can't have any trust about your data safety even if you never put it in the cloud.
doe88 5y ago
What if under a secret order Apple is instructed to insert non-CSAM photos, to match whatever a relevant juridiction wants to match? Effectively, you now have a black box operating on your own computer, this is deeply disturbing. This is not going to end well. And by the way this is like all the DNA stuff, you're not only at risk from your own computer but also the all the libraries your picture happened to be in.
smsm42 5y ago
So, how long before the same technology is used to fight "misinformation" and "domestic terrorism" (e.g. memes that disagree with official point of view)? How long until it's used to ban "hateful speech", or any speech unapproved by the gov^H^H^HReal Government - the Almighty Apple?
headShrinker 5y ago
Apple: An Apple NeuralMatch Agent has positively matched images on device "Apple iPhone 12 (Red™)" geotagged at the coup attempt against Prime Minister Trump occurring on January 30th, 2024. Images and documents have been submitted to the Thump Bureau of Investigation for further evaluation. Your Apple account has been permanently suspended. You can request a review of Apple's actions, however we are receiving higher than normal requests at this time. This means we may be unable to review your account. Sorry for any inconvenience.
mancerayder 5y ago
Will Google by some miracle be the company that offers freedom from forced snooping? It seems not, so we have a choice between this and Google's always-track-you setup. I'm sure no one here has any illegal pictures, but the precedent is terrifying. What about copyrighted images or downloaded videos without DRM ?
jcadam 5y ago
So now anyone can frame you by sending unsolicited child porn to your iPhone.
DSingularity 5y ago
To everyone here who wishes we do nothing about this: you should rethink your position. There is a difference between “let’s be careful and design the right solution” and “let’s do nothing to maximize individual liberty”. In 2017 a 369% increase in CSAM in terms of reports happened. The only explanation is that platforms now facilitate the sharing and a growing problem continues to grow. In raw percentages the data clear: too many children are being abused and the numbers are growing. https://storage.googleapis.com/pub-tools-public-publication-data/pdf/b6555a1018a750f39028005bfdb9f35eaee4b947.pdf https://storage.googleapis.com/pub-tools-public-publication-...

https://archive.ph/ys5Q5#selection-1501.0-1501.54 https://archive.ph/ys5Q5#selection-1501.0-1501.54

1-6 5y ago

Apple should perhaps dogfood their system on their own employees first: https://www.losaltosonline.com/news/hills-man-arrested-on-child-porn-charges/article_9bbe51f7-aaa4-5814-98eb-a9a6b0f51a42.html https://www.losaltosonline.com/news/hills-man-arrested-on-ch...

kwhitefoot 5y ago
Not employees but all of their board and all of their investors.
- maerF0x0 5y ago
  Also lets draft on all the children of senate and congress https://quotepark.com/quotes/1911588-immortal-technique-the-first-payback-that-i-would-accomplish-id-dra/ https://quotepark.com/quotes/1911588-immortal-technique-the-...

laputan_machine 5y ago

Ah, the classic 'think of the children' approach to increasing surveillance. If you're opposed to it, what are you, a nonce?

Markoff 5y ago
It worked pretty well for anyone opposing vaccination with technologies first time approved for use year (mRNA) or two years (viral vector) ago. Anyone not buying into these proven 1-2 years old technologies is crazy anti-vaxxer. Same reasoning can be used with anyone opposing scanning your private images, after all you have nothing to hide.
- vimacs2 5y ago
  Except the difference is nudity is natural and sometimes instrumental - for washing, bathroom use, and medical care for example - and denying basic healthcare to your children is not, unless you plan to be instrumental in your child's demise and that comes natural to you.
  - skinkestek 5y ago
    I've taken all vaccines I've been offered including BioNTech/Pfizer but: carefulness regarding new medical procedures is also completely natural. We aren't robots and we need to cut one another some slack. Different persons being mire or less risk averse is even a strength for society.
    - vimacs2 5y ago
      I don't disagree there, I'm just pointing out that there is a difference between the two cases.
      - leereeves 5y ago
        Your argument was wrong, though, because those vaccines aren't being given to children anyway.
        vimacs2 5y ago
        Reading back now, I realize my comment was indeed just dumb and borne from not properly reading the OP's comment and thinking of the anti-vaxxer movement in general, not the actually rational aversion to untested ones during the Covid crisis. I would have editing a disclaimer on it but it's too late.
      - skinkestek 5y ago
        Here is what you wrote: > and denying basic healthcare to your children is not, unless you plan to be instrumental in your child's demise and that comes natural to you. You wrote that in reply to someone clearly not talking about vaccines generally but the new vaccines specifically.
CiaTransMatters 5y ago
Excuse me, sweetie, but "think of the trans" is a very important movement. It prevents suicides, helps bring gender equality, raises awareness, and creates tech jobs for trans activists to assist in stopping online transphobia.
mhh__ 5y ago
Well I was in a Pizza express in woking, move along.

imroot 5y ago

https://towardsdatascience.com/black-box-attacks-on-perceptual-image-hashes-with-gans-cc1be11f277?gi=cfb3a66600e5 https://towardsdatascience.com/black-box-attacks-on-perceptu... I mean, who's to say that my hash is actually what they think it is... ...and if it's not uploaded to iCloud for someone to manually review what the image is, is that going to be enough to get a search warrant for my phone? "AUSA SOMEONE: But, Your Honor, The Defendant's phone has an image that matched a hash of a CSAM picture!" "DEFENDANT: I'm not going to give you my phone's password." "THE COURT: Bonk: Stay in jail for contempt." I understand why they're doing it. I don't agree with it. At all, and I say that as a survivor of childhood sexual abuse.

Crontab 5y ago
I have always been concerned that this system could be weaponized as a way gain access to someone's account. For example: - Add the hash of a non-pornographic image to the database - Using a burner email address, email the non-pornographic image to the target's Gmail address. The target wouldn't think anything of it. - The innocent image would trigger a CP alert, giving law enforcement the pretense it needs to access the account
- smsm42 5y ago
  This is how the police gets warrantless searches "for drugs" - having identified the subject, call for a dog and have it "alert" on the subject (or the vehicle belonging to the subject), after which there's a probable cause for a search and everything that is found can be used against the subject. The fact that there was no drugs is irrelevant. Now let's translate it to digital world and eliminate the non-scalable components - and you get AI "alerting" on the content of your account and it's a probable cause for searching you. But the best part is of course the Apple doesn't even need a probable cause to search you - they aren't the police, so they are free to search anything at their pleasure and then report to the police. And this report, by itself, would be the probable cause, of course.
zug_zug 5y ago
EDIT: It appears they are using perceptual hashes, which are likely much more prone to collisions. The following math does NOT apply to the tech Apple is using. >> I mean, who's to say that my hash is actually what they think it is... Let's do the math. assume they are using a UUID for the hash. "For example, the number of random version-4 UUIDs which need to be generated in order to have a 50% probability of at least one collision is 2.71 quintillion" [1] "The odds of This number is equivalent to generating 1 billion UUIDs per second for about 85 years." And that's for just 1 collision between two random photos. Considering that the pool of bad hashes is not going to be 2.7 quintillion but a fraction of that, this is more than adequate. And of course they could use a longer checksum if that's the concern. (Not to take a stance on the parent issue, just to explain that hash's are probably a higher standard of proof than DNA matches) 1. https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions https://en.wikipedia.org/wiki/Universally_unique_identifier#...

MeinBlutIstBlau 5y ago

Whose to say what is abuse? Are Caesar Milan tactics on a child abuse? This is outright dangerous beyond belief. Nobody is gonna wanna be a parent if you have to worry about your kid being mad at you and then posts something like that. IIRC there was a teenage girl that lied about her father molesting her back in the 2000s. No evidence, just pure hearsay. Dude was locked up for like 10 years. Now imagine that on a greater scale with several videos and photos taken out of context.

HWR_14 5y ago
That would be false testimony, not hearsay. Hearsay ("John Smith told me he heard MeinBlut kidnapped the Lindbergh baby") is not allowed. Witnesses lying about what they saw and or participated in is allowed, although it can be lies. Sorry, and as for "what is abuse", these are looking for exploited children. There are a lot of questions here, the morality of mass surveillance, the risk factors due to hash collisions (intentional or otherwise), etc, the one thing I doubt is an issue is a fuzzy line on "is this abuse"
- cdolan 5y ago
  I don't think society can function without harsh penalties for child abuse, and I think thats a widely held belief everyone agrees with. I do think your comment is conflicting though. The fuzziness of 'is this abuse' is real because collisions can occur, mass surveillance never goes in reverse, etc.
  - codr7 5y ago
    Doesn't seem to be working very well from here; risk of penalty isn't going to stop most of these people, more likely make their game even more exciting. Of course they have to be prevented from hurting people, to the best of our ability to do so, but punishment isn't going anywhere.
  - HWR_14 5y ago
    Huh, I specifically said "concerns about collisions" is a real concern. I said that "mass surveillance" is a real concern. What is not a real concern is fuzziness in the question "is this image of child abuse". And I really don't want to continue a discussion where you try to come up with gray areas. A computer may not be able to tell the difference, but humans can.

xkcd-sucks 5y ago

So is the phash code actually going to run locally (I.e. can dump it and figure out the function) or is it only running on icloud?

Rebelgecko 5y ago
IIRC they've already been running it on icloud photos for a year or two. Hashing local files is the step they're taking now

mikece 5y ago

Scan for child abuse today; scan for wrong-think tomorrow. I'm liking the idea of Graphene OS more and more each day... or just going phone-free entirely.

alexfromapex 5y ago
We will not be able to have nice things until we get better privacy laws or come up with novel ways to prevent unwarranted snooping. It's sad that it's come to this.

tibbydudeza 5y ago

The dumbest idea ever -so glad that I am an Android user.

partiallypro 5y ago

I really don't see how this is going to end well, there could be perfectly innocent photos on someone's phone of their own children doing perfectly normal things that kids do. Like a kid running butt naked around the house, or a photo of something like a rash that is sent to a nurse friend for advice on what it is etc. I'm all for protecting children from being abused, but how are they going to filter what is normal and what is abuse without human intervention? And at that point isn't that a vast invasion of privacy to the perfectly innocent? It's different if it's online because that puts it in the public realm or on devices/servers owned by Apple, etc.

cm2187 5y ago
What I don't understand is how is it even constitutional in the US. Isn't it a warrant-less search? It's not like scanning content on a cloud. The cloud owns the device on which you store your data. But you own your mobile.
- thrill 5y ago
  Not if you give permission, which you do by your purchase agreement.
  - BitwiseFool 5y ago
    You're correct, but I would love to see a TOS that gives Apple permission to exercise cruel and unusual punishment for violating their terms. The eighth amendment only applies to the government, after all! /s?
- shuckles 5y ago
  The constitution constrains what the Federal government can do, not what private companies can do.
  - cm2187 5y ago
    Not that I am qualified to hold a legal debate, but as far as I know, if Apple implements this under the pressure from the state, and that it reports what it findings to the state, it is effectively an agent of the state and it should apply.

forgingahead 5y ago

https://archive.ph/ys5Q5 https://archive.ph/ys5Q5

ohgreatwtf 5y ago

The key item in the article is that if enough content is marked contentious apple will automatically decrypt your phone for you and disable your encryption. So apple's not merely helping law enforcement find contentious content, they are enabling a back door that makes phone encryption essentially swiss cheese. In 5 years: Apple and Google to scan your phones for politically contentious content and automatically remove it for you/report it to the authorities so you can get the proper help you need to understand what you should believe. I've loved my apple phones for many years now, but it seems this is the end of my relationship with apple. I will be using an open source phone next.

julkali 5y ago

https://news.ycombinator.com/item?id=28068741 https://news.ycombinator.com/item?id=28068741 This topic already made it to the current frontpage.

mkoryak 5y ago

Hopefully they are using phash to match the phone's images to known child abuse image hashes, because if they are using ML, I can only imagine what a false positive will do to someone's life.

heavyset_go 5y ago
Phashes can have collisions, and the false positive rate increases exponentially when you use a fuzzy metric like hamming distance to find derivatives of a source image.

KingMachiavelli 5y ago

This could backfire in a huge way. I think a lot of people, even EFF/privacy focused people are fine with a private company doing CSAM detection (or anything really) on uploaded user content since it is actually voluntary. Now that CSAM detection is being deployed to user devices including those previously sold without it, there's a motivation to circumvent/block it. Unless Apple is doing CSAM scanning 100% inside a 100% secure processor, eventually someone will figure out the specifics of the current perceptual hash algorithm. There are already methods to make perceptual hash collisions. All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless. Sure eventually they will improve the model/dataset but it just starts a cat & mouse game between Apple/Governments & honest citizens/users which actually benefits the people committing these crimes.

luckylion 5y ago
> All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless. And even better: someone could create false-positives and just send them to people, or put them on websites. While you're watching funny kittens tumbling around, your iphone is calling the cops on you because it thinks you're watching child porn.
- slumdev 5y ago
  Like swatting, but worse.
- mataug 5y ago
  A high rate of false positives will create distrust in the system, and ultimately make it less effective since no one will take it seriously.
  - dontblink 5y ago
    Could be used to target specific individuals and discredit them (e.g. critics of government).

chaganated 5y ago

as of noon, looks like most commenters disagree with apple's plan. apple will do it anyway. how is plantation life agreeing with everyone here today?

itake 5y ago

WhatsApp automatically saves photos sent to the phone to the Photo Album. Does this mean they will get a knock on their door if someone sent a bad image?

seanicus 5y ago
First thing I thought of too. A 'prank' that has far greater consequences for the target than swatting.

bastawhiz 5y ago

> Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system. Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse. It's weird and invasive that they'd search users devices, but this makes it sound like they're just searching for files with certain hashes, which (to me) is fairly uninteresting. What I don't understand is why this is called "neuralMatch". Is it some sort of perceptual hashing (versus cryptographic hashing)? What exactly makes it "neural"? If neuralMatch is to Google's reverse image search as Apple Maps was to Google Maps when it launched, this is going to have so many false positives. > According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities Is this separate? Or is the article just confusing iCloud storage for on-device storage? Regardless, there's some very lazy journalism going on here.

alfalfasprout 5y ago
The name is certainly concerning. It's one thing to check for specific hashes. It's another entirely to do approximate hashing. Locality sensitive hashing even in its best implementations has tons of false positives. It's also worrying that eg; someone could stealthily airdrop or iMessage a suspect photo to frame someone.

sandworm101 5y ago

As any of the lawyers here can attest, the legal definitions here open to huge debates. Some material is unambiguous, other images reside extensive grey areas. The legality of material can even change based on context. Something that is perfectly legal in one area can be absolutely illegal in another. Example: If I had to list the most iconic photographs in world history, one of them is of a totally nude child (Vietnam). That isn't an illegal image, but try explaining that to a computer. At the moment these nuances are handled by cops, prosecutors and in extreme cases the courts. Is Apple going to put a marker down on a particular universal definition? Will that definition become the de facto world standard?

alkonaut 5y ago
Who would add “gray area” pictures to the database of CP images that are matched?
- sandworm101 5y ago
  Material that is deemed illegal in jurisdiction A but not in B. Material for which someone was once convicted but if found on someone else's device would not be illegal. Material isn't added to these databases after sober legal deliberation. Police agencies can add basically anything they see fit.
  - alkonaut 5y ago
    This is a system for just one jurisdiction though? Is there talk about doing this outside the US? For multiple jurisdictions you’d need different sets of known hashes.

mrkramer 5y ago

Imagine if Microsoft does this with Windows, lots of people would get arrested rightly so. But like some people say this can be dangerous because evidence can be planted on your device. I remember reading articles how through torrent clients files could be planted but I never actually saw it in the wild.

TuringNYC 5y ago
A good reminder to turn off AirDrop "Auto Accept from Everyone"
- rudian 5y ago
  There is no such thing. AirDrop can "appear" to everyone, but not accept. At most you get a popup.
em500 5y ago
No need to imagine, it's already done in OneDrive which comes with Windows 10 by default: https://en.wikipedia.org/wiki/PhotoDNA https://en.wikipedia.org/wiki/PhotoDNA

Rebelgecko 5y ago

In a world where all police were incorruptible and all laws were just, maybe I'd be ok with this. But since we live in this world, I feel like it's only a matter of time until the "think of the children" technology is coopted to scan for other "bad" images. I wonder how long it'll take for China to compel Apple to add winnie_the_pooh.jpg to the set of naughty images for users in China and HK? And like we saw when bing altered the search results for "tank man" even in the countries like the US, at that point we'd be one configuration error away from those hashes leaking into the US dataset.

r00fus 5y ago
Forget just the government. Imagine if corporations could abuse regulatory structure to enforce copyright on private images?
ksec 5y ago
>I wonder how long it'll take for China... All iCloud Data in China are already within their control. The whole DataCenter are done with "partnership" agreement. HK user have their Data in US but I believe there could be a temporary server in HK DC before it is sent to US.

r00fus 5y ago

I bet Apple understands this is a bad idea (dilutes their "privacy" brand image) I wonder what forces have pushed Apple to this point.

bitwize 5y ago
Privacy was only a talking point to get them some advantage over the Eye of Sauron (Google). I have no doubt that they were ready to jettison their privacy protection marketing angle at a moment's notice. You can't have access to people's data in the cloud and on their phones, and not use it in certain ways. To do so would be to leave money, and the good will of the government, on the table.
maerF0x0 5y ago
perhaps they're being required by law and gagged for indicating as such https://theintercept.com/2018/06/25/att-internet-nsa-spy-hubs/#:~:text=Jim%20Greer%2C%20an,with%20the%20law.%E2%80%9D https://theintercept.com/2018/06/25/att-internet-nsa-spy-hub...
ksec 5y ago
>I bet Apple understands this is a bad idea That would be Steve Job's Apple. Tim Cook's Apple will think they are doing it for the greater good. The road to hell is paved with good intentions

throwaway287391 5y ago

> Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities. Haven't Apple previously claimed that their encryption works such that Apple themselves can't decrypt it no matter what? (IIRC this came up during the San Bernadino attack when the FBI requested that Apple decrypt/unlock a suspect's iPhone and Apple said they couldn't.) Is this not a change? Maybe I'm confusing iCloud encryption vs. iPhone encryption and the former doesn't have any of those guarantees?

WrtCdEvrydy 5y ago
iCloud backups are not encrypted, this is a major knowledge point.
perryizgr8 5y ago
> Apple themselves can't decrypt it no matter what? Apple has root on every iPhone. They can do whatever they want by pushing whatever code they like to any iphone.

runawaybottle 5y ago

It’s 2035, a major terrorist attack occurs. Apple scans all phones for location and images near the area over the last 12 months, sends the list over since the government declared Marshall law. You are now suspicious until proven not suspicious. The real terrorists don’t even use iPhones. But here we are. Edit: Can any legal-heads explain how this does not constitute unlawful search? Apple is not the government, but once they hand the information over, isn’t the government indirectly committing unlawful search? Don’t you need a warrant for this? ____________ Second Edit: Wanted to reply to another comment but being rate limited: As of June 2016, the Terrorist Watch List [No-Fly-List] is estimated to contain over 2,484,442 records, consisting of 1,877,133 individual identities. The number of Muslims in America: A 2017 study estimated that 3.35 million Muslims were living in the United States, about 1.1 percent of the total U.S. population. Lol. Jesus Christ, you do the math.

kevin_thibedeau 5y ago
> isn’t the government indirectly committing unlawful search? No. That's the beauty of the third-party doctrine. Your rights disappear in a puff of twisted logic because you agreed to the fine print.

kwhitefoot 5y ago

Better not take any more pictures of the kids splashing in the bath for granny then.

dehrmann 5y ago

> ...ongoing demands from governments, law enforcement agencies and child safety campaigners for more assistance in criminal investigations, including terrorism and child pornography. I get why they're making these demands, but this is in the US. This would clearly be an unreasonable search by Fourth Amendment standards if it were done by the government, so I'm not sure why government agencies think they can make demands that they, themelves can't legally execute of a private company. To be clear, it's legal for Apple to do this, and legal for the government to ask them to. The government is just laundering its door-to-door search.

nullifidian 5y ago
It's also indicative of the unhealthy relationship these giant semi-monopolistic companies have with the government(s), that they implement the government's supposed will out of the fear of anti-trust actions/legislation, without any need for the government to pass legislation. Same goes for the demands to curb "misinformation" of various kinds, which basically constitutes government mandated speech regulation, even though the standing jurisprudence clearly forbids it.

cptskippy 5y ago

Does this paywalled article have any further details? Aside from a cryptic twitter post that mentions CSAM, are there any further details? CSAM also means Cyber Security Asset Management.

kwhitefoot 5y ago

Are they going to compensate the owners for the CPU time, electricity and bandwidth this will cost?

wayneftw 5y ago
I asked a similar question in both threads here and both times I got downvoted by some courageous folks without an answer. I also would like to know if I already agreed to this and where can I read the agreement that I signed?

Someone1234 5y ago

If it matches, am I guilty until found innocent? I have no illicit images, but false positives are always going to be a problem and even at sub-1% rates if you're scanning literally every image stored on an iOS devices that could still be thousands of wrong matches. If I lose the false-positive lottery, am I going to have the police calling and have my mugshot in the evening news for "CP on their device" in particular in my state's months long backlog for digital forensics. Law enforcement aren't exactly known for their understanding of technical evidence like hash-matching. I'm very sympathetic/supportive of anti-CP initiatives in general, but this still seems problematic. PS - And before someone replies with "they aren't contacting the police," don't forget the "yet." Once this is in place they'll immediately get pressured by politicians and well-intentioned people alike to flag any positive detections to authorities who will overreact (as law enforcement has a history of doing, particularly around this topic).

shuckles 5y ago
In the United States, the process of reporting potential images of CSAM is defined by the government, and a technology company’s active responsibility ends with a report to the National Center for Missing and Exploited Children’s CyberTipline: https://www.missingkids.org/gethelpnow/cybertipline#whathappenstoinformationinacybertip https://www.missingkids.org/gethelpnow/cybertipline#whathapp...
- __blockcipher__ 5y ago
  From the article: > The automated system would proactively alert a team of human reviewers if it believes illegal imagery is detected, who would then contact law enforcement if the material can be verified. The scheme will initially roll out only in the US. This is dystopian and needs to be opposed, not applauded.
  - shuckles 5y ago
    This is how every single cloud service has worked for a decade. You’ll need to elaborate on how, exactly, you believe it’s dystopian. The article is clearly wrong, because Apple cannot know which jurisdiction to forward a report to. That is NCMEC’s responsibility.

bitwize 5y ago

This just means child molesters will seek out PinePhones, Purism Librems, etc. And those phones will acquire reputations as "pedophones", and anyone buying or using one will be immediately considered sus. People with nothing to hide won't mind an OS with surveillance features.

exo762 5y ago
Until few high profile cases of people with nothing to hide getting swatted because of false positive / image crafted by a GAN to create false positive.
- bitwize 5y ago
  That is most likely to happen when the FBI "discovers" child abuse material on the phone of an uncooperative potential informant. In which case they are going to supermax prison where they will be ventilated by the inmates and no one will miss them.
- eunos 5y ago
  New malware wont be inserting CP, but running GAN to create CP inside the target.

smnrchrds 5y ago

When it comes to Apple and privacy, it's important to differentiate two different types of privacy: (1) privacy against snooping by the state, and (2) privacy against snooping by all others. As we had seen with Apple and China, it's only the second type of privacy that Apple cares about. Sharing your information with the state is not out of character for Apple. In fact, had they done the opposite, it would have been out of character.

macintux 5y ago
Apple cares about #1 when they feel the requests are not legal. https://en.wikipedia.org/wiki/FBI–Apple_encryption_dispute https://en.wikipedia.org/wiki/FBI–Apple_encryption_dispute Apple has to comply with the law, however.
- smnrchrds 5y ago
  What law prompted this new feature we are discussing here?
  - macintux 5y ago
    I don’t know, and I’m certainly troubled by the story. There is a federal law requiring electronic service providers to notify if they spot such material. Apple may be subject to that. See the “Federal CSAM Law” section here: https://cyberlaw.stanford.edu/blog/2020/01/earn-it-act-how-ban-end-end-encryption-without-actually-banning-it https://cyberlaw.stanford.edu/blog/2020/01/earn-it-act-how-b...
    - diebeforei485 5y ago
      Wow. Okay, so my theory about them trying to stave off future legislation is likely correct.
  - diebeforei485 5y ago
    I assume 18 USC 2258A, but they seem to be going above and beyond. Under current law, they are only required to report, they are not required to prevent[1]. Presumably, they are doing this voluntarily now to stave off future legislation around this that might require more invasive scanning. 1. 18 USC 2258C(c)

marricks 5y ago

Isn't Apple going to start auto scanning the text in your images too for searching/easy copying? How many years until the hash of text in your images is compared to a database of... god knows what. Sensitive information? Dangerous text? I honestly can't think where it's most likely headed but I don't like it.

mistercool 5y ago

> Apple intends to install software on American iPhones to scan for child abuse imagery > Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system Why is there any need for Apple to install software on the iPhone if they are isolating the algorithm to run only on cloud storage, not local images? Not a programmer, so maybe a simple explanation. Also, if this is isolated to iCloud, won't criminals just start using a different cloud backup provider?

ec109685 5y ago
Doesn't really answer your question, but the article says this: > According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities.
rubyist5eva 5y ago
They are scanning client and server-side photos.

agilob 5y ago

Will Edrogan use this to remove pictures where he is presented in bad light?

anticensor 5y ago
Sure.

simion314 5y ago

Do we know why did Apple put the effort to create this? Who forced them to do it? Like it is a move that is bad for PR and also does not make them money, I suspect some big government is demanding it but maybe I am to simple in my thinking and this makes sense.

morpheuskafka 5y ago

rudian 5y ago
The article is a mess, it could be anything. Locally? On iCloud? Hashed? Content detection? Who knows, the article has all of these keywords.

verifex 5y ago

Non pay-walled site. https://www.msn.com/en-gb/money/technology/apple-plans-to-scan-us-iphones-for-child-abuse-imagery-ft/ar-AAMYKxe https://www.msn.com/en-gb/money/technology/apple-plans-to-sc...

coldacid 5y ago

Today kiddy porn, tomorrow wrongthink.

1-6 5y ago
Like having pictures of Tim Cook eating ice cream?
BitwiseFool 5y ago
Isn't PayPal already investigating users' purchases to help the ADL put pressure on transactions which they claim to be funding hate speech?
- coldacid 5y ago
  Yes they are!
yepthatsreality 5y ago
Wrongthink different.

smlss_sftwr 5y ago

I have yet to deep-dive into the techical details of Pegasus so this is pure speculation, but if Pegasus demonstrates the feasibility of deploying and executing arbitrary payloads on remote devices without the owner's knowledge or consent wouldn't that raise a blackmail vector with this new program? Again I'm not familiar with the specific exploits used by Pegasus, but is there anything stopping another malicious actor from relying on the same exploits to say install a bootloader that downloads images that would trigger a positive scan?

nullifidian 5y ago

While they are at it they should up the ante and also add COVID misinformation filters, so that a phone refuses to save it and/or doesn't send it anywhere. Would be very topical. There is also racism and other almost universally recognized wrongthink. If you decide to police offline activity as if it's a cloud platform, why not implement all the online platform's automatic content filtering/censorship? /s

__blockcipher__ 5y ago
COVID "misinformation", anti-lockdown protests, any movements that are commonly labelled as domestic terrorism (boog bois, antifa, maybe even BLM; in China it will be anything that speaks out against the CCP, etc) - the potential for abuse is massive. Anyone that trusts the feds to do the right thing here is incredibly naive. They get to operate a black-box database; if your phone has any photos that match an entry in the db you will be reported to the authorities. I can't believe on HN of all places there's not 10x more of an outcry.
- nullifidian 5y ago
  >if your phone has any photos that match an entry in the db I believe it's not "an entry in the db" but a neural classifier. The article calls it "neuralMatch". So it's much more general than a limited number of known images.

micah94 5y ago

So everyone here just has an FT subscription? I don't see anything but a paywall.

seph-reed 5y ago

I'm too old for this, but I really wonder what it's going to be like for high-school kids and nudes 10 years on. I mean, if you get a really good nude from someone you're going to want to keep it. But for how long? Is there now a social rule of how long to keep nudes for? Is 21 too old, and you should delete them? What if you're still with the person? Personally, I'm glad I don't have to figure it out, but teens sending each-other nudes just has all sorts of confusing ethical quandries to it.

flibble12345 5y ago

so you’ll be able to ruin someone’s life by sending them a live photo where one frame has child abuse. i find the law enforcement part of these well intentioned anti child abuse initiatives to be absolutely terrifying because the standard is not that you were proven to have been a party to child abuse but that an offending file was found on your device.

omgwtfbbq 5y ago

For Apple, who has been trying to highlight their pro-privacy stances, to go through with this is pretty astonishing.

yepthatsreality 5y ago

This will enable Apple to send hashes of matching fingerprint and face scans to their online database. This also completely disabled their argument that their product is safe because the data remains encrypted on your device. Instead of passing the data to the spy service they are sending a report on data to the cloud which will allowed to be much more opaque and misleading. I just ordered a Iphone 12 Mini to replace an Iphone SE, but now I’m going to return it. No amount of cool tech in a product is worth enabling this slippery slope against human privacy rights.

Cycl0ps 5y ago

> The proposals are Apple’s attempt to find a compromise between its own promise to protect customers’ privacy and ongoing demands from governments, law enforcement agencies and child safety campaigners for more assistance in criminal investigations, including terrorism and child pornography. The 1st and 4th amendments tell the fed to go pound sand, Apple should follow suit.

throwaway97904 5y ago

This article raises serious privacy concerns. Just building the infrastructure for on-device scanning and reporting is extremely troubling. A slippery slope, or a break in the dam, as others have said. My view is that we've been on that slope for a long time, and this changes little. We just have to trust Apple, as has always been the case. When I look at the technical details, it seems to me to be a reasonable compromise. It allows Apple and government to do something about the worst offenders, whereas it has no impact on anyone else. Two technical reasons for this: - The "neuralMatch" algorithm suggests some sort of CV, whereas the article talks about matching against a hash of know images. My guess is that the actual technology is something like Microsoft's PhotoDNA (https://news.microsoft.com/on-the-issues/2018/09/12/how-photodna-for-video-is-being-used-to-fight-online-child-exploitation/ https://news.microsoft.com/on-the-issues/2018/09/12/how-phot...). Hash collisions aside, this should only produce matches against images that are already in a government database. It will match manipulated images (e.g., rotated or cropped), but it won't match new images. - As described here, the scanning only applies to images also uploaded to iCloud ("[the] algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system"). While we don't know whether this description is accurate, it suggests that if you don't back up images to iCloud your device won't do any scanning locally. Apple already has the keys to your iCloud backups, so if you value privacy you're probably not backing up to iCloud anyhow. - It sounds like it doesn't flag a single image, but requires multiple hash hits. So, if you want to feel better about this, understand that it is a system that will flag people who are downloading storing known child exploitation images on their devices and naively backing those up to iCloud. This is the sort of privacy compromise that works in practice. Serious offenders are either caught or diverted to other channels. Minor offenders are probably not caught. The risk to non-offenders is zero or close to it. Apple has the control to do all sorts of invasive things to our privacy. They could be scanning and reporting all kinds of things already, and we might not even know. Or they could start doing so tomorrow. From this point of view we're already trusting them to do right by us as their customers, and this feature doesn't change that.

__blockcipher__ 5y ago
> As described here, the scanning only applies to images also uploaded to iCloud ("[the] algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system"). While we don't know whether this description is accurate, it suggests that if you don't back up images to iCloud your device won't do any scanning locally. Apple already has the keys to your iCloud backups, so if you value privacy you're probably not backing up to iCloud anyhow. If we had a way to guarantee that it would stay that way, it'd be less concerning. As it stands, clearly there are plans to scan on the device itself, otherwise there would be no need to roll out software on iPhones at all. > Hash collisions aside, this should only produce matches against images that are already in a government database. It will match manipulated images (e.g., rotated or cropped), but it won't match new images. Agreed, but the issue is the government can stick whatever photos in the DB they want. And we have no way to verify what's in there, since they're not gonna release thousands of CP images for us to audit them. > Apple has the control to do all sorts of invasive things to our privacy. They could be scanning and reporting all kinds of things already, and we might not even know. Or they could start doing so tomorrow. From this point of view we're already trusting them to do right by us as their customers, and this feature doesn't change that. It changes plenty. Currently Apple more or less promises not to spy on you, and so they have to actually be lying to do so. Is lying an option for them? Sure, but that's different than them just announcing in advance "hey we're gonna start scanning every photo you have and crossreferencing against a somewhat-transformation-invariant secret government database". I'd prefer that they have to lie to me to spy on me, since that at least has the potential for backlash if discovered. If they get this feature rolled out we're all fucked.

devwastaken 5y ago

Reminder that even accusations of abuse imagery will get you in cuffs and your children taken away (see article at the end) Linux truly is the last bastion of sanity. I am so glad that linux desktops took off and work as well as they do. Last thing I should be worrying about is if an employee at Microsoft or Apple decides their "automated system with no errors" decides that a hash collision or a picture of a baby is close enough to actual abuse imagery. There are plenty of people that believe even a picture of a naked baby in a bathtub is abuse imagery. Anyone remember the case of Walmart? They took their kids away "pending investigation". https://jonathanturley.org/2009/09/18/arizona-couple-sues-wal-mart-after-store-calls-the-police-on-them-for-developing-pictures-of-this-children-in-a-bathtub-and-the-children-are-taken-by-the-state/ https://jonathanturley.org/2009/09/18/arizona-couple-sues-wa...

deleted 5y ago
[deleted]

greatgib 5y ago

To laugh, I just imagine the society of 30 years ago and what would have happened if someone would have tried to suggest to do something like that: ie, have private company give themselves the right to go have a look through your belonging and in your home because you might be a criminal. It is now well known that most citizens of most countries are criminals, so no one deserve privacy anymore... But sadly, it looks like that intelligence and common sense has generally decreased in the population. Btw, the appleTV and iphone, that are always listening to your conversations in your home to detect some bad keywords and then denounce you to the police is not yet ready? Nazi collaborators, from their tombs, are still waiting for this nice feature! Anyway, iphone owners, let me tell you that you did a good job by stupidly giving your money and power to Apple!

analyte123 5y ago

Things that would reduce child sexual abuse more than scanning everyone's phones on a daily basis: - Encourage marriage and remove benefits penalties for two-parent households, as children are substantially more likely to be abused when their biological father is not in the household - Fund undercover police work and informants, which is how most large busts already happen now. Ideally highly independent from existing courts and law enforcement to deal with Epstein- or Dutroux-type situations. The worst offenders are not swapping child porn on Facebook or Dropbox (or now iMessage and Apple Photos) - Enforce obscenity laws against incest and other extreme forms of pornography - Institute the death penalty for child sexual abuse and make sure it is performed swiftly and publicly Since these aren't under consideration, we can tell this isn't about reducing child abuse. It's about setting up a permanent and ever-expanding digital system for political control.

anigbrowl 5y ago
Enforce obscenity laws against incest and other extreme forms of pornography How would you go about enforcing this, and how does that differ from existing proposals about child porn? Institute the death penalty for child sexual abuse and make sure it is performed swiftly and publicly All the worries about people being framed or wrongly convicted apply equally well to that proposal.

josh_today 5y ago

I’ve been meaning to start a home cloud for pictures on top of Raspberry Pi. Looks like that will get done sooner than I planned

therealjumbo 5y ago

Along with this: https://www.reuters.com/technology/exclusive-facebook-tech-giants-target-manifestos-militias-database-2021-07-26/ https://www.reuters.com/technology/exclusive-facebook-tech-g... Just like the Patriot Act and the no fly list, it's only a very short matter of time until this is expanded to Android and more importantly, non-CP content. The only difference is its the other party doing it this time, in both cases its "to catch bad guys" and "for your own good".

kingsloi 5y ago

I had a very sick 8 month old baby who I obviously had a million photos of. Unfortunately, my lil girl had 9 surgeries and her little body had gone through more than people can think of. She passed in May, and it's really difficult to look back at old pictures because of how much stuff was going on, various levels of critical ICU care, pre/post surgery, etc. But it wasn't until a few months after she was born I had gone to do something in Google and realised my little girl's very severe diaper rash, open sternum, chest tubes, etc, etc, etc photos were backed up and tagged in Photos. This is sort of the premise to why I've de-Googled/de-Cloud myself because I didn't want my little girls pain and suffering to train some computer's image detection AI. How would this work for a parent of a sick child who sends their doctor, S/O, possibly graphic in nature and personal photos of their own child? I was in close contact with my daughter's doctor, and would send him all sorts, I can only imagine the same is true for most parents of chronic children.

judge2020 5y ago
This isn't scanning for naked children, it's known CSAM photos that are already circulating online.
- kingsloi 5y ago
  Ah, I should have paid more attention to the article. I appreciate the clarification.
- rantwasp 5y ago
  i feel better already /s think about the children!! also, think about poor law enforcement and their mass surveillance attempts. throw them a bone!

wayneftw 5y ago

> the screening is done on the phone. Where is my right to refuse to dedicate my devices cpu, battery time and network bandwidth for this activity? Did I agree to this already? (Sorry for asking a question! I'll try to simply trust Apple at all times from now on...)

InTheArena 5y ago

On one hand, I applaud this - CASM is a massive issue, and the internet community has never taken it as seriously as it should have. As many others have noted - the potential for misuse is rife. Also, realize that Apple has a single set of infrastructure for their phones and macs, especially now that M1 exists. I've read accounts in the person where disaffected people "planted" porn on other peoples system, then called the police. Imagine a web server with a page and a hash map matching image, with a frame-size of 1px. It's now in your cache. Also, it's almost inevitable that this is a slippery slope. Today is may be pre-hashed images. Tommorow it may be AI detection. The day after it may be vaccine falsehoods. The day after it may be other bad government speach. This is needed. This is a major major problem. But I hope someone is thinking of how to mitigate the potential for abuse.

bxjaii 5y ago

Related to this, I love Canva.com but then I noticed they were scanning my designs. Some warning popped up. This idea of perpetually renting services than monitoring the use is something that should be criticized.

rajacombinator 5y ago

Privacy, except for thoughtcrimals.

antisol 5y ago

I'd just like to say thanks for wasting my time by linking to a paywall.

olliej 5y ago

Standard "think of the children" logic. Plenty of countries will put up hashes of LGBT content, "incorrect" religious content, etc

underseacables 5y ago

This is extremely alarming, I don’t care what the reason, I don’t want apple or anyone looking through my personal photos. It’s none of their business, and the sheer chance of false positives and ruining peoples lives is far too great to let this go forward.

csw-001 5y ago
Not disagreeing with you - but this reminds me of my college job spending all day looking through personal photos. I put myself through school working at a one-hour photo shop in a drug store. It was my job to feed the machine, keep the chemicals balanced, paper loaded and perform white-balancing and cropping to make sure the end result matched customers' expectations. It was a relatively small town, and I lived in the same neighborhood that I worked in - so I basically saw 80% of my neighbor's photos during the development, cropping, and packing process. I saw hundreds of baby's in bath tubs, and bare a* toddlers on the run, in the mud, etc. Also saw a lot more bedroom photos than you (or at least 17yo me) might expect. In the three years of work, I only had one roll of film for which I called the police. It was obvious child abuse. I stopped the machine, called the police and they collected the film directly into evidence and made the arrest when the person came to pick it up. The guy in question thought that the 1hour process was "all automated" and he was pissed that somebody was looking at all his photos. Not sure that has anything to do with the iPhone AI scanning - but it might be an anecdote of interest to those that don't remember a time when somebody looked at all your photos as part of the creation process.

Animats 5y ago

Next, something that scans for pictures of cops, to detect people taking pictures of law enforcement.

JohnFen 5y ago

Wow, this seems like a terrible, terrible idea on so many counts. It would certainly keep me from buying an iPhone -- the last thing I need is to get a visit from the cops because of false positives. It also means that you can't have any trust about your data safety even if you never put it in the cloud.

doe88 5y ago

What if under a secret order Apple is instructed to insert non-CSAM photos, to match whatever a relevant juridiction wants to match? Effectively, you now have a black box operating on your own computer, this is deeply disturbing. This is not going to end well. And by the way this is like all the DNA stuff, you're not only at risk from your own computer but also the all the libraries your picture happened to be in.

smsm42 5y ago

So, how long before the same technology is used to fight "misinformation" and "domestic terrorism" (e.g. memes that disagree with official point of view)? How long until it's used to ban "hateful speech", or any speech unapproved by the gov^H^H^HReal Government - the Almighty Apple?

headShrinker 5y ago

Apple: An Apple NeuralMatch Agent has positively matched images on device "Apple iPhone 12 (Red™)" geotagged at the coup attempt against Prime Minister Trump occurring on January 30th, 2024. Images and documents have been submitted to the Thump Bureau of Investigation for further evaluation. Your Apple account has been permanently suspended. You can request a review of Apple's actions, however we are receiving higher than normal requests at this time. This means we may be unable to review your account. Sorry for any inconvenience.

mancerayder 5y ago

Will Google by some miracle be the company that offers freedom from forced snooping? It seems not, so we have a choice between this and Google's always-track-you setup. I'm sure no one here has any illegal pictures, but the precedent is terrifying. What about copyrighted images or downloaded videos without DRM ?

jcadam 5y ago

So now anyone can frame you by sending unsolicited child porn to your iPhone.

DSingularity 5y ago

To everyone here who wishes we do nothing about this: you should rethink your position. There is a difference between “let’s be careful and design the right solution” and “let’s do nothing to maximize individual liberty”. In 2017 a 369% increase in CSAM in terms of reports happened. The only explanation is that platforms now facilitate the sharing and a growing problem continues to grow. In raw percentages the data clear: too many children are being abused and the numbers are growing. https://storage.googleapis.com/pub-tools-public-publication-data/pdf/b6555a1018a750f39028005bfdb9f35eaee4b947.pdf https://storage.googleapis.com/pub-tools-public-publication-...