8 ms·

Apple wants to redefine what it means to violate your privacy. We mustn’t let it

162 points by aral 5y ago

foxyv 5y ago
The thing that scares me isn't so much the violation of privacy. It's the idea that some computer algorithm can accuse me of a crime automatically with no evidence and generate an investigation. Judging by how police respond to these leads, you can end up in jail based on this "Evidence." While you wait for a 6 month investigation to be completed you lose your job and get an arrest record. Even if your photo is just a picture of static on a TV which produced a false positive. It reminds me of the dog who always indicates the presence of drugs 100% of the time. Probable cause, made to order.
- throw03172019 5y ago
  There is a human review (ugh.) as well. But still, your example is very concerning.
  - falcolas 5y ago
    From Apple's announcements, there is a human comparison of the visual hashes. They don't appear to be forcing humans to view the images themselves.
    - y7 5y ago
      The technical assessments speak about "visual derivatives". I'm guessing they mean a low resolution thumbnail.
  - foxyv 5y ago
    They say there is a human review, but fail to explain the parameters of the review. How many photos is a reviewer expected to review in an hour? What are their incentives and performance metrics? Do they have a 3 party voting system to confirm? For positive human review is there additional scrutiny? Is data anonymized properly? Does law enforcement receive a copy? Is this going to be limited to child exploitation? Without this info I just have to assume it's some guy in a box expected to review 1000 pictures an hour and gets scrutinized if he doesn't click at least 10% positive. So he's just counting 1 positive 9 negatives.
    - throw03172019 5y ago
      I would hope they aren’t incentivized to incriminate innocent people.
      - foxyv 5y ago
        I would hope so as well. However, my hopes for these policing algorithms have been dashed too many times. The potential for abuse is just too great without effective laws and appropriate procedures. We've seen the results at YouTube, Pasco County Florida, the UK, and Facebook. There is a reason that a key part of justice is being able to face your accusers. Secret algorithms are being used to abrogate this right. You can't cross examine an algorithm. Expert witnesses to contradict its expertise are too expensive for most defendants. Then the developers of the algorithm can claim with plausible deniability that the algorithm is 1 in a trillion accurate. It's essentially the new bite mark analysis and polygraph pseudoscience used to elicit false admissions of guilt or convince gullible juries. "We have the hash of your photo George, we KNOW you are a pedophile. Just admit it and you can go home. We'll go easy on you. We promise!" https://www.forbes.com/sites/nicksibilla/2021/04/26/lawsuit-florida-county-uses-predictive-policing-to-arrest-residents-for-petty-code-violations/ https://www.forbes.com/sites/nicksibilla/2021/04/26/lawsuit-...
    - gjsman-1000 5y ago
      Remember they said the odds of an account being falsely reported were "1 in a trillion." We have no math to back that, some have called it BS. However... that would mean that one person could easily review only a few photos a week even if it was way off.
- aNoob7000 5y ago
  I want to add that they are scanning your property without your permission. The crazy part is that the government can't do this with a warrant or probable cause, but Apple decided that it can and will. Amazing that Apple destroyed their image about "Privacy" in minutes.
  - Scaevolus 5y ago
    Not that anyone reads them, but every cloud storage EULA has a clause describing how they may remove content at will, especially if it is illegal. That's their justification for why this scanning is happening on photos that are being uploaded to iCloud, and not simply all photos on every device. https://www.apple.com/legal/internet-services/icloud/ https://www.apple.com/legal/internet-services/icloud/ > Apple reserves the right at all times to determine whether Content is appropriate and in compliance with this Agreement, and may screen, move, refuse, modify and/or remove Content at any time, without prior notice and in its sole discretion, if such Content is found to be in violation of this Agreement or is otherwise objectionable.
    - aNoob7000 5y ago
      I have no problem with Apple, Google, Microsoft, etc.. Removing and reporting illegal content from their servers. My issue is with Apple doing it on your device or machine. When did Apple become law enforcement?
  - api 5y ago
    I really wonder about their motives for this. It really has destroyed their image to the point that I'm thinking about abandoning the platform if nothing changes. This will not just be about CP. In many countries this will instantly be used to detect any kind of dissent. In the US it will take longer, but eventually I can imagine "misinformation" as determined by some stupid algorithm flagging you for quiet social credit style lists. What constitutes misinformation will change with the political winds. Hey... maybe we can let people select which reality they wish to be enforced! Maybe Republicans can get flagged if they spread memes about universal health care while Democrats can be flagged for discussing LGBT rights! The only thing I can imagine is that Apple is facing enormous pressure from behind the scenes to become part of the panopticon. Private companies can't really resist governments.
- jrm4 5y ago
  Say it louder. Recently I opposed this and was met with "I bet you don't have children." I do, and I am black and so are they, and I'm quite familiar of the present state of facial recognition tech (and, of course, the justice system generally) when it comes to people who look like us. This is all unacceptable, and the fact that some of it has already come to pass is no reason to not fix it.
  - zepto 5y ago
    What does any of this have to do with facial recognition tech or race?
    - jrm4 5y ago
      I get that questions like these are perhaps meant to be "more precise" about the specific tech in question, but honestly, they do more to display the naivete of the questioner in terms of understanding how this sort of thing plays out in real life, when it's not just tech folks, but other policymakers and stakeholders are involved and have the power to make decisions. Even if the thing Apple is talking about right now can be distinguished from "facial recognition" on a technical level, it would be much MORE of a mistake to NOT lump them in together if we're trying to bring this debate to the general public, which we, of course, we should. (don't get me started on race..)
      - zepto 5y ago
        > it would be much MORE of a mistake to NOT lump them in together if we're trying to bring this debate to the general public, which we, of course, we should. Frankly this suggests that you think it’s a good idea to mislead the general public. I think that is one of the ways we harm our public discourse. I could be missing a connection between the two that is obvious to you but not to me, in which case I apologize. I am not naïve about facial recognition. I am not white, and I have known about ML technology since the 90s. Long before it became a known problem it was obvious to me that ML models would end up simply reflecting the biases of the corpus they were trained on, and this would lead to them embodying discrimination of one kind or another, some of which would not be obvious in advance. The perceived ‘neutrality’ of algorithms would be and in fact is invoked to minimize this problem, when of course the problem is not with the algorithms but with what people feed into them. Please let me know if that doesn’t capture the problem with facial recognition adequately. So, given that I’m not naïve about facial recognition, I ask again - what is the connection you see here between racially biased facial recognition and Apple’s CSAM countermeasures?
- zepto 5y ago
  > The thing that scares me isn't so much the violation of privacy. It's the idea that some computer algorithm can accuse me of a crime automatically with no evidence and generate an investigation. Nothing about apple’s proposal involves algorithms accusing people with a crime without evidence. All actual alerts are done by humans checking the photos.
  - foxyv 5y ago
    Predictive policing models require a human in the loop as well. However people tend to trust these algorithms far beyond their reliability without knowing how they work. As I understand it, human reviewers for this program do not see the photo itself, but instead see the hash and make a determination from that. A positive result may be enough to see all of your devices in evidence bags for the next 6 months and serve as probable cause for warrants. In addition, how hard is it to subpoena Apple for a report on how many times a whistleblower's device has been flagged then use it as probable cause. Or, cherry pick which reports Apple makes to target undesirables.
    - zepto 5y ago
      > As I understand it, human reviewers for this program do not see the photo itself, but instead see the hash and make a determination from that. No, this is not correct. Human reviewers see a visual derivative which is separate from the hash. It’s basically a blurred thumbnail - enough to visually confirm that the image is not a false positive, buy not enough that the reviewers are constantly exposed fo child porn. Also remember that multiple matches are required to even get to the human review. The rest of your comment really doesn’t seem to match the system being described. It’s not predictive policing or anything like it, and it is obviously very much against Apple’s interest for it to generate false positives.
      - smsm42 5y ago
        I'm not buying it. If you can recognize what it is, enough to be able to be sure, you're exposed to it. If not, you're just guessing - and your guess is heavily biased towards confirming what the machine said, because Apple spend $MASSIVE_AMOUNT on this technology, and who are you to question it based on a blurry picture? Also, you'd be saving children, and if you're wrong (which you're likely not - remember, industry-leading AI!) - well, the police surely will find that out very quickly and everything will be fine. It's not like it's the first time such systems are built. We have a cases of Big Social banning people for random stuff and then saying "it was a technical error" when the noise in the media is strong enough. We have chess channels banned for racist hatespeech. Only this time the question is not whether you will be denied the opportunity to post cat pictures on facewitter for a week. It's pretty much the most shameful accusation one can be subjected to in our society. Once the press gets hold of it - and it'd get hold of it the minute the police does - there's no coming back from it for the person affected (well, maybe if they are Hunter Biden, but not otherwise). And all that will be hinged on an anonymous drone looking at a blurry picture?
- heavyset_go 5y ago
  This is why algorithmic and predictive policing is so terrifying. Everyone in the justice system now has a great way to shirk liability for their actions and decisions: they can just defer to the infallible black box system that tells them who is a criminal or not. Want to side step accusations of bias or targeting? They were just going after whoever the black box said was guilty, and computers purportedly are not biased. The corollary to this shirking of responsibility is how it lets people, and not just law enforcement, justify abuse of whoever the system says is guilty of a crime, because after all, the company that made the black box says that there is a one in a trillion chance of encountering a false positive. Judges will throw the book at defendants because, statistically, the black box system is almost never wrong, and the system says the defendants are monsters. But the most Kafkaesque part is that people will never get an answer for how these systems determined they were guilty. It's a trade secret, it's part of on-going investigations, it's critical to national security, or in Apple's case, it's literally illegal to look and find out how their system came to conclusion because viewing the data itself is a crime. These systems often inscrutable ML models, as well, and we all know just how buggy and error prone computers and software can be.
0x0 5y ago
Will the next NSOGroup Pegasus malware feature a swatting-as-a-service plugin that makes the victim phones self-report?
squarefoot 5y ago
I know about nothing on this topic, so my only comment is that every time I read something along "please think of the children!" it suddenly raises warning flags.
- mypastself 5y ago
  There are numerous valuable child protection laws and services against which your knee-jerk would be unwarranted. It just happens to be warranted this time around.
  - ohdannyboy 5y ago
    He didn't say that all measures to protect children should be rejected out of hand. He is just saying that if someones argument starts and ends with fear baiting around children then they probably have ulterior motives.
    - squarefoot 5y ago
      Thanks, that's exactly my point.
    - mypastself 5y ago
      They professed unfamiliarity with the details of the situation, so they could not have known that the emotional appeal was the full extent of the argument.
  - onepunchedman 5y ago
    It's the same excuse the EU are currently using to infringe upon its citizens' privacy and require messaging application providers to install backdoors. It's an appeal to emotion, and since we have to assume that these legislators are intelligent, it's a disgusting overreach.
    - mypastself 5y ago
      They are indeed overreaches, but even a valuable child protection law or service might presumably be pushed with an appeal to emotion. It’s better to familiarize yourself with the situation before dismissing it out of hand. While preserving a healthy dose of skepticism, of course.
vxNsr 5y ago
If you wanna screw someone over and you know they have an iPhone with icloud backup set up, you can whatsapp them a pic that matches CP signature.
- jjgreen 5y ago
  ... an innocuous hash-collision one FTW
- SavageBeast 5y ago
  Digital Swatting
- valparaiso 5y ago
  Same is applicable for Android for the past 10 years if you have enabled backup to Google Photo (everyone has). Or to Microsoft OneDrive or even Dropbox. People tend to focus only on Apple and ignore everything else.
- gjsman-1000 5y ago
  You need to meet the threshold, and they must add those photos to their iCloud Photos library. Remember iCloud Backups are E2E encrypted, iCloud Photo Library is not E2E.
- bogwog 5y ago
  This could actually have some very serious consequences. Quote from the apple announcement: > Apple then manually reviews each report to confirm there is a match, disables the user’s account, and sends a report to NCMEC. So a match doesn't just get you reported to the authorities, it disables your account. Even if the feds do investigate you and find you innocent due to being swatted or whatever, you still need to file an appeal with Apple and hope they give you back your digital life. This could also lead to large spam campaigns of CP being sent to random people, either to just fuck with them, or to undermine the system/overwhelm investigators with a bunch of false positives.
  - heavyset_go 5y ago
    > This could also lead to large spam campaigns of CP being sent to random people, either to just fuck with them, or to undermine the system/overwhelm investigators with a bunch of false positives. If you're a nation-state that wants to sow discord and distrust in another nation's society, what better way than to frame random, respected or important people? The attacker wouldn't need to worry about being arrested, due to being in a foreign country and most likely associated with foreign intelligence agencies. Those same agencies would have the evidence they could plant on people's devices or cloud storage, and the resources not to get caught. They wouldn't even have to burn zero days or accounts/numbers to send messages, just look for database dumps of services that the target country uses, and then log in and upload the evidence. It could be cheap, easy and scalable.
rvz 5y ago
Exactly. It is rephrasing the definition and re-defining what 'they really mean' of their intentions. Hence, Apple Inc. has a very strange definition of what they think 'privacy' means. Always with privacy in mind.™ /s
SavageBeast 5y ago
Apple apparently has the ability to look at pix stored in iCloud. I wonder who's pix they will start looking at first? The Fappening Part II By Apple https://en.wikipedia.org/wiki/ICloud_leaks_of_celebrity_photos https://en.wikipedia.org/wiki/ICloud_leaks_of_celebrity_phot... That which CAN happen WILL happen.
- rasz 5y ago
  https://www.washingtonpost.com/technology/2021/06/09/apple-repair-privacy-settlement-lawsuit/ https://www.washingtonpost.com/technology/2021/06/09/apple-r... https://metro.co.uk/2016/10/13/apple-geniuses-caught-stealing-nudes-from-customers-phones-6189642/ https://metro.co.uk/2016/10/13/apple-geniuses-caught-stealin... https://www.sbs.com.au/news/apple-staff-fired-after-allegedly-sharing-customer-photos-and-rating-female-bodies https://www.sbs.com.au/news/apple-staff-fired-after-allegedl...
josh_today 5y ago
What Apple is doing is the equivalent of the police one day deciding to search everyone’s physical home photo albums just in case there’s a picture of an illegal activity.
- gjsman-1000 5y ago
  Not really, actually. They only do the scanning if you use iCloud Photos backup. If you use, say, local storage instead of iCloud, you don't get scanned (which would be a more apt comparison to a physical photo book). Also, they aren't scanning for new illegal activity, but images that you somehow obtained of past illegal activity that the government already knows about.
  - oceanghost 5y ago
    This to me is the most unbelievable aspect of the whole thing. Apple is basically saying: "So hey, if you have anything you don't want us to know about, here's how to get around our scans (wink, wink)." What would be the point of implementing a system and then telling people how to avoid it? I think at some point in the near future they will flip a switch and start scanning private photos.
  - josh_today 5y ago
    I see your distinction and it makes sense- don’t want to be subject to Apple’s CSAM checks, don’t upload to iCloud. I’d say we can do better. We need better tools for individualized data ownership and interoperability.
system16 5y ago
I can't understand how anyone - even Apple's usual cheerleaders like Gruber - can justify defending this with a straight face. It's scanning your content on your device, without your consent. Full stop. Apple's FAQ to try and quell some of the backlash for this just makes it sound even worse in my opinion with gems like this: _Could governments force Apple to add non-CSAM images to the hash list?_ _Apple will refuse any such demands._ Bullshit. If China "requests" this with threat of banning them from selling the iPhone in China? They'll just say "Apple must operate under the laws of the countries it operates in" and its hands are tied. Which is most likely how this whole thing started. https://www.apple.com/child-safety/pdf/Expanded_Protections_for_Children_Frequently_Asked_Questions.pdf https://www.apple.com/child-safety/pdf/Expanded_Protections_... Maybe there will be more of an uproar when this inevitably comes to macOS.
- gjsman-1000 5y ago
  It is with your consent. If you don't use iCloud Photos, your files don't get scanned.
  - bogwog 5y ago
    Oh right, so if I stop using the thing I paid for, I can avoid giving my consent. I could also just not buy an iPhone for the same effect. (at least until Google gets around to copying Apple)
    - zepto 5y ago
      This isn’t true unless you paid for additional storage in iCloud photos which you are under no obligation to do. There are numerous competing storage photo storage services with both free and paid tiers.
- weikju 5y ago
  > Maybe there will be more of an uproar when this inevitably comes to macOS. It's coming to the next macOS release this fall.
valparaiso 5y ago
As a back-end engineer I can't understand this outrage. Apple's approach is less intrusive than Google and Microsoft since they don't touch your photos in iCloud except when you passed threshold and Apple workers will have technical ability to decrypt your detected (not regular) photos and manually compare with images from the database. Also iPhone doesn't trigger photo scanning if you don't upload them to iCloud. From technical and privacy standpoint they have the best approach and it seems people are mad don't even understanding what Apple is doing. Android users never cared but when news come to Apple everyone is losing their shit. I can't believe people are that weird.
jaywalk 5y ago
A lot of people seem to be forgetting/not know about the other aspect of what Apple will be doing, which is scanning iMessage pics sent or received by minors for nudity. If it's detected, their parents will be notified and have the ability to view the pic in question.
- tzs 5y ago
  > If it's detected, their parents will be notified and have the ability to view the pic in question. There are some other steps in there. If such a photo is detected the minor is notified that it may be intended to harm them and the subject of the photo may not have consented to sharing it. They are asked if they are sure they want to view the photo. If they say they are sure and they are between 13 and 17 they are shown a blurred version of the photo. Their parents are not notified. If they say they are sure and they are under 13 they are told it is their choice but their parents want to make sure they are OK and will be notified so they can check. If they then elect to view the photo they are shown a blurred version of the photo and their parents are notified. This is all off by default. Parents have to expressly opt in to it when they set up a child's device with Family Sharing.
deleted 5y ago
[deleted]
mortenjorck 5y ago
I've been persuaded on this over the past few days. Initially, I saw the controversy as overblown: It's the exact same content scan that already occurs when uploading images to iCloud; it will still only occur when uploading, and the only change is where it takes place. I now see that as a reductionist take. Where it takes place does matter. The lines between client and server have been slowly blurred over the past decade to the point where a move like this may seem trivial to many, but ultimately, it is not. It becomes a foothold for so much more, and despite Apple's detailed assurances of all the friction they've installed onto this particular slippery slope, to step onto it at all is a step too far.
magwa101 5y ago
There has to be a legal angle to this and that's why Apple is doing it. Section 230 protects ISPs, but is Apple/iPhone an ISP? It seems to me that some legal position has forced Apple. This story is just starting, of that I'm certain.
bogwog 5y ago
Serious question: why did Apple bother making that announcement at all? I can't imagine they're naive enough to think it would be good press for them? They could have done this quietly without telling anyone, maybe with a vaguely-worded update to the terms of service for the next mandatory iOS update that nobody reads anyways.
- Zealotux 5y ago
  Hiding such a massive thing in a quiet update would have generated a much larger shitstorm, I don't know if the internal memo I've read is legit but by the look of it: Apple made its calculations and decided it was more interesting to release it this way.
- heavyset_go 5y ago
  > Serious question: why did Apple bother making that announcement at all? I can't imagine they're naive enough to think it would be good press for them? Because they are proud of it. I think for some people, this was a feel good project. I know I'd feel better about working for a company that profits from forced labor[1] and lobbies to water down legislation against forced labor[2] if I was working to help children there. Also, who wouldn't want to stop CSAM distribution? They probably thought dissent would be shamed and outnumbered. [1] https://www.theverge.com/2021/5/10/22428899/apple-suppliers-china-uyghur-forced-labor-report https://www.theverge.com/2021/5/10/22428899/apple-suppliers-... [2] https://www.washingtonpost.com/technology/2020/11/20/apple-uighur/ https://www.washingtonpost.com/technology/2020/11/20/apple-u...
ksec 5y ago
>I will not write another line of code for their platforms ever again. Surprised that this is still a thing. Apple has made it very clear in their App Store case, they do not need developers and Apps on their platform. And Apple operating their App Store has been a benefits, or more like a gift to developers for access to their users.

162 points by aral 5y ago

The thing that scares me isn't so much the violation of privacy. It's the idea that some computer algorithm can accuse me of a crime automatically with no evidence and generate an investigation. Judging by how police respond to these leads, you can end up in jail based on this "Evidence." While you wait for a 6 month investigation to be completed you lose your job and get an arrest record. Even if your photo is just a picture of static on a TV which produced a false positive. It reminds me of the dog who always indicates the presence of drugs 100% of the time. Probable cause, made to order.

throw03172019 5y ago
There is a human review (ugh.) as well. But still, your example is very concerning.
- falcolas 5y ago
  From Apple's announcements, there is a human comparison of the visual hashes. They don't appear to be forcing humans to view the images themselves.
  - y7 5y ago
    The technical assessments speak about "visual derivatives". I'm guessing they mean a low resolution thumbnail.
- foxyv 5y ago
  They say there is a human review, but fail to explain the parameters of the review. How many photos is a reviewer expected to review in an hour? What are their incentives and performance metrics? Do they have a 3 party voting system to confirm? For positive human review is there additional scrutiny? Is data anonymized properly? Does law enforcement receive a copy? Is this going to be limited to child exploitation? Without this info I just have to assume it's some guy in a box expected to review 1000 pictures an hour and gets scrutinized if he doesn't click at least 10% positive. So he's just counting 1 positive 9 negatives.
  - throw03172019 5y ago
    I would hope they aren’t incentivized to incriminate innocent people.
    - foxyv 5y ago
      I would hope so as well. However, my hopes for these policing algorithms have been dashed too many times. The potential for abuse is just too great without effective laws and appropriate procedures. We've seen the results at YouTube, Pasco County Florida, the UK, and Facebook. There is a reason that a key part of justice is being able to face your accusers. Secret algorithms are being used to abrogate this right. You can't cross examine an algorithm. Expert witnesses to contradict its expertise are too expensive for most defendants. Then the developers of the algorithm can claim with plausible deniability that the algorithm is 1 in a trillion accurate. It's essentially the new bite mark analysis and polygraph pseudoscience used to elicit false admissions of guilt or convince gullible juries. "We have the hash of your photo George, we KNOW you are a pedophile. Just admit it and you can go home. We'll go easy on you. We promise!" https://www.forbes.com/sites/nicksibilla/2021/04/26/lawsuit-florida-county-uses-predictive-policing-to-arrest-residents-for-petty-code-violations/ https://www.forbes.com/sites/nicksibilla/2021/04/26/lawsuit-...
  - gjsman-1000 5y ago
    Remember they said the odds of an account being falsely reported were "1 in a trillion." We have no math to back that, some have called it BS. However... that would mean that one person could easily review only a few photos a week even if it was way off.
aNoob7000 5y ago
I want to add that they are scanning your property without your permission. The crazy part is that the government can't do this with a warrant or probable cause, but Apple decided that it can and will. Amazing that Apple destroyed their image about "Privacy" in minutes.
- Scaevolus 5y ago
  Not that anyone reads them, but every cloud storage EULA has a clause describing how they may remove content at will, especially if it is illegal. That's their justification for why this scanning is happening on photos that are being uploaded to iCloud, and not simply all photos on every device. https://www.apple.com/legal/internet-services/icloud/ https://www.apple.com/legal/internet-services/icloud/ > Apple reserves the right at all times to determine whether Content is appropriate and in compliance with this Agreement, and may screen, move, refuse, modify and/or remove Content at any time, without prior notice and in its sole discretion, if such Content is found to be in violation of this Agreement or is otherwise objectionable.
  - aNoob7000 5y ago
    I have no problem with Apple, Google, Microsoft, etc.. Removing and reporting illegal content from their servers. My issue is with Apple doing it on your device or machine. When did Apple become law enforcement?
- api 5y ago
  I really wonder about their motives for this. It really has destroyed their image to the point that I'm thinking about abandoning the platform if nothing changes. This will not just be about CP. In many countries this will instantly be used to detect any kind of dissent. In the US it will take longer, but eventually I can imagine "misinformation" as determined by some stupid algorithm flagging you for quiet social credit style lists. What constitutes misinformation will change with the political winds. Hey... maybe we can let people select which reality they wish to be enforced! Maybe Republicans can get flagged if they spread memes about universal health care while Democrats can be flagged for discussing LGBT rights! The only thing I can imagine is that Apple is facing enormous pressure from behind the scenes to become part of the panopticon. Private companies can't really resist governments.
jrm4 5y ago
Say it louder. Recently I opposed this and was met with "I bet you don't have children." I do, and I am black and so are they, and I'm quite familiar of the present state of facial recognition tech (and, of course, the justice system generally) when it comes to people who look like us. This is all unacceptable, and the fact that some of it has already come to pass is no reason to not fix it.
- zepto 5y ago
  What does any of this have to do with facial recognition tech or race?
  - jrm4 5y ago
    I get that questions like these are perhaps meant to be "more precise" about the specific tech in question, but honestly, they do more to display the naivete of the questioner in terms of understanding how this sort of thing plays out in real life, when it's not just tech folks, but other policymakers and stakeholders are involved and have the power to make decisions. Even if the thing Apple is talking about right now can be distinguished from "facial recognition" on a technical level, it would be much MORE of a mistake to NOT lump them in together if we're trying to bring this debate to the general public, which we, of course, we should. (don't get me started on race..)
    - zepto 5y ago
      > it would be much MORE of a mistake to NOT lump them in together if we're trying to bring this debate to the general public, which we, of course, we should. Frankly this suggests that you think it’s a good idea to mislead the general public. I think that is one of the ways we harm our public discourse. I could be missing a connection between the two that is obvious to you but not to me, in which case I apologize. I am not naïve about facial recognition. I am not white, and I have known about ML technology since the 90s. Long before it became a known problem it was obvious to me that ML models would end up simply reflecting the biases of the corpus they were trained on, and this would lead to them embodying discrimination of one kind or another, some of which would not be obvious in advance. The perceived ‘neutrality’ of algorithms would be and in fact is invoked to minimize this problem, when of course the problem is not with the algorithms but with what people feed into them. Please let me know if that doesn’t capture the problem with facial recognition adequately. So, given that I’m not naïve about facial recognition, I ask again - what is the connection you see here between racially biased facial recognition and Apple’s CSAM countermeasures?
zepto 5y ago
> The thing that scares me isn't so much the violation of privacy. It's the idea that some computer algorithm can accuse me of a crime automatically with no evidence and generate an investigation. Nothing about apple’s proposal involves algorithms accusing people with a crime without evidence. All actual alerts are done by humans checking the photos.
- foxyv 5y ago
  Predictive policing models require a human in the loop as well. However people tend to trust these algorithms far beyond their reliability without knowing how they work. As I understand it, human reviewers for this program do not see the photo itself, but instead see the hash and make a determination from that. A positive result may be enough to see all of your devices in evidence bags for the next 6 months and serve as probable cause for warrants. In addition, how hard is it to subpoena Apple for a report on how many times a whistleblower's device has been flagged then use it as probable cause. Or, cherry pick which reports Apple makes to target undesirables.
  - zepto 5y ago
    > As I understand it, human reviewers for this program do not see the photo itself, but instead see the hash and make a determination from that. No, this is not correct. Human reviewers see a visual derivative which is separate from the hash. It’s basically a blurred thumbnail - enough to visually confirm that the image is not a false positive, buy not enough that the reviewers are constantly exposed fo child porn. Also remember that multiple matches are required to even get to the human review. The rest of your comment really doesn’t seem to match the system being described. It’s not predictive policing or anything like it, and it is obviously very much against Apple’s interest for it to generate false positives.
    - smsm42 5y ago
      I'm not buying it. If you can recognize what it is, enough to be able to be sure, you're exposed to it. If not, you're just guessing - and your guess is heavily biased towards confirming what the machine said, because Apple spend $MASSIVE_AMOUNT on this technology, and who are you to question it based on a blurry picture? Also, you'd be saving children, and if you're wrong (which you're likely not - remember, industry-leading AI!) - well, the police surely will find that out very quickly and everything will be fine. It's not like it's the first time such systems are built. We have a cases of Big Social banning people for random stuff and then saying "it was a technical error" when the noise in the media is strong enough. We have chess channels banned for racist hatespeech. Only this time the question is not whether you will be denied the opportunity to post cat pictures on facewitter for a week. It's pretty much the most shameful accusation one can be subjected to in our society. Once the press gets hold of it - and it'd get hold of it the minute the police does - there's no coming back from it for the person affected (well, maybe if they are Hunter Biden, but not otherwise). And all that will be hinged on an anonymous drone looking at a blurry picture?
heavyset_go 5y ago
This is why algorithmic and predictive policing is so terrifying. Everyone in the justice system now has a great way to shirk liability for their actions and decisions: they can just defer to the infallible black box system that tells them who is a criminal or not. Want to side step accusations of bias or targeting? They were just going after whoever the black box said was guilty, and computers purportedly are not biased. The corollary to this shirking of responsibility is how it lets people, and not just law enforcement, justify abuse of whoever the system says is guilty of a crime, because after all, the company that made the black box says that there is a one in a trillion chance of encountering a false positive. Judges will throw the book at defendants because, statistically, the black box system is almost never wrong, and the system says the defendants are monsters. But the most Kafkaesque part is that people will never get an answer for how these systems determined they were guilty. It's a trade secret, it's part of on-going investigations, it's critical to national security, or in Apple's case, it's literally illegal to look and find out how their system came to conclusion because viewing the data itself is a crime. These systems often inscrutable ML models, as well, and we all know just how buggy and error prone computers and software can be.

0x0 5y ago

Will the next NSOGroup Pegasus malware feature a swatting-as-a-service plugin that makes the victim phones self-report?

squarefoot 5y ago

I know about nothing on this topic, so my only comment is that every time I read something along "please think of the children!" it suddenly raises warning flags.

mypastself 5y ago
There are numerous valuable child protection laws and services against which your knee-jerk would be unwarranted. It just happens to be warranted this time around.
- ohdannyboy 5y ago
  He didn't say that all measures to protect children should be rejected out of hand. He is just saying that if someones argument starts and ends with fear baiting around children then they probably have ulterior motives.
  - squarefoot 5y ago
    Thanks, that's exactly my point.
  - mypastself 5y ago
    They professed unfamiliarity with the details of the situation, so they could not have known that the emotional appeal was the full extent of the argument.
- onepunchedman 5y ago
  It's the same excuse the EU are currently using to infringe upon its citizens' privacy and require messaging application providers to install backdoors. It's an appeal to emotion, and since we have to assume that these legislators are intelligent, it's a disgusting overreach.
  - mypastself 5y ago
    They are indeed overreaches, but even a valuable child protection law or service might presumably be pushed with an appeal to emotion. It’s better to familiarize yourself with the situation before dismissing it out of hand. While preserving a healthy dose of skepticism, of course.

vxNsr 5y ago

If you wanna screw someone over and you know they have an iPhone with icloud backup set up, you can whatsapp them a pic that matches CP signature.

jjgreen 5y ago
... an innocuous hash-collision one FTW
SavageBeast 5y ago
Digital Swatting
valparaiso 5y ago
Same is applicable for Android for the past 10 years if you have enabled backup to Google Photo (everyone has). Or to Microsoft OneDrive or even Dropbox. People tend to focus only on Apple and ignore everything else.
gjsman-1000 5y ago
You need to meet the threshold, and they must add those photos to their iCloud Photos library. Remember iCloud Backups are E2E encrypted, iCloud Photo Library is not E2E.
bogwog 5y ago
This could actually have some very serious consequences. Quote from the apple announcement: > Apple then manually reviews each report to confirm there is a match, disables the user’s account, and sends a report to NCMEC. So a match doesn't just get you reported to the authorities, it disables your account. Even if the feds do investigate you and find you innocent due to being swatted or whatever, you still need to file an appeal with Apple and hope they give you back your digital life. This could also lead to large spam campaigns of CP being sent to random people, either to just fuck with them, or to undermine the system/overwhelm investigators with a bunch of false positives.
- heavyset_go 5y ago
  > This could also lead to large spam campaigns of CP being sent to random people, either to just fuck with them, or to undermine the system/overwhelm investigators with a bunch of false positives. If you're a nation-state that wants to sow discord and distrust in another nation's society, what better way than to frame random, respected or important people? The attacker wouldn't need to worry about being arrested, due to being in a foreign country and most likely associated with foreign intelligence agencies. Those same agencies would have the evidence they could plant on people's devices or cloud storage, and the resources not to get caught. They wouldn't even have to burn zero days or accounts/numbers to send messages, just look for database dumps of services that the target country uses, and then log in and upload the evidence. It could be cheap, easy and scalable.

rvz 5y ago

Exactly. It is rephrasing the definition and re-defining what 'they really mean' of their intentions. Hence, Apple Inc. has a very strange definition of what they think 'privacy' means. Always with privacy in mind.™ /s

SavageBeast 5y ago

Apple apparently has the ability to look at pix stored in iCloud. I wonder who's pix they will start looking at first? The Fappening Part II By Apple https://en.wikipedia.org/wiki/ICloud_leaks_of_celebrity_photos https://en.wikipedia.org/wiki/ICloud_leaks_of_celebrity_phot... That which CAN happen WILL happen.

rasz 5y ago
https://www.washingtonpost.com/technology/2021/06/09/apple-repair-privacy-settlement-lawsuit/ https://www.washingtonpost.com/technology/2021/06/09/apple-r... https://metro.co.uk/2016/10/13/apple-geniuses-caught-stealing-nudes-from-customers-phones-6189642/ https://metro.co.uk/2016/10/13/apple-geniuses-caught-stealin... https://www.sbs.com.au/news/apple-staff-fired-after-allegedly-sharing-customer-photos-and-rating-female-bodies https://www.sbs.com.au/news/apple-staff-fired-after-allegedl...

josh_today 5y ago

What Apple is doing is the equivalent of the police one day deciding to search everyone’s physical home photo albums just in case there’s a picture of an illegal activity.

gjsman-1000 5y ago
Not really, actually. They only do the scanning if you use iCloud Photos backup. If you use, say, local storage instead of iCloud, you don't get scanned (which would be a more apt comparison to a physical photo book). Also, they aren't scanning for new illegal activity, but images that you somehow obtained of past illegal activity that the government already knows about.
- oceanghost 5y ago
  This to me is the most unbelievable aspect of the whole thing. Apple is basically saying: "So hey, if you have anything you don't want us to know about, here's how to get around our scans (wink, wink)." What would be the point of implementing a system and then telling people how to avoid it? I think at some point in the near future they will flip a switch and start scanning private photos.
- josh_today 5y ago
  I see your distinction and it makes sense- don’t want to be subject to Apple’s CSAM checks, don’t upload to iCloud. I’d say we can do better. We need better tools for individualized data ownership and interoperability.

system16 5y ago

I can't understand how anyone - even Apple's usual cheerleaders like Gruber - can justify defending this with a straight face. It's scanning your content on your device, without your consent. Full stop. Apple's FAQ to try and quell some of the backlash for this just makes it sound even worse in my opinion with gems like this: _Could governments force Apple to add non-CSAM images to the hash list?_ _Apple will refuse any such demands._ Bullshit. If China "requests" this with threat of banning them from selling the iPhone in China? They'll just say "Apple must operate under the laws of the countries it operates in" and its hands are tied. Which is most likely how this whole thing started. https://www.apple.com/child-safety/pdf/Expanded_Protections_for_Children_Frequently_Asked_Questions.pdf https://www.apple.com/child-safety/pdf/Expanded_Protections_... Maybe there will be more of an uproar when this inevitably comes to macOS.

gjsman-1000 5y ago
It is with your consent. If you don't use iCloud Photos, your files don't get scanned.
- bogwog 5y ago
  Oh right, so if I stop using the thing I paid for, I can avoid giving my consent. I could also just not buy an iPhone for the same effect. (at least until Google gets around to copying Apple)
  - zepto 5y ago
    This isn’t true unless you paid for additional storage in iCloud photos which you are under no obligation to do. There are numerous competing storage photo storage services with both free and paid tiers.
weikju 5y ago
> Maybe there will be more of an uproar when this inevitably comes to macOS. It's coming to the next macOS release this fall.

valparaiso 5y ago

As a back-end engineer I can't understand this outrage. Apple's approach is less intrusive than Google and Microsoft since they don't touch your photos in iCloud except when you passed threshold and Apple workers will have technical ability to decrypt your detected (not regular) photos and manually compare with images from the database. Also iPhone doesn't trigger photo scanning if you don't upload them to iCloud. From technical and privacy standpoint they have the best approach and it seems people are mad don't even understanding what Apple is doing. Android users never cared but when news come to Apple everyone is losing their shit. I can't believe people are that weird.

jaywalk 5y ago

A lot of people seem to be forgetting/not know about the other aspect of what Apple will be doing, which is scanning iMessage pics sent or received by minors for nudity. If it's detected, their parents will be notified and have the ability to view the pic in question.

tzs 5y ago
> If it's detected, their parents will be notified and have the ability to view the pic in question. There are some other steps in there. If such a photo is detected the minor is notified that it may be intended to harm them and the subject of the photo may not have consented to sharing it. They are asked if they are sure they want to view the photo. If they say they are sure and they are between 13 and 17 they are shown a blurred version of the photo. Their parents are not notified. If they say they are sure and they are under 13 they are told it is their choice but their parents want to make sure they are OK and will be notified so they can check. If they then elect to view the photo they are shown a blurred version of the photo and their parents are notified. This is all off by default. Parents have to expressly opt in to it when they set up a child's device with Family Sharing.

deleted 5y ago

[deleted]

mortenjorck 5y ago

I've been persuaded on this over the past few days. Initially, I saw the controversy as overblown: It's the exact same content scan that already occurs when uploading images to iCloud; it will still only occur when uploading, and the only change is where it takes place. I now see that as a reductionist take. Where it takes place does matter. The lines between client and server have been slowly blurred over the past decade to the point where a move like this may seem trivial to many, but ultimately, it is not. It becomes a foothold for so much more, and despite Apple's detailed assurances of all the friction they've installed onto this particular slippery slope, to step onto it at all is a step too far.

magwa101 5y ago

There has to be a legal angle to this and that's why Apple is doing it. Section 230 protects ISPs, but is Apple/iPhone an ISP? It seems to me that some legal position has forced Apple. This story is just starting, of that I'm certain.

bogwog 5y ago

Serious question: why did Apple bother making that announcement at all? I can't imagine they're naive enough to think it would be good press for them? They could have done this quietly without telling anyone, maybe with a vaguely-worded update to the terms of service for the next mandatory iOS update that nobody reads anyways.

Zealotux 5y ago
Hiding such a massive thing in a quiet update would have generated a much larger shitstorm, I don't know if the internal memo I've read is legit but by the look of it: Apple made its calculations and decided it was more interesting to release it this way.
heavyset_go 5y ago
> Serious question: why did Apple bother making that announcement at all? I can't imagine they're naive enough to think it would be good press for them? Because they are proud of it. I think for some people, this was a feel good project. I know I'd feel better about working for a company that profits from forced labor[1] and lobbies to water down legislation against forced labor[2] if I was working to help children there. Also, who wouldn't want to stop CSAM distribution? They probably thought dissent would be shamed and outnumbered. [1] https://www.theverge.com/2021/5/10/22428899/apple-suppliers-china-uyghur-forced-labor-report https://www.theverge.com/2021/5/10/22428899/apple-suppliers-... [2] https://www.washingtonpost.com/technology/2020/11/20/apple-uighur/ https://www.washingtonpost.com/technology/2020/11/20/apple-u...

ksec 5y ago

>I will not write another line of code for their platforms ever again. Surprised that this is still a thing. Apple has made it very clear in their App Store case, they do not need developers and Apps on their platform. And Apple operating their App Store has been a benefits, or more like a gift to developers for access to their users.