6 ms·

My understanding was: it was only ever scanning things on the client that were uploaded to iCloud Photos, in the same exact way they are scanned server-side tod

My understanding was: it was only ever scanning things on the client that were uploaded to iCloud Photos, in the same exact way they are scanned server-side today but arguably a step more “transparently” for the user (at least if the user is a reverse engineer, heh). What was crazy about this? The media outrage around the PR was incoherent to me at the time, and it seemed no one took any time to understand the present realities or details.

mindslight 4y ago
The crazy part is forcing a user to run software that undermines their own interests, on a device they purportedly own that is likely a huge part of their life. Software running on a computer should represent that computer owner's interest, period. Apple can run software that represents Apple's interests on their own servers.
- brookst 4y ago
  Meh. Running it on device means that it is narrowly scoped and transparent. There is no way to secretly add a new hash or to vary what you're searching for depending on a user's location or identity. Running it on their servers means nobody knows how expansive the scans are, and there is nothing preventing ad-hoc scans for arbitrary content for individual users.
- sitzkrieg 4y ago
  users havent owned devices in quite a while in this space
- ec109685 4y ago
  The software uploaded an encrypted fingerprint as metadata as part of every photo. The actual scanning and apple self-interest all happened server side.
  - mindslight 4y ago
    Truthfully generating that fingerprint is not in the user's interest. User representing software would skip doing that. If the storage service required it, it would generate plausible dummy values.
- theshrike79 4y ago
  What part of comparing checksums of images on your device to KNOWN child pornography checksums is "undermining your interests"? This was not some Google-style hotdog or not algorithm that was using "AI" to guess if a photo had a nude person on it or not.
  - saagarjha 4y ago
    1. What's to guarantee that the checksums are of child pornography? 2. What's going on with the "fuzzy comparison"?
    - theshrike79 4y ago
      The database is generated by the NCMEC[0]. I trust them to have a rigorous process for adding stuff to it. [0] https://en.wikipedia.org/wiki/National_Center_for_Missing_%26_Exploited_Children https://en.wikipedia.org/wiki/National_Center_for_Missing_%2...
      - saagarjha 4y ago
        A lot of the concern is that others may not trust the NCMEC, or that they don't trust that other images won't be added in whatever ends up on your device.
  - mindslight 4y ago
    First, the huge possibility of false positives. They're not "checksums", but rather "perceptual hashes", which are nowhere near as solid as a cryptographic hash. When you multiply a small chance of a false positive across hundreds of millions of deployed devices, you end up with many false positives - false positives that result in people being characterized as "child abusers". Second, the overall aim of the technology also targets legitimate or inadvertent activities, that the law has also unjustly criminalized - say taking family pictures of your kids, 17 year olds sexting, or someone trolling you. Once again unjustly branding people as "child abusers", this time with "evidence" to back up the narrative. But even in the case of a proper match to the NCMEC list for its bona fide purpose, that still is undermining the phone's owner's interests. That's the philosophical contour, even if you personally wish to brush it aside with the desire to catch people looking at evil images. Our society respects similar longstanding privacy boundaries, even if it does end up helping some "bad people". It's really not in the NCMEC's interest to respect any of this, as their foundational dynamic is that there is a horrible thing happening in the world, and they must do everything possible to stop it. That kind of advocacy is certainly needed, but we shouldn't just accept their desires as if they're an unbiased neutral party.
- infotogivenm 4y ago
  It took me a while of ruminating but I think I like this answer the best. Fundamentally here some people are upset that something they own would betray them. Which is weird, because it implies anything you used but didn’t buy can and should betray you, and it’s your own fault. You should have been smarter. But it makes a bit of sense, for some odd reason I can’t quite make tangible. When the samsung TV I bought starts showing me ads, when my pricey cable or Hulu package interrupts for commercials, I feel a tinge of this. So now I think I somewhat get the feeling. Thank you.
  - mindslight 4y ago
    > because it implies anything you used but didn’t buy can and should betray you, and it’s your own fault The contrapositive isn't automatically true. You can be upset when a cloud host or ISP violates your expectations, even though you don't own them. You can expect Apple to basically work in your interests even on their own servers, since you're paying them. The ownership thing is more about the structure of society. If computing and communications technology is going to move us forward as a distributed democratic society, then those capabilities must remain distributed throughout the population rather than being controlled by a handful of centralized gatekeepers. And yes, we've strayed quite far from this ideal. Most people have little control over what their mobile phone does. Still, there is a distinction between a device not having your desired functionality because the manufacturer didn't create it (or even disallowed others from creating it), and the manufacturer deliberately creating functionality that harms users.
TechBro8615 4y ago
Pretty much every "file" you create on your iphone is uploaded to iCloud by default. For example if you take a screenshot, it saves to your photos library which defaults to uploading new photos to iCloud immediately.
- endisneigh 4y ago
  The same files would’ve been scanned anyway. What’s the problem?
unityByFreedom 4y ago
You'd have a hard time verifying that said on-device scanning would only have been run only on iCloud-uploaded content. Once the feature exists your local data might become accessible to a government warrant, which would make the iPhone the opposite of a privacy oriented device. If it's only for iCloud uploaded data they can simply do the scanning there. There's no reason to use customer's CPU/battery against them.
- lathiat 4y ago
  This necessitates a workflow where both the photos and decryption keys are accessible by the same server and that there is a security workflow to request the users decryption keys without the user involved. This is specifically what Apple is trying to avoid - they are intentionally pushing an environment where the user must be involved to get the keys, by way of their account password and/or other enforcement mechanisms designed to ensure only the real user can access such keys. This is discussed in detail in their Apple Platform Security guide: https://help.apple.com/pdf/security/en_US/apple-platform-security-guide.pdf https://help.apple.com/pdf/security/en_US/apple-platform-sec... All of the facial recognition, object identification, etc is all done on-device for the same reason. By contrast Google can and does do this in the cloud - and there are Google servers that intentionally have access to decrypt your photos. iCloud backup was previously a vector that bypassed this, however they also today announced they are fixing that: https://news.ycombinator.com/item?id=33897793 https://news.ycombinator.com/item?id=33897793 "Apple introduces end-to-end encryption for backups"
  - _jal 4y ago
    Sure, from a technical perspective, it is a nice solution to a set of problems. But there are many serious problems with it. I'll isolate one: an on-device content surveillance mechanism is a slippery slope to a bad, bad place. As the saying from the 90s goes, "child porn [CSAM] is the root password to the US Constitution." It is bad enough that people are willing to suspend their better judgement to do something about it. But after you already have the mechanism accepted an in place, it is far easier to add to it. Pick your boogeyman: you're giving all of them a lovely tool to address their desires. Dissent suppression and Winnie-the-Poo detection for Xi, Erdogan gets to sniff out Golem memes, and choose-your-own horror for the coming dictator of the US. It is far harder to tell a sovereign, "we could easily do that, but will not" than "we don't have a mechanism to do that." And pretending that it won't happen doesn't pass the laugh test - we have seen this show many, many times. But if you want to argue, start by explaining how Apple's jumping to implement the 10-minute-max sharing limit shows how they'd stand up to China about this.
n8cpdx 4y ago
Your apartment complex decides they don’t want to be party to anything illegal. Just in case, they set up a police precinct in the lobby. They set up hidden cameras in every room of your apartment, and if their AI model detects anything suspicious, they send the video to the detective. Because you aren’t doing anything illegal, you have nothing to worry about, right? What’s crazy about this? Another way of expressing the concern: does your iPhone work for you, with the help of Apple’s services, or does your iPhone work for Apple? Working for me means not having software designed to report me to the police for how I use my device. The hash database for now includes CSAM hashes; there’s no reason it couldn’t be extended to include similarly heinous material, like Winnie the Pooh memes, Hong Kong freedom posters, rainbow flags, hormone therapy instructions, or anything else offensive to the regime. People that are comfortable with their device working for a company with the assistance of the user choose Android.
- xionon 4y ago
  The apartment analogy doesn't quite work, as the scanning only happened with photos that were going to be uploaded to iCloud. It wasn't like constantly monitoring your private residence, more like setting up a breathalyzer checkpoint on the way to the freeway.
  - nyani 4y ago
    Closer to scanning your vehicle for meth on the way to the freeway. Rather fraught analog either way perhaps.
  - magicjosh 4y ago
    Fixed: Your apartment complex decides they don’t want to be party to anything illegal. Just in case, they set up a police precinct in the lobby. They set up hidden cameras in the lobby, the hallway leading to your apartment, and the balcony of your apartment. "All public spaces of course" the crowd consents. And if their AI model detects anything suspicious, they send the video to the detective. Because you aren’t doing anything illegal, you have nothing to worry about, right?
archagon 4y ago
It's my device, working against me; and it's also not going to catch anyone remotely intelligent, since everyone and their mother will know about this feature.

mindslight 4y ago

The crazy part is forcing a user to run software that undermines their own interests, on a device they purportedly own that is likely a huge part of their life. Software running on a computer should represent that computer owner's interest, period. Apple can run software that represents Apple's interests on their own servers.

brookst 4y ago
Meh. Running it on device means that it is narrowly scoped and transparent. There is no way to secretly add a new hash or to vary what you're searching for depending on a user's location or identity. Running it on their servers means nobody knows how expansive the scans are, and there is nothing preventing ad-hoc scans for arbitrary content for individual users.
sitzkrieg 4y ago
users havent owned devices in quite a while in this space
ec109685 4y ago
The software uploaded an encrypted fingerprint as metadata as part of every photo. The actual scanning and apple self-interest all happened server side.
- mindslight 4y ago
  Truthfully generating that fingerprint is not in the user's interest. User representing software would skip doing that. If the storage service required it, it would generate plausible dummy values.
theshrike79 4y ago
What part of comparing checksums of images on your device to KNOWN child pornography checksums is "undermining your interests"? This was not some Google-style hotdog or not algorithm that was using "AI" to guess if a photo had a nude person on it or not.
- saagarjha 4y ago
  1. What's to guarantee that the checksums are of child pornography? 2. What's going on with the "fuzzy comparison"?
  - theshrike79 4y ago
    The database is generated by the NCMEC[0]. I trust them to have a rigorous process for adding stuff to it. [0] https://en.wikipedia.org/wiki/National_Center_for_Missing_%26_Exploited_Children https://en.wikipedia.org/wiki/National_Center_for_Missing_%2...
    - saagarjha 4y ago
      A lot of the concern is that others may not trust the NCMEC, or that they don't trust that other images won't be added in whatever ends up on your device.
- mindslight 4y ago
  First, the huge possibility of false positives. They're not "checksums", but rather "perceptual hashes", which are nowhere near as solid as a cryptographic hash. When you multiply a small chance of a false positive across hundreds of millions of deployed devices, you end up with many false positives - false positives that result in people being characterized as "child abusers". Second, the overall aim of the technology also targets legitimate or inadvertent activities, that the law has also unjustly criminalized - say taking family pictures of your kids, 17 year olds sexting, or someone trolling you. Once again unjustly branding people as "child abusers", this time with "evidence" to back up the narrative. But even in the case of a proper match to the NCMEC list for its bona fide purpose, that still is undermining the phone's owner's interests. That's the philosophical contour, even if you personally wish to brush it aside with the desire to catch people looking at evil images. Our society respects similar longstanding privacy boundaries, even if it does end up helping some "bad people". It's really not in the NCMEC's interest to respect any of this, as their foundational dynamic is that there is a horrible thing happening in the world, and they must do everything possible to stop it. That kind of advocacy is certainly needed, but we shouldn't just accept their desires as if they're an unbiased neutral party.
infotogivenm 4y ago
It took me a while of ruminating but I think I like this answer the best. Fundamentally here some people are upset that something they own would betray them. Which is weird, because it implies anything you used but didn’t buy can and should betray you, and it’s your own fault. You should have been smarter. But it makes a bit of sense, for some odd reason I can’t quite make tangible. When the samsung TV I bought starts showing me ads, when my pricey cable or Hulu package interrupts for commercials, I feel a tinge of this. So now I think I somewhat get the feeling. Thank you.
- mindslight 4y ago
  > because it implies anything you used but didn’t buy can and should betray you, and it’s your own fault The contrapositive isn't automatically true. You can be upset when a cloud host or ISP violates your expectations, even though you don't own them. You can expect Apple to basically work in your interests even on their own servers, since you're paying them. The ownership thing is more about the structure of society. If computing and communications technology is going to move us forward as a distributed democratic society, then those capabilities must remain distributed throughout the population rather than being controlled by a handful of centralized gatekeepers. And yes, we've strayed quite far from this ideal. Most people have little control over what their mobile phone does. Still, there is a distinction between a device not having your desired functionality because the manufacturer didn't create it (or even disallowed others from creating it), and the manufacturer deliberately creating functionality that harms users.

TechBro8615 4y ago

Pretty much every "file" you create on your iphone is uploaded to iCloud by default. For example if you take a screenshot, it saves to your photos library which defaults to uploading new photos to iCloud immediately.

endisneigh 4y ago
The same files would’ve been scanned anyway. What’s the problem?

unityByFreedom 4y ago

You'd have a hard time verifying that said on-device scanning would only have been run only on iCloud-uploaded content. Once the feature exists your local data might become accessible to a government warrant, which would make the iPhone the opposite of a privacy oriented device. If it's only for iCloud uploaded data they can simply do the scanning there. There's no reason to use customer's CPU/battery against them.

lathiat 4y ago
This necessitates a workflow where both the photos and decryption keys are accessible by the same server and that there is a security workflow to request the users decryption keys without the user involved. This is specifically what Apple is trying to avoid - they are intentionally pushing an environment where the user must be involved to get the keys, by way of their account password and/or other enforcement mechanisms designed to ensure only the real user can access such keys. This is discussed in detail in their Apple Platform Security guide: https://help.apple.com/pdf/security/en_US/apple-platform-security-guide.pdf https://help.apple.com/pdf/security/en_US/apple-platform-sec... All of the facial recognition, object identification, etc is all done on-device for the same reason. By contrast Google can and does do this in the cloud - and there are Google servers that intentionally have access to decrypt your photos. iCloud backup was previously a vector that bypassed this, however they also today announced they are fixing that: https://news.ycombinator.com/item?id=33897793 https://news.ycombinator.com/item?id=33897793 "Apple introduces end-to-end encryption for backups"
- _jal 4y ago
  Sure, from a technical perspective, it is a nice solution to a set of problems. But there are many serious problems with it. I'll isolate one: an on-device content surveillance mechanism is a slippery slope to a bad, bad place. As the saying from the 90s goes, "child porn [CSAM] is the root password to the US Constitution." It is bad enough that people are willing to suspend their better judgement to do something about it. But after you already have the mechanism accepted an in place, it is far easier to add to it. Pick your boogeyman: you're giving all of them a lovely tool to address their desires. Dissent suppression and Winnie-the-Poo detection for Xi, Erdogan gets to sniff out Golem memes, and choose-your-own horror for the coming dictator of the US. It is far harder to tell a sovereign, "we could easily do that, but will not" than "we don't have a mechanism to do that." And pretending that it won't happen doesn't pass the laugh test - we have seen this show many, many times. But if you want to argue, start by explaining how Apple's jumping to implement the 10-minute-max sharing limit shows how they'd stand up to China about this.

n8cpdx 4y ago

Your apartment complex decides they don’t want to be party to anything illegal. Just in case, they set up a police precinct in the lobby. They set up hidden cameras in every room of your apartment, and if their AI model detects anything suspicious, they send the video to the detective. Because you aren’t doing anything illegal, you have nothing to worry about, right? What’s crazy about this? Another way of expressing the concern: does your iPhone work for you, with the help of Apple’s services, or does your iPhone work for Apple? Working for me means not having software designed to report me to the police for how I use my device. The hash database for now includes CSAM hashes; there’s no reason it couldn’t be extended to include similarly heinous material, like Winnie the Pooh memes, Hong Kong freedom posters, rainbow flags, hormone therapy instructions, or anything else offensive to the regime. People that are comfortable with their device working for a company with the assistance of the user choose Android.

xionon 4y ago
The apartment analogy doesn't quite work, as the scanning only happened with photos that were going to be uploaded to iCloud. It wasn't like constantly monitoring your private residence, more like setting up a breathalyzer checkpoint on the way to the freeway.
- nyani 4y ago
  Closer to scanning your vehicle for meth on the way to the freeway. Rather fraught analog either way perhaps.
- magicjosh 4y ago
  Fixed: Your apartment complex decides they don’t want to be party to anything illegal. Just in case, they set up a police precinct in the lobby. They set up hidden cameras in the lobby, the hallway leading to your apartment, and the balcony of your apartment. "All public spaces of course" the crowd consents. And if their AI model detects anything suspicious, they send the video to the detective. Because you aren’t doing anything illegal, you have nothing to worry about, right?

archagon 4y ago

It's my device, working against me; and it's also not going to catch anyone remotely intelligent, since everyone and their mother will know about this feature.