10 ms·

I feel like this is a false sense of security. Even before this change, they can easily access and scan photos on your device. If they do any post-processing of

by tmdg 5y ago

I feel like this is a false sense of security. Even before this change, they can easily access and scan photos on your device. If they do any post-processing of the image on device, they already do.

shuckles 5y ago
Apple already scans your photos for faces and syncs found faces through iCloud. I’d imagine updating that machine learning model is at least as straightforward as this one.
- Wowfunhappy 5y ago
  They're searching for different things though. To my knowledge, before now iOS has never scanned for fingerprints of specific photographs. It would be so darn easy to replace the CSAM database with fingerprints of known tiananmen square photos...
  - shuckles 5y ago
    That is a distinction without a difference. I’m sure you could put together quite a good tank man classifier (proof: Google Reverse Image Search works quite well), and it’d catch variations which a perceptual hash wouldn’t. The only difference is intent. The technical risk has not changed at all.
    - dpedu 5y ago
      That is to say face scanning is equally insidious as the new feature?
      - shuckles 5y ago
        The technical risk to user privacy - if your threat model is a coerced Apple building surveillance features for nation state actors - is exactly the same between CSAM detection and Photos intelligence which sync results through iCloud. In fact, the latter is more generalizable, has no threshold protections, and so is likely worse.
        int_19h 5y ago
        It's the legal risk that is the biggest problem here. Now that every politician out there knows that this can be done for child porn, there'll be plenty demanding the same for other stuff. And this puts Apple in a rather difficult position, since, with every such demand, they have to either accede, or explain why it's not "important enough" - which then is easily weaponized to bash them. And not just Apple. Once technical feasibility is proven, I can easily see governments mandating this scheme for all devices sold. At that point, it can get even more ugly, since e.g. custom ROMs and such could be seen as a loophole, and cracked down upon.
        shuckles 5y ago
        This hypothetical lacks an explanation for why every politician has not demanded Apple (or say Google) do this scope creep already for photos stored in the cloud where the technical feasibility and legal precedent has already been established by existing CSAM scanning solutions deployed at scale.
        int_19h 5y ago
        I have to note that one of those solutions deployed at scale is Google's. But the big difference is that when those were originally rolled out, they didn't make quite that big of a splash, especially outside of tech circles. I will also note that, while it may be a hypothetical in this particular instance as yet, EU already went from passing a law that allows companies to do something similar voluntarily (previously, they'd be running afoul of privacy regulations), to a proposed bill making it mandatory - in less than a year's time. I don't see why US would be any different in that regard.
        shuckles 5y ago
        Ok but now you’ve said that the precedent established by Google and others already moved the legislation to require terrible invasions of privacy far along. You started by saying Apple’s technology (and, in particular, its framing of the technology) has brought new legal risk. What I’m instead hearing is the risk would be present in a counter factual world where nothing was announced last week. At this point of the discussion, people usually pivot to scope creep: the on-device scanning could scan all your device data, instead of just the data you put on the cloud. This claim assumes that legislators are too dumb to connect the fact that if their phone can search for dogs with “on-device processing,” then it could also search for contraband. I doubt it. And even if they are, the national security apparatus will surely discover this argument for them, aided by the Andurils and NSOs of the world. As I have repeatedly said: the reaction to this announcement sounds more like a collective reckoning of where we are as humans and not any particular new risk introduced by Apple. In the Apple vs. FBI letter, Tim urged us to have a discussion about encryption, when we want it, why we want it, and to what extent we should protect it. Instead, we elected Trump.
  - samstave 5y ago
    So you have User A - they upload a pic with User A and Peoples B,C,D,E,Z icloud scans for those faces finds those faces and ties them to other ID accounts via face - then via fingerprint recognition to a device, and to a location based on IMEI etc. Apple's platform is literally the foundation for the most dystopian digital tool-set in history... Once the government is able to crack the apple war chest, everything is fucked.
    - samstave 5y ago
      --MBS has entered the chat. Go fuck yourself.
- fraa-orolo 5y ago
  A false positive in matching faces results in a click to fix it or a wrongly categorized photo. A false positive in this new thing may land you in jail or have your life destroyed. Even an allegation of something so heinous is enough to ruin a life. The "one in trillion" chance of false positives is Apple's invention. They haven't scanned trillions of photos and it's a guess. And you need multiple false positives, yet no one says how many, so it could be a low number. Either way, even with how small the chance of it being wrong is, the consequences for the individual are catastrophic. No one sane should accept that kind of risk/reward ratio. "Oh, and one more thing, and we think you'll love it. You can back up your entire camera roll for just $10 a month and a really infinitesimally minuscule chance that you and your family will be completely disgraced in the public eye, and you'll get raped and murdered in prison for nothing."
  - systoll 5y ago
    Ok. So iCloud Photos circa 2020 [and Google Photos and Facebook and Dropbox and OneDrive] aren’t a risk you should be willing to take. This feature doesn’t change anything in that regard; the scanning was already happening.
    - XorNot 5y ago
      I literally do not take that risk in 2021. I do, currently, make the reasoned assurance that the computational overhead of pushing changes down to my phone, and the general international security community, are keeping me approximately abreast of whether my private device is actively spying on me (short answer: it definitely is, longer answer: but to what specific intent?) Apple's new policy is: "of course your phone is scanning and flagging your private files to our server - that's normal behavior! Don't worry about it".
lifty 5y ago
It’s not a false sense of security, it’s a clear delimitation between theirs and mine; Debian package maintainers can also slip a scanner on your machine but that is a big line to cross on purpose and without notifying the user.
- totetsu 5y ago
  But with a debian package you can choose not to accept the upgrade and see any funny business in the release source code..
  - gpm 5y ago
    Somewhere along the line someone is producing and signing the binaries that find their way onto their computer, they could produce those binaries from different source code and I would be none the wiser. Debian tries to be reproducible, so to avoid being caught they might need to control the mirror to so that they could send it to only me. I.e. if I'm lucky it would take a total of 2 people to put malicious binaries on my computer (1 with a signing key, 1 with access to the mirror I download things from).
  - least 5y ago
    That is technically true but in a real very practical sense everyone here using OSS absolutely is trusting a third party because they are not auditing every bit of code they run. For less technical people there is effectively zero difference between open and closed software. It’s really disingenuous to suggest that open source isn’t dependent on trust, you just change who you are trusting. Even if the case is someone else is auditing that code, you’re trusting that person instead of the repository owners. I’ll concede that at least that possibility to audit exists but personally I do have to trust to a certain extent that third parties aren’t trying to fuck me over.
    - totetsu 5y ago
      Thinking about this.. I guess my trust, is that someone smarter than I will notice it, cause a fuss, and the community will raise pitch forks and.. git forks. My trust is in the community, I hope it can stay healthy and diverse for all time.
stjohnswarts 5y ago
The only way what you said is not true for any networked device is to just go down to the river and throw it in and never use a digital device again. It's not a false sense of security, it's a calculated position on security and what you will accept, moving spying from the server to the phone was the last straw for a lot of people.

by tmdg 5y ago

I feel like this is a false sense of security. Even before this change, they can easily access and scan photos on your device. If they do any post-processing of the image on device, they already do.

shuckles 5y ago

Apple already scans your photos for faces and syncs found faces through iCloud. I’d imagine updating that machine learning model is at least as straightforward as this one.

Wowfunhappy 5y ago
They're searching for different things though. To my knowledge, before now iOS has never scanned for fingerprints of specific photographs. It would be so darn easy to replace the CSAM database with fingerprints of known tiananmen square photos...
- shuckles 5y ago
  That is a distinction without a difference. I’m sure you could put together quite a good tank man classifier (proof: Google Reverse Image Search works quite well), and it’d catch variations which a perceptual hash wouldn’t. The only difference is intent. The technical risk has not changed at all.
  - dpedu 5y ago
    That is to say face scanning is equally insidious as the new feature?
    - shuckles 5y ago
      The technical risk to user privacy - if your threat model is a coerced Apple building surveillance features for nation state actors - is exactly the same between CSAM detection and Photos intelligence which sync results through iCloud. In fact, the latter is more generalizable, has no threshold protections, and so is likely worse.
      - int_19h 5y ago
        It's the legal risk that is the biggest problem here. Now that every politician out there knows that this can be done for child porn, there'll be plenty demanding the same for other stuff. And this puts Apple in a rather difficult position, since, with every such demand, they have to either accede, or explain why it's not "important enough" - which then is easily weaponized to bash them. And not just Apple. Once technical feasibility is proven, I can easily see governments mandating this scheme for all devices sold. At that point, it can get even more ugly, since e.g. custom ROMs and such could be seen as a loophole, and cracked down upon.
        shuckles 5y ago
        This hypothetical lacks an explanation for why every politician has not demanded Apple (or say Google) do this scope creep already for photos stored in the cloud where the technical feasibility and legal precedent has already been established by existing CSAM scanning solutions deployed at scale.
        int_19h 5y ago
        I have to note that one of those solutions deployed at scale is Google's. But the big difference is that when those were originally rolled out, they didn't make quite that big of a splash, especially outside of tech circles. I will also note that, while it may be a hypothetical in this particular instance as yet, EU already went from passing a law that allows companies to do something similar voluntarily (previously, they'd be running afoul of privacy regulations), to a proposed bill making it mandatory - in less than a year's time. I don't see why US would be any different in that regard.
        shuckles 5y ago
        Ok but now you’ve said that the precedent established by Google and others already moved the legislation to require terrible invasions of privacy far along. You started by saying Apple’s technology (and, in particular, its framing of the technology) has brought new legal risk. What I’m instead hearing is the risk would be present in a counter factual world where nothing was announced last week. At this point of the discussion, people usually pivot to scope creep: the on-device scanning could scan all your device data, instead of just the data you put on the cloud. This claim assumes that legislators are too dumb to connect the fact that if their phone can search for dogs with “on-device processing,” then it could also search for contraband. I doubt it. And even if they are, the national security apparatus will surely discover this argument for them, aided by the Andurils and NSOs of the world. As I have repeatedly said: the reaction to this announcement sounds more like a collective reckoning of where we are as humans and not any particular new risk introduced by Apple. In the Apple vs. FBI letter, Tim urged us to have a discussion about encryption, when we want it, why we want it, and to what extent we should protect it. Instead, we elected Trump.
- samstave 5y ago
  So you have User A - they upload a pic with User A and Peoples B,C,D,E,Z icloud scans for those faces finds those faces and ties them to other ID accounts via face - then via fingerprint recognition to a device, and to a location based on IMEI etc. Apple's platform is literally the foundation for the most dystopian digital tool-set in history... Once the government is able to crack the apple war chest, everything is fucked.
  - samstave 5y ago
    --MBS has entered the chat. Go fuck yourself.
fraa-orolo 5y ago
A false positive in matching faces results in a click to fix it or a wrongly categorized photo. A false positive in this new thing may land you in jail or have your life destroyed. Even an allegation of something so heinous is enough to ruin a life. The "one in trillion" chance of false positives is Apple's invention. They haven't scanned trillions of photos and it's a guess. And you need multiple false positives, yet no one says how many, so it could be a low number. Either way, even with how small the chance of it being wrong is, the consequences for the individual are catastrophic. No one sane should accept that kind of risk/reward ratio. "Oh, and one more thing, and we think you'll love it. You can back up your entire camera roll for just $10 a month and a really infinitesimally minuscule chance that you and your family will be completely disgraced in the public eye, and you'll get raped and murdered in prison for nothing."
- systoll 5y ago
  Ok. So iCloud Photos circa 2020 [and Google Photos and Facebook and Dropbox and OneDrive] aren’t a risk you should be willing to take. This feature doesn’t change anything in that regard; the scanning was already happening.
  - XorNot 5y ago
    I literally do not take that risk in 2021. I do, currently, make the reasoned assurance that the computational overhead of pushing changes down to my phone, and the general international security community, are keeping me approximately abreast of whether my private device is actively spying on me (short answer: it definitely is, longer answer: but to what specific intent?) Apple's new policy is: "of course your phone is scanning and flagging your private files to our server - that's normal behavior! Don't worry about it".

lifty 5y ago

It’s not a false sense of security, it’s a clear delimitation between theirs and mine; Debian package maintainers can also slip a scanner on your machine but that is a big line to cross on purpose and without notifying the user.

totetsu 5y ago
But with a debian package you can choose not to accept the upgrade and see any funny business in the release source code..
- gpm 5y ago
  Somewhere along the line someone is producing and signing the binaries that find their way onto their computer, they could produce those binaries from different source code and I would be none the wiser. Debian tries to be reproducible, so to avoid being caught they might need to control the mirror to so that they could send it to only me. I.e. if I'm lucky it would take a total of 2 people to put malicious binaries on my computer (1 with a signing key, 1 with access to the mirror I download things from).
- least 5y ago
  That is technically true but in a real very practical sense everyone here using OSS absolutely is trusting a third party because they are not auditing every bit of code they run. For less technical people there is effectively zero difference between open and closed software. It’s really disingenuous to suggest that open source isn’t dependent on trust, you just change who you are trusting. Even if the case is someone else is auditing that code, you’re trusting that person instead of the repository owners. I’ll concede that at least that possibility to audit exists but personally I do have to trust to a certain extent that third parties aren’t trying to fuck me over.
  - totetsu 5y ago
    Thinking about this.. I guess my trust, is that someone smarter than I will notice it, cause a fuss, and the community will raise pitch forks and.. git forks. My trust is in the community, I hope it can stay healthy and diverse for all time.

stjohnswarts 5y ago

The only way what you said is not true for any networked device is to just go down to the river and throw it in and never use a digital device again. It's not a false sense of security, it's a calculated position on security and what you will accept, moving spying from the server to the phone was the last straw for a lot of people.