7 ms·

> (though that could easily be the same even without all these CSAM measures.. not a new threat) Apple has historically avoided being pressured by governments

> (though that could easily be the same even without all these CSAM measures.. not a new threat) Apple has historically avoided being pressured by governments to allow this kind of surveillance by arguing they can't be forced to create functionality that doesn't exist, or hand over information they don't have. It's the argument they made in the San Bernadino case. If they release this, it'll be much harder to avoid giving governments that existing ability for other, non-CSAM uses.

shuckles 5y ago
The request from the FBI in the San Bernardino case was to change a passcode limit constant and retry timeouts. Those are about as trivial to implement as any of the convoluted government coercion database attacks against CSAM detection being proposed here.
- ajsnigrutin 5y ago
  The difference here is, that apple would have to develop a new feature for them, test it, and waste millions in lawers to protect themselves from accusations of tampering with the evidence (which a software update definitely is, and who knows what FBI wanted in that software update, maybe even to insert a fake sms to the sms database, or many other things a good defense lawyer could bring to the jury). Here, it's different.. let's say there's a new wikileaks, photos of secret documents... and again, first a few journalists get the data, start slowly writing articles, and FBI just adds the hashes to the database, and they can find out who has the photos, with metadata even who had the first, before the journalist, and they can find a leak.
  - azinman2 5y ago
    But the FBI can’t just add the hashes to the db. That’s why it’s the intersection of two dbs in two jurisdictions… to prevent exactly that kind of attack. Then they need to pass a human reviewer as well.
    - xkcd-sucks 5y ago
      Five Eyes
      - simondotau 5y ago
        You are suggesting that another country could launder the request on behalf of the USA in order to circumvent the 4th Amendment. Okay then, let's play that out. Australia contacts Apple and demands they augment CSAM detection so that every iPhone in the USA is now scanning for image hashes supplied by Australia. Apple says no. End of hypothetical.
        salawat 5y ago
        You haven't heard of how GCHQ would happily hand over intelligence they had on U.S. Citizens? I know for a fact there are efforts at creating fusion centers across national boundaries. The question you need to ask is not if but what will make you interesting enough to mobilize against. Thou shalt not build the effing Panopticon, nor it's predecessors. Is that so hard to not do.
        simondotau 5y ago
        I've no doubt that GCHQ would hand whatever they like to whomever they like. But why would Apple comply with a demand from the GCHQ to spy on US citizens?
        brokenmachine 5y ago
        They will come up with some kind of fuckery to get around the rules, like some crypto ridiculousness that makes files that match a certain hash go to servers in a different jurisdiction? Far-fetched, I know...
        nonlocutor 5y ago
        https://www.lawfareblog.com/legal-tetris-and-fbis-anom-program https://www.lawfareblog.com/legal-tetris-and-fbis-anom-progr... “ The Australian Telecommunications and Other Legislation Amendment (Assistance and Access) Act 2018 (TOLA) allows government agencies to issue “technical assistance” and “technical capability” notices to providers of communications services. The notices require that the providers give the authorities help in conducting criminal enforcement intercepts, and that they make changes in their systems to ensure that they can give that help.” … “In any event, the FBI chose a curiously roundabout way of getting access to ANOM messages. For ANOM devices operating outside of the United States, “an encrypted [blind carbon copy or BCC]” of each message that a user sent was transmitted to a server located outside of the United States, which then decrypted and reencrypted the message with an encryption key known to the FBI. Those reencrypted messages were sent to another server that was owned by the FBI, outside of the United States. In the summer of 2019, the FBI started negotiating to build the legal structure that would make this technical architecture work. In essence, the FBI went looking for a third country that would host the BCC server and could lawfully accept all of the decrypted messages and send the copies to the FBI. As the affidavit notes, “Unlike the Australian beta test, the third country would not review the content in the first instance.” This would have been a fascinating negotiation. Both participants wanted to make criminal cases and avoid privacy scandals. The U.S. would want to be sure that the third country had full legal authority to intercept the contents of every ANOM message, and that the country was also willing to share the full ANOM take with the U.S. in something like real time.”
  - snowwrestler 5y ago
    > FBI just adds the hashes to the database This is the crux of the argument right here, and I have yet to see a detailed description of how the FBI would go about doing that. The most detail I’ve seen is in this thread, which suggests that it would be difficult for the FBI to do it, or at least do it more than once. https://twitter.com/pwnallthethings/status/1424873629003702273 https://twitter.com/pwnallthethings/status/14248736290037022... Has anyone seen something like this in the other direction? Something that walks through “the FBI would do this, then this, etc. and now they’ve coopted Apple’s system”?
- simondotau 5y ago
  The passcode limit constant is enforced by the secure enclave. I don't know if it's been proven that the secure enclave component of the device can be changed without the device being unlocked. I'm not even sure it possible for any operating system updates to occur on a device which is locked.
  - shuckles 5y ago
    Not on the iPhone 5C which was the phone used by the terrorist and did not have an SE. Locked iPhones can be updated from DFU, but I think SE firmware can’t.
  - snowwrestler 5y ago
    The technical feasibility of the FBI’s request was never the question, nor the basis of Apple’s objection. Of course it’s even easier for Apple to say “no” to the government if they literally cannot do what the government is asking. That’s the basis of the EFF’s objection to Apple’s plans: they think that by implementing this CSAM system, Apple will turn an impossibility into a possibility.
    - shuckles 5y ago
      We were never in a world where Apple was unable to do what the government was asking. Nothing impossible has been made possible, and Apple has made a stand against small changes - like changing constants - before.
      - HWR_14 5y ago
        It's not "impossibility". It's cost. The FBI cannot force Apple to do free work to circumvent security. Can they force them to add one more hash? Is that work?
        shuckles 5y ago
        I’m not a lawyer, but at the very least Apple would have to train its reviewers to recognize new image types, redact or update all statements where they explicitly drew a line in the sand about what the feature searched for, likely produce a custom build of iOS, update its knowledge base to distinguish between the root hash of the CSAM database and FBI Imagery database, perhaps re-present consent, add code to target this new database (or subset of the full database) to US users, etc. I’m not a product manager either, but superficially simple changes are often quite complex!
pfranz 5y ago
Ehh. I'd argue Apple has a mixed record. It's well known that iCloud backups are unencrypted and often handed over to authorities. In Jan 2020[1], it was reported they planned to encrypt backups, but dropped the rollout due to pressure from the FBI. I'm surprised they don't get called out because the difference between this and San Bernadino makes sense to me from a technical standpoint, from a practical standpoint and to the laymen it seems hypocritical. In this case, I actually kind of buy Apple's argument that this will make it harder to cowtow to governments (assuming they encrypt iCloud photos at some point). Right now they can scan iCloud data and hand over photos and accounts without user's knowledge. They can do that without informing users (like they currently do with backups). With this in place the database and logic ships with the OS. They would have to implement and ship any changes world-wide. Users will have to install that update. An alternative is Apple silently making a server-side change affecting specific customers or countries. With that said, I do understand people's concern over the change. [1] https://www.reuters.com/article/us-apple-fbi-icloud-exclusive-idUSKBN1ZK1CT https://www.reuters.com/article/us-apple-fbi-icloud-exclusiv...
- dwaite 5y ago
  iCloud backups _are_ encrypted, but have an HSM-escrowed key process on court order. People like to pretend crypto is always E2E or nothing, but escrowed keys do mean that the process of checking cloud-backed data has an auditable release process, that the keys to your data are outside the cloud-hosted infrastructure, and that there is no support for blanket data scanning. > assuming they encrypt iCloud photos at some point iCloud photos are already encrypted. See above.
anonuser123456 5y ago
Not really. Apple simply tells the DOJ if any non-CSAM content is added surreptitiously to the DB, Apple will drop CSAM scanning. Also, if DOJ makes any request to scan for non CSAM by court order or warrant, Apple will in kind drop support for the technology. Apple is making the good faith effort here. If DOJ makes a bad faith effort, Apple is in now way required to continue to participate.
- istingray 5y ago
  In this scenario, how would the first criteria be met? "Content is added surreptitiously to the DB"? From my understanding Apple blindly accepts the list of hashes from their trusted soueces.
  - anonuser123456 5y ago
    Apple will see the results of the matched images however. Given that Apple believes their algorithm will generate an impossibly small quantity of false positives, if they start to see political images, sensitive documents etc. show up, that tips them off the DB is corrupt. All of these processes will be subject to discovery in the very first trial generated by this technology. If there is evidence that governments are polluting the DB with non-CSAM content, then warrants issued on evidence from the DB can be overturned. Prosecutors in DOJ and FBI etc. have a strong incentive to ensure that DB doesn't turn into a free-for-all dragnet because it could work to invalidate their ability to use evidence from legitimate CSAM cases.

shuckles 5y ago

The request from the FBI in the San Bernardino case was to change a passcode limit constant and retry timeouts. Those are about as trivial to implement as any of the convoluted government coercion database attacks against CSAM detection being proposed here.

ajsnigrutin 5y ago
The difference here is, that apple would have to develop a new feature for them, test it, and waste millions in lawers to protect themselves from accusations of tampering with the evidence (which a software update definitely is, and who knows what FBI wanted in that software update, maybe even to insert a fake sms to the sms database, or many other things a good defense lawyer could bring to the jury). Here, it's different.. let's say there's a new wikileaks, photos of secret documents... and again, first a few journalists get the data, start slowly writing articles, and FBI just adds the hashes to the database, and they can find out who has the photos, with metadata even who had the first, before the journalist, and they can find a leak.
- azinman2 5y ago
  But the FBI can’t just add the hashes to the db. That’s why it’s the intersection of two dbs in two jurisdictions… to prevent exactly that kind of attack. Then they need to pass a human reviewer as well.
  - xkcd-sucks 5y ago
    Five Eyes
    - simondotau 5y ago
      You are suggesting that another country could launder the request on behalf of the USA in order to circumvent the 4th Amendment. Okay then, let's play that out. Australia contacts Apple and demands they augment CSAM detection so that every iPhone in the USA is now scanning for image hashes supplied by Australia. Apple says no. End of hypothetical.
      - salawat 5y ago
        You haven't heard of how GCHQ would happily hand over intelligence they had on U.S. Citizens? I know for a fact there are efforts at creating fusion centers across national boundaries. The question you need to ask is not if but what will make you interesting enough to mobilize against. Thou shalt not build the effing Panopticon, nor it's predecessors. Is that so hard to not do.
        simondotau 5y ago
        I've no doubt that GCHQ would hand whatever they like to whomever they like. But why would Apple comply with a demand from the GCHQ to spy on US citizens?
        brokenmachine 5y ago
        They will come up with some kind of fuckery to get around the rules, like some crypto ridiculousness that makes files that match a certain hash go to servers in a different jurisdiction? Far-fetched, I know...
      - nonlocutor 5y ago
        https://www.lawfareblog.com/legal-tetris-and-fbis-anom-program https://www.lawfareblog.com/legal-tetris-and-fbis-anom-progr... “ The Australian Telecommunications and Other Legislation Amendment (Assistance and Access) Act 2018 (TOLA) allows government agencies to issue “technical assistance” and “technical capability” notices to providers of communications services. The notices require that the providers give the authorities help in conducting criminal enforcement intercepts, and that they make changes in their systems to ensure that they can give that help.” … “In any event, the FBI chose a curiously roundabout way of getting access to ANOM messages. For ANOM devices operating outside of the United States, “an encrypted [blind carbon copy or BCC]” of each message that a user sent was transmitted to a server located outside of the United States, which then decrypted and reencrypted the message with an encryption key known to the FBI. Those reencrypted messages were sent to another server that was owned by the FBI, outside of the United States. In the summer of 2019, the FBI started negotiating to build the legal structure that would make this technical architecture work. In essence, the FBI went looking for a third country that would host the BCC server and could lawfully accept all of the decrypted messages and send the copies to the FBI. As the affidavit notes, “Unlike the Australian beta test, the third country would not review the content in the first instance.” This would have been a fascinating negotiation. Both participants wanted to make criminal cases and avoid privacy scandals. The U.S. would want to be sure that the third country had full legal authority to intercept the contents of every ANOM message, and that the country was also willing to share the full ANOM take with the U.S. in something like real time.”
- snowwrestler 5y ago
  > FBI just adds the hashes to the database This is the crux of the argument right here, and I have yet to see a detailed description of how the FBI would go about doing that. The most detail I’ve seen is in this thread, which suggests that it would be difficult for the FBI to do it, or at least do it more than once. https://twitter.com/pwnallthethings/status/1424873629003702273 https://twitter.com/pwnallthethings/status/14248736290037022... Has anyone seen something like this in the other direction? Something that walks through “the FBI would do this, then this, etc. and now they’ve coopted Apple’s system”?
simondotau 5y ago
The passcode limit constant is enforced by the secure enclave. I don't know if it's been proven that the secure enclave component of the device can be changed without the device being unlocked. I'm not even sure it possible for any operating system updates to occur on a device which is locked.
- shuckles 5y ago
  Not on the iPhone 5C which was the phone used by the terrorist and did not have an SE. Locked iPhones can be updated from DFU, but I think SE firmware can’t.
- snowwrestler 5y ago
  The technical feasibility of the FBI’s request was never the question, nor the basis of Apple’s objection. Of course it’s even easier for Apple to say “no” to the government if they literally cannot do what the government is asking. That’s the basis of the EFF’s objection to Apple’s plans: they think that by implementing this CSAM system, Apple will turn an impossibility into a possibility.
  - shuckles 5y ago
    We were never in a world where Apple was unable to do what the government was asking. Nothing impossible has been made possible, and Apple has made a stand against small changes - like changing constants - before.
    - HWR_14 5y ago
      It's not "impossibility". It's cost. The FBI cannot force Apple to do free work to circumvent security. Can they force them to add one more hash? Is that work?
      - shuckles 5y ago
        I’m not a lawyer, but at the very least Apple would have to train its reviewers to recognize new image types, redact or update all statements where they explicitly drew a line in the sand about what the feature searched for, likely produce a custom build of iOS, update its knowledge base to distinguish between the root hash of the CSAM database and FBI Imagery database, perhaps re-present consent, add code to target this new database (or subset of the full database) to US users, etc. I’m not a product manager either, but superficially simple changes are often quite complex!

pfranz 5y ago

Ehh. I'd argue Apple has a mixed record. It's well known that iCloud backups are unencrypted and often handed over to authorities. In Jan 2020[1], it was reported they planned to encrypt backups, but dropped the rollout due to pressure from the FBI. I'm surprised they don't get called out because the difference between this and San Bernadino makes sense to me from a technical standpoint, from a practical standpoint and to the laymen it seems hypocritical. In this case, I actually kind of buy Apple's argument that this will make it harder to cowtow to governments (assuming they encrypt iCloud photos at some point). Right now they can scan iCloud data and hand over photos and accounts without user's knowledge. They can do that without informing users (like they currently do with backups). With this in place the database and logic ships with the OS. They would have to implement and ship any changes world-wide. Users will have to install that update. An alternative is Apple silently making a server-side change affecting specific customers or countries. With that said, I do understand people's concern over the change. [1] https://www.reuters.com/article/us-apple-fbi-icloud-exclusive-idUSKBN1ZK1CT https://www.reuters.com/article/us-apple-fbi-icloud-exclusiv...

dwaite 5y ago
iCloud backups _are_ encrypted, but have an HSM-escrowed key process on court order. People like to pretend crypto is always E2E or nothing, but escrowed keys do mean that the process of checking cloud-backed data has an auditable release process, that the keys to your data are outside the cloud-hosted infrastructure, and that there is no support for blanket data scanning. > assuming they encrypt iCloud photos at some point iCloud photos are already encrypted. See above.

anonuser123456 5y ago

Not really. Apple simply tells the DOJ if any non-CSAM content is added surreptitiously to the DB, Apple will drop CSAM scanning. Also, if DOJ makes any request to scan for non CSAM by court order or warrant, Apple will in kind drop support for the technology. Apple is making the good faith effort here. If DOJ makes a bad faith effort, Apple is in now way required to continue to participate.

istingray 5y ago
In this scenario, how would the first criteria be met? "Content is added surreptitiously to the DB"? From my understanding Apple blindly accepts the list of hashes from their trusted soueces.
- anonuser123456 5y ago
  Apple will see the results of the matched images however. Given that Apple believes their algorithm will generate an impossibly small quantity of false positives, if they start to see political images, sensitive documents etc. show up, that tips them off the DB is corrupt. All of these processes will be subject to discovery in the very first trial generated by this technology. If there is evidence that governments are polluting the DB with non-CSAM content, then warrants issued on evidence from the DB can be overturned. Prosecutors in DOJ and FBI etc. have a strong incentive to ensure that DB doesn't turn into a free-for-all dragnet because it could work to invalidate their ability to use evidence from legitimate CSAM cases.