3 ms·

> Apple previously withdrew plans for its own CSAM-scanning feature for iCloud Photos, following pushback from customers and human rights groups. Apple’s soluti

by cj 3y ago

> Apple previously withdrew plans for its own CSAM-scanning feature for iCloud Photos, following pushback from customers and human rights groups. Apple’s solution was more privacy-preserving than what is now proposed by the UK government. Preface: I don't have an opinion on one side or the other. Question: what were the arguments lobbied against apple that caused them to withdraw? Also, is CSAM the motivating factor for UK or do they want backdoor for more general national security surveillance (or whatever you want to call it)?

can16358p 3y ago
Of course it's just about surveillance. CSAM is just an excuse that is a sensitive topic so it's easier to justify which masks their actual intentions.
alephnerd 3y ago
> Also, is CSAM the motivating factor for UK or do they want backdoor for more general national security surveillance Baroness Beeban Kidron has been lobbying for stringent anti-CSAM measures in tech for years [0]. She's lead a major pressure campaign in the US, Canada, and the UK for years [1]. Her charity 5Rights and WeProtect both have been able to back Labour and the Tories so she's able to lobby across the aisles. It doesn't hurt that the publishing company her parents founded (Pluto Press) has a strong niche in the political space. The Molly Russell suicide also played a role [2], which she leveraged to highlight the need for restrictive anti-CSAM measures, especially as it became a top tabloid story in the UK. [0] - https://en.m.wikipedia.org/wiki/Beeban_Kidron https://en.m.wikipedia.org/wiki/Beeban_Kidron [1] - https://www.politico.com/news/2023/06/14/british-baroness-online-safety-laws-00101854 https://www.politico.com/news/2023/06/14/british-baroness-on... [2] - https://www.nytimes.com/2022/10/01/business/instagram-suicide-ruling-britain.html https://www.nytimes.com/2022/10/01/business/instagram-suicid...
- tacticalturtle 3y ago
  I know I’m from a different culture - but I don’t understand how the UK still has the House of Lords in 2023. It just doesn’t seem aligned with the principles of a republic or democracy to have unelected lifetime members who are able to draft legislation.
  - wpietri 3y ago
    The British people broadly agree with you: https://en.wikipedia.org/wiki/Reform_of_the_House_of_Lords#Opinion_polling https://en.wikipedia.org/wiki/Reform_of_the_House_of_Lords#O...
  - pxeger1 3y ago
    I, broadly, think the House of Lords is a good thing in principle. Having a board of experts who can trounce things like the Illegal Migration Bill's potential for human rights abuse is a good thing. FYI, the House of Lords can't reject the same bill more than three times, and can't reject a bill that calls an election, so its undemocraticness isn't as bad as you might assume. There are many issues with it but I don't think many of them are much worse than those that already exist with the House of Commons. (Lobbying: exists in both. Unelectedness: the first past the post system is pretty bad here too. Ability of unelected people to introduce bills: private members' bills exist too). All of these are problems I'd like solved but I wouldn't support removing the House of Lords.
  - blibble 3y ago
    it's fine because it's subordinate to the commons in every way it acts as a brake for controversial legislation, which is generally a good thing
detourdog 3y ago
I think it was a preemptive attempt by Apple to avoid the current situation. They looked at the what was supported by 4th Amendment and designed a technical system that was the best compromise of no access and the ability to search for known images. They would rather have maintained the status quo but thought they had hit on something that would work. Since they didn't need to do anything pulling the software was an easy fix to the controversy. The governments are the ones that are pushing this so let them take the heat. Apple has a working solution and they can just wait and see what happens and react accordingly.
tiltowait 3y ago
One common argument was that bad state actors would get non-CSAM, but still "undesirable" (as defined by the state), material into the database and use it to target dissidents, badthink, etc. (Personally, I think this was an overblown risk, since the plan was for the images to first be reviewed by Apple before being sent to the authorities. Presumably if an Apple employee/contractor saw a photo of, say, Tank Man, they wouldn't pass it on. But it's reasonable to be concerned.)
- detourdog 3y ago
  I agree. That was just an argument using Apple's CSAM implementation as a vector for a bad state actor. The same misuse could be facilitate any number of ways. I have come to the conclusion that part of corporate combat involves apparent grassroots outrage. If a system without a wide open backdoor is proven effective it becomes more difficult to argue for a bigger backdoor.
vorpalhex 3y ago
The fear was that Apple would be coerced to add in non-CSAM hashes to the hit list, much as Apple is now compromised fully in China. Thus any undesirable material - a campaign poster, a meme, a video leak of classified info - would be yanked from customer devices and trigger this automated reporting flow. Apple already gives the Chinese government access to iCloud data for Chinese citizens. Being bullied into searching for eg anti-Xi material is a small jump. The other concern is that the hashes themselves are prone to false positives. This is true of the source hashes and concerns in the algorithm. The National Center for Missing & Exploited Children data has been known to match on pictures of tricycles and monkeys (clearly not CSAM). Do now we are in a world of pushing automated child abuse charges against people because a faulty algorithm. The whole thing is just prone to abuse.
kmeisthax 3y ago
Apple's solution was to run every photo through a perceptual hash algorithm with a neural network inside of it and then compare that against a list of known hashes. This is actually not materially different from the system that every image host has used for a while now. The main difference was that they were going to use a bunch of cryptography to ensure that... - Images could be scanned only after a threshold number of illegal images had been flagged - Law enforcement agencies could not use the system to flag non-CSAM images Part of the way they achieve the former - and what makes the system unworkable - is that it's client-sided. To be clear, every service already scans for CSAM. But the naive way of doing this - comparing SHA-2 message digests - can be trivially manipulated into producing false negatives. Just flip a bit in the image and it becomes a new image. Perceptual hash algorithms instead use notions of image similarity to protect against this, but it also introduces the possibility of false positives - i.e. images that look innocuous but match against CSAM and get you flagged. This is not merely a problem to be solved, but a consequence of such systems being differentiable, which is necessary for them to actually work. For remote scans, you can at least keep the perceptual hash algorithm secret and avoid leaking information about what images do and don't get flagged. This will frustrate attempts at adversarial training. However, when you put the perceptual hash on everyone's phone, it can be extracted, and people who don't like the idea of running a spying dragnet will deliberately break the hash just to tell you to fuck off. So people were making tools that would modify one image to look like another, which could be used to either hide CSAM or, worse, making cat pictures that get your iCloud flagged. There's also a bit of moral insult involved with having your phone do things that aren't in your benefit, even if this ostensibly were to keep the feds from demanding explicit backdoors[0]. This system is laughably easy to defeat if you're jailbroken - just turn the daemon off. Hence why Apple made no attempt to integrate it into macOS where users have root access[1]. Law enforcement would never firewall their capabilities to "just CSAM." That's not how the law works. You can't sue your drug dealer for not giving you the weed you paid for, and for similar reasons, law enforcement is never going to only wiretap pedophiles and not drug dealers. There's a few exceptions to this[2], but generally, you should assume that if you're about to give the feds the legal capability to spy, it will be used for every crime down to and potentially including routine traffic stops. Apple's bulwark against that was to have the system only scan for images that were flagged by multiple independent child protection agencies. However, this isn't a guarantee; the western world routinely collaborates in very not independent ways. There's no guarantee that, say, Europe's pedo-fighters wouldn't have had their arms twisted by their countries' intelligence apparatus, operating in concert with the American CIA who was arm-twisting NCMEC. Apple promised they'd be reviewing all reports before forwarding them to law enforcement, but in this situation there's nothing preventing them from having their arms twisted here, too[3]. [0] For the record, there was no evidence that Apple was using this system to enable iCloud end-to-end encryption. They'd already turned that off because of a UX problem: too many people were permanently locking themselves out of their own data and Apple couldn't break into it for them. [1] Root access on macOS is complicated by the fact that SIP and boot security exists. They could at least theoretically have made the kernel refuse to terminate the CSAM scanner process at all, but that would be a speed bump at best. Users absolutely can turn off SIP or boot modified kernels on Macs that would remove whatever protection Apple added to the CSAM scanner. And Apple currently has no interest in locking owners out of macOS like they did with iOS. [2] The NSA actually does try to keep their spying tools to themselves, because they have to fight nation-states, not criminals. Even then, they'll still slip hints under the door and tell the FBI they can only use them if they can make up a story that doesn't expose how the NSA's spying apparatus works. [3] National Security Letters are one hell of a drug. Apple could also have their arm twisted to add backdoors, but they've already shown that they'll yell loudly if that happens. Furthermore, the existence of software signatures means that such a backdoor would generate irrefutable paper trails, which is the sort of thing that intelligence agencies hate. It's much easier for them to just quietly extend an existing quasi-backdoor than to create an explicit one that loudly screams "DO NOT USE YOUR PHONE TO DO CRIMES"
gigel82 3y ago
Apple's "solution" was the technical equivalent of Microsoft having Windows Defender scanning every single file on a user computer and attaching a "hash" / tag to it. Then downloading a database of "forbidden hashes" from an unspecified government authority. Their initial plan was to only compare against the database when you upload to iCloud (in the comparison, OneDrive) and notify the authorities, but of course once the infrastructure for local scanning was in place, extending it to other "scenarios" was just a small update away (I'm sure RIAA/MPAA would love to hear about your old mp3 folder you kept moving over from pc to pc over the years).

by cj 3y ago

can16358p 3y ago

Of course it's just about surveillance. CSAM is just an excuse that is a sensitive topic so it's easier to justify which masks their actual intentions.

alephnerd 3y ago

> Also, is CSAM the motivating factor for UK or do they want backdoor for more general national security surveillance Baroness Beeban Kidron has been lobbying for stringent anti-CSAM measures in tech for years [0]. She's lead a major pressure campaign in the US, Canada, and the UK for years [1]. Her charity 5Rights and WeProtect both have been able to back Labour and the Tories so she's able to lobby across the aisles. It doesn't hurt that the publishing company her parents founded (Pluto Press) has a strong niche in the political space. The Molly Russell suicide also played a role [2], which she leveraged to highlight the need for restrictive anti-CSAM measures, especially as it became a top tabloid story in the UK. [0] - https://en.m.wikipedia.org/wiki/Beeban_Kidron https://en.m.wikipedia.org/wiki/Beeban_Kidron [1] - https://www.politico.com/news/2023/06/14/british-baroness-online-safety-laws-00101854 https://www.politico.com/news/2023/06/14/british-baroness-on... [2] - https://www.nytimes.com/2022/10/01/business/instagram-suicide-ruling-britain.html https://www.nytimes.com/2022/10/01/business/instagram-suicid...

tacticalturtle 3y ago
I know I’m from a different culture - but I don’t understand how the UK still has the House of Lords in 2023. It just doesn’t seem aligned with the principles of a republic or democracy to have unelected lifetime members who are able to draft legislation.
- wpietri 3y ago
  The British people broadly agree with you: https://en.wikipedia.org/wiki/Reform_of_the_House_of_Lords#Opinion_polling https://en.wikipedia.org/wiki/Reform_of_the_House_of_Lords#O...
- pxeger1 3y ago
  I, broadly, think the House of Lords is a good thing in principle. Having a board of experts who can trounce things like the Illegal Migration Bill's potential for human rights abuse is a good thing. FYI, the House of Lords can't reject the same bill more than three times, and can't reject a bill that calls an election, so its undemocraticness isn't as bad as you might assume. There are many issues with it but I don't think many of them are much worse than those that already exist with the House of Commons. (Lobbying: exists in both. Unelectedness: the first past the post system is pretty bad here too. Ability of unelected people to introduce bills: private members' bills exist too). All of these are problems I'd like solved but I wouldn't support removing the House of Lords.
- blibble 3y ago
  it's fine because it's subordinate to the commons in every way it acts as a brake for controversial legislation, which is generally a good thing

detourdog 3y ago

I think it was a preemptive attempt by Apple to avoid the current situation. They looked at the what was supported by 4th Amendment and designed a technical system that was the best compromise of no access and the ability to search for known images. They would rather have maintained the status quo but thought they had hit on something that would work. Since they didn't need to do anything pulling the software was an easy fix to the controversy. The governments are the ones that are pushing this so let them take the heat. Apple has a working solution and they can just wait and see what happens and react accordingly.

tiltowait 3y ago

One common argument was that bad state actors would get non-CSAM, but still "undesirable" (as defined by the state), material into the database and use it to target dissidents, badthink, etc. (Personally, I think this was an overblown risk, since the plan was for the images to first be reviewed by Apple before being sent to the authorities. Presumably if an Apple employee/contractor saw a photo of, say, Tank Man, they wouldn't pass it on. But it's reasonable to be concerned.)

detourdog 3y ago
I agree. That was just an argument using Apple's CSAM implementation as a vector for a bad state actor. The same misuse could be facilitate any number of ways. I have come to the conclusion that part of corporate combat involves apparent grassroots outrage. If a system without a wide open backdoor is proven effective it becomes more difficult to argue for a bigger backdoor.

vorpalhex 3y ago

The fear was that Apple would be coerced to add in non-CSAM hashes to the hit list, much as Apple is now compromised fully in China. Thus any undesirable material - a campaign poster, a meme, a video leak of classified info - would be yanked from customer devices and trigger this automated reporting flow. Apple already gives the Chinese government access to iCloud data for Chinese citizens. Being bullied into searching for eg anti-Xi material is a small jump. The other concern is that the hashes themselves are prone to false positives. This is true of the source hashes and concerns in the algorithm. The National Center for Missing & Exploited Children data has been known to match on pictures of tricycles and monkeys (clearly not CSAM). Do now we are in a world of pushing automated child abuse charges against people because a faulty algorithm. The whole thing is just prone to abuse.

kmeisthax 3y ago

Apple's solution was to run every photo through a perceptual hash algorithm with a neural network inside of it and then compare that against a list of known hashes. This is actually not materially different from the system that every image host has used for a while now. The main difference was that they were going to use a bunch of cryptography to ensure that... - Images could be scanned only after a threshold number of illegal images had been flagged - Law enforcement agencies could not use the system to flag non-CSAM images Part of the way they achieve the former - and what makes the system unworkable - is that it's client-sided. To be clear, every service already scans for CSAM. But the naive way of doing this - comparing SHA-2 message digests - can be trivially manipulated into producing false negatives. Just flip a bit in the image and it becomes a new image. Perceptual hash algorithms instead use notions of image similarity to protect against this, but it also introduces the possibility of false positives - i.e. images that look innocuous but match against CSAM and get you flagged. This is not merely a problem to be solved, but a consequence of such systems being differentiable, which is necessary for them to actually work. For remote scans, you can at least keep the perceptual hash algorithm secret and avoid leaking information about what images do and don't get flagged. This will frustrate attempts at adversarial training. However, when you put the perceptual hash on everyone's phone, it can be extracted, and people who don't like the idea of running a spying dragnet will deliberately break the hash just to tell you to fuck off. So people were making tools that would modify one image to look like another, which could be used to either hide CSAM or, worse, making cat pictures that get your iCloud flagged. There's also a bit of moral insult involved with having your phone do things that aren't in your benefit, even if this ostensibly were to keep the feds from demanding explicit backdoors[0]. This system is laughably easy to defeat if you're jailbroken - just turn the daemon off. Hence why Apple made no attempt to integrate it into macOS where users have root access[1]. Law enforcement would never firewall their capabilities to "just CSAM." That's not how the law works. You can't sue your drug dealer for not giving you the weed you paid for, and for similar reasons, law enforcement is never going to only wiretap pedophiles and not drug dealers. There's a few exceptions to this[2], but generally, you should assume that if you're about to give the feds the legal capability to spy, it will be used for every crime down to and potentially including routine traffic stops. Apple's bulwark against that was to have the system only scan for images that were flagged by multiple independent child protection agencies. However, this isn't a guarantee; the western world routinely collaborates in very not independent ways. There's no guarantee that, say, Europe's pedo-fighters wouldn't have had their arms twisted by their countries' intelligence apparatus, operating in concert with the American CIA who was arm-twisting NCMEC. Apple promised they'd be reviewing all reports before forwarding them to law enforcement, but in this situation there's nothing preventing them from having their arms twisted here, too[3]. [0] For the record, there was no evidence that Apple was using this system to enable iCloud end-to-end encryption. They'd already turned that off because of a UX problem: too many people were permanently locking themselves out of their own data and Apple couldn't break into it for them. [1] Root access on macOS is complicated by the fact that SIP and boot security exists. They could at least theoretically have made the kernel refuse to terminate the CSAM scanner process at all, but that would be a speed bump at best. Users absolutely can turn off SIP or boot modified kernels on Macs that would remove whatever protection Apple added to the CSAM scanner. And Apple currently has no interest in locking owners out of macOS like they did with iOS. [2] The NSA actually does try to keep their spying tools to themselves, because they have to fight nation-states, not criminals. Even then, they'll still slip hints under the door and tell the FBI they can only use them if they can make up a story that doesn't expose how the NSA's spying apparatus works. [3] National Security Letters are one hell of a drug. Apple could also have their arm twisted to add backdoors, but they've already shown that they'll yell loudly if that happens. Furthermore, the existence of software signatures means that such a backdoor would generate irrefutable paper trails, which is the sort of thing that intelligence agencies hate. It's much easier for them to just quietly extend an existing quasi-backdoor than to create an explicit one that loudly screams "DO NOT USE YOUR PHONE TO DO CRIMES"

gigel82 3y ago

Apple's "solution" was the technical equivalent of Microsoft having Windows Defender scanning every single file on a user computer and attaching a "hash" / tag to it. Then downloading a database of "forbidden hashes" from an unspecified government authority. Their initial plan was to only compare against the database when you upload to iCloud (in the comparison, OneDrive) and notify the authorities, but of course once the infrastructure for local scanning was in place, extending it to other "scenarios" was just a small update away (I'm sure RIAA/MPAA would love to hear about your old mp3 folder you kept moving over from pc to pc over the years).