12 ms·

Given that they obviously already have the capability to send software updates that add new on-device capability, this seems like a meaningless distinction. It’

Given that they obviously already have the capability to send software updates that add new on-device capability, this seems like a meaningless distinction. It’s already “just policy” preventing them from sending any conceivable software update to their phones.

Dylan16807 5y ago
Making a custom OS update is pretty different from adding an entry into a database, and could also be mitigated against.
hn_throwaway_99 5y ago
I disagree, strongly. Let's say you're authoritarian government EvilGov. Before this announcement, if you went to Apple and said "we want you to push this spyware to your iPhones", Apple would and could have easily pushed back both in the court of public opinion and the court of law. Now though, Apple is already saying "We'll take this database of illegal image hashes provided by the government and use it to scan your phone." It's now quite trivial for a government to say "We don't have a special database of just CSAM, and a different database of just Winnie the Pooh memes. We just have one big DB of 'illegal images', and that's what we want you to use to scan the phones."
- ribosometronome 5y ago
  Why do you think essentially no one is complaining about using ML to understand the content of photos, then (especially in comparison to this rather targeted CSAM feature)? My impression is that both Apple and Google have already been doing that since what? 2016? Earlier? There's been no need for a database of photos, either company could silently update those algorithms to ping on guns, drugs, Winnie the Pooh memes, etc.
  - salawat 5y ago
    raises hand Have been whining about big data for and ML recognition systems for years.
  - alwillis 5y ago
    Why do you think essentially no one is complaining about using ML to understand the content of photos… At last in Apple's case, ML is used only if a minor child (less than 13 years old) who is on a Family account where the parent/guardian has opted-in to the ability to be alerted if potentially bad content is either sent or received using the Messages app.
    - ChrisKnott 5y ago
      By default iPhones autocategorise your photos, don't they?
      - katbyte 5y ago
        The sure do, and now extract text from images
    - ribosometronome 5y ago
      Google and Apple both use ML to understand the content of photos already. Go into your app and you can search for things like "dog", "beer", "birdhouse", and "revolver". Given that it can do all that (and already knows a difference between type of guns), it doesn't seem like a stretch to think it could, if Apple or Google wanted, understand "cocaine", "Winnie the Pooh memes", or whatever repressive-government style worries are had. And it's existed for years, without slippery sloping into anything other than this Messages feature for children.
      - ctdonath 5y ago
        Bingo. All Apple has done is add another category for “nude child” with a simple notification when the AI finds one. The tech has arrived; consequences happen.
- emodendroket 5y ago
  Could they though? An authoritarian government could just as easily say "if you do not implement this feature, we will not allow you to operate in this country" and refuse to entertain legal challenges.
  - adrianN 5y ago
    Apple is a fairly large company, that gives them some monetary leverage over governments. It's not as simple as you make it seem, I think.
    - emodendroket 5y ago
      Google is pretty big too but they implemented the filtering in China as requested. Then when they decided they didn't want to do that anymore there went their site.
    - pjerem 5y ago
      Should this fact be reassuring or even more frightening ? Because it’s something to push back against governments by saying « I can’t ». But it’s a societal issue if a corporation can say « I don’t want » to a government. It’s really not the same thing and Apple just burned their « I can’t » card and are implying they can just say « no » to governments. Which is quite an even more dystopian thing.
      - spacedcowboy 5y ago
        They can say "no" but they have to cope with the fallout from that. The statement from Apple is that they will say no. Whether that actually ever happens is something we will see, one way or another.
        pseudalopex 5y ago
        Apple won't say no to a court order.
        5y ago
- deleted 5y ago
  [deleted]
- new_realist 5y ago
  Authoritarian governments have never needed preexisting technical gizmos in order to abuse their citizens. Look at what Russia and the Nazis did with no magic tech at all. Privacy stems from valuing human rights, not easily disproven lies about it being difficult to actually implement the spying.
  - kragen 5y ago
    On the contrary, the reason Stalin and Hitler could enact repressive measures unprecedented in human history was precisely new technological developments like the telephone, the battle tank, and IBM cards. Bad people will never value human rights; getting privacy from bad people requires you to make it actually difficult to implement the spying, rather than just lie about it. That's why the slogan says, "cypherpunks write code."
    - Grustaf 5y ago
      No, they could enact repressive measures because they had almost absolute power. The form that oppression took would obviously rely on the then current technology, but the fundamental enabler of oppression was power, not technology.
      - wizzwizz4 5y ago
        > but the fundamental enabler of oppression was power, not technology. But bad people will always have power.
      - vimacs2 5y ago
        Power which was extended through the use of said technology. I'm fine with focusing on regressive hierarchical systems as the fundamental root of oppression but pretending like the information and mechanical technology introduced in the late 19th century to early 20th century did not massively increase the reach of the state is just being ridiculous. Prior to motorised vehicles and the telegraph for example, borders were almost impossible to enforce on the general population without significant leakage. Ironically in spite of powered flight, freedom of movement is in a sense significantly less than even 200 years ago when it comes to movement between countries that do not already have diplomatic ties to one another allowing for state authorized travel. Incidentally, this is part of why many punishments in the medieval age were so extreme - the actual ability of the state to enforce the law was so pathetically limited by today's standards that they needed a strong element of fear to attempt to control the population.
- strogonoff 5y ago
  > Apple is already saying "We'll take this database of illegal image hashes provided by the government and use it to scan your phone." This is incorrect. Apple has been saying—since 2019![0]—that they can scan for any potentially illegal content, not just images, not just CSAM, and not even strictly illegal. That's what should be opposed. CSAM is a red herring. [0] https://www.macobserver.com/analysis/apple-scans-uploaded-content/ https://www.macobserver.com/analysis/apple-scans-uploaded-co...
  - noduerme 5y ago
    The difference is that's scanning in the cloud, on their servers, of things people choose to upload. It's not scanning on the user's private device, and currently they have no way to scan what's on a private device.
    - strogonoff 5y ago
      The phrase is “pre-screening” of uploaded content, which is what is happening. I'm pretty sure this change in ToS was made to enable this CSAM feature.
- spacedcowboy 5y ago
  There is so much disinformation about this with people not being informed. Apple have put out clear documentation, it would be a good idea to read it before fear-mongering. 1) The list of CSAM hashes is that provided by the relevant US government agency, that is it. 2) The project is not targeted to roll out anywhere other than the USA. 3) The hashes are baked into the OS, there is no capability to update them other than on signed and delivered Apple OS updates. 4) Apple has exactly the same leverage with EvilGov as at any other time. They can refuse to do as asked, and risk being ejected from the country. Their stated claim is that this is what will happen. We will see.
  - benmmurphy 5y ago
    Regarding 3: it’s very easy to make a mistake in the protocol that would allow apple to detect hashes outside the CSAM list. Without knowing exactly how their protocol works it’s difficult to know whether it is correct. https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni... For example here is a broken PSI protocol in terms of point 3. I don’t think normally in PSI this is considered broken because the server knows the value so it is part of its private set. Server computes M_s = g . H(m) . S_s where g is a generator of an elliptic curve, H(m) is the neural hash of the image and S_s is the server blinding secret. The client computes M_sc = M_s . S_c where S_c is the client ephemeral secret. This M_sc value is the shared key. The client also computes M_c = g . H(m) . S_c and sends the M_c value to the server. The server can now compute M_cs = M_c . S_s = M_sc since they both used the same H(m) values. This allows the server and client to share a key based on the shared image. However, what happens if the client does it’s step using the ‘wrong’ image. If 3) is to hold it should not be possible for the server to compute the key. Client computes: M_sc = M_s . S_c M_c = g. H(m’) . S_c The clients final key share is: M_sc = g . H(m) . S_c . S_s Now server computes: M_cs = M_c . S_s = g . H(m’) . S_c . S_s The secret shares don’t match. But if the server knows H(m’) it can compute: M_cs’ = M_cs . inv(H(m’)) . H(m) and this secret share will match Normally this client side list in PSI is just used to speed up the protocol so the server does not have to do a crypto operation for every element in its set. It is not a pre-commitment from the server. Also, maybe the way I’m doing it here is just normally broken because it is not robust against low entropy inputs to the hash function. I've also reversed some of apple's non-public crypto that is used in some of it's services and they have made dubious design decisions in the past they have created weird weaknesses. Without knowing exactly what they are doing I would not try and infer properties that might not exist or trust their implementation.
- OneLeggedCat 5y ago
  > just Winnie the Pooh memes I think it's hilarious that one of the world' great nations would scan for exactly this.
kelnos 5y ago
There's a difference -- often a big one -- between being theoretically able to build some feature into a phone (plus the server-side infra and staffing to support it), and actually having built it. That's the capability angle. However, the difference between having a feature enabled based on the value of a seemingly-unrelated user-controlled setting, versus having that feature enabled all the time... is basically zero. Additionally, extending this feature to encompass other kinds of content is a matter of data entry, not engineering. That's the policy angle. When you don't yet have a capability, it might take a lot of work and commitment to develop it. But on the contrary, policy can be changed with the flip of a switch.

Dylan16807 5y ago

Making a custom OS update is pretty different from adding an entry into a database, and could also be mitigated against.

hn_throwaway_99 5y ago

I disagree, strongly. Let's say you're authoritarian government EvilGov. Before this announcement, if you went to Apple and said "we want you to push this spyware to your iPhones", Apple would and could have easily pushed back both in the court of public opinion and the court of law. Now though, Apple is already saying "We'll take this database of illegal image hashes provided by the government and use it to scan your phone." It's now quite trivial for a government to say "We don't have a special database of just CSAM, and a different database of just Winnie the Pooh memes. We just have one big DB of 'illegal images', and that's what we want you to use to scan the phones."

ribosometronome 5y ago
Why do you think essentially no one is complaining about using ML to understand the content of photos, then (especially in comparison to this rather targeted CSAM feature)? My impression is that both Apple and Google have already been doing that since what? 2016? Earlier? There's been no need for a database of photos, either company could silently update those algorithms to ping on guns, drugs, Winnie the Pooh memes, etc.
- salawat 5y ago
  raises hand Have been whining about big data for and ML recognition systems for years.
- alwillis 5y ago
  Why do you think essentially no one is complaining about using ML to understand the content of photos… At last in Apple's case, ML is used only if a minor child (less than 13 years old) who is on a Family account where the parent/guardian has opted-in to the ability to be alerted if potentially bad content is either sent or received using the Messages app.
  - ChrisKnott 5y ago
    By default iPhones autocategorise your photos, don't they?
    - katbyte 5y ago
      The sure do, and now extract text from images
  - ribosometronome 5y ago
    Google and Apple both use ML to understand the content of photos already. Go into your app and you can search for things like "dog", "beer", "birdhouse", and "revolver". Given that it can do all that (and already knows a difference between type of guns), it doesn't seem like a stretch to think it could, if Apple or Google wanted, understand "cocaine", "Winnie the Pooh memes", or whatever repressive-government style worries are had. And it's existed for years, without slippery sloping into anything other than this Messages feature for children.
    - ctdonath 5y ago
      Bingo. All Apple has done is add another category for “nude child” with a simple notification when the AI finds one. The tech has arrived; consequences happen.
emodendroket 5y ago
Could they though? An authoritarian government could just as easily say "if you do not implement this feature, we will not allow you to operate in this country" and refuse to entertain legal challenges.
- adrianN 5y ago
  Apple is a fairly large company, that gives them some monetary leverage over governments. It's not as simple as you make it seem, I think.
  - emodendroket 5y ago
    Google is pretty big too but they implemented the filtering in China as requested. Then when they decided they didn't want to do that anymore there went their site.
  - pjerem 5y ago
    Should this fact be reassuring or even more frightening ? Because it’s something to push back against governments by saying « I can’t ». But it’s a societal issue if a corporation can say « I don’t want » to a government. It’s really not the same thing and Apple just burned their « I can’t » card and are implying they can just say « no » to governments. Which is quite an even more dystopian thing.
    - spacedcowboy 5y ago
      They can say "no" but they have to cope with the fallout from that. The statement from Apple is that they will say no. Whether that actually ever happens is something we will see, one way or another.
      - pseudalopex 5y ago
        Apple won't say no to a court order.
        5y ago
deleted 5y ago
[deleted]
new_realist 5y ago
Authoritarian governments have never needed preexisting technical gizmos in order to abuse their citizens. Look at what Russia and the Nazis did with no magic tech at all. Privacy stems from valuing human rights, not easily disproven lies about it being difficult to actually implement the spying.
- kragen 5y ago
  On the contrary, the reason Stalin and Hitler could enact repressive measures unprecedented in human history was precisely new technological developments like the telephone, the battle tank, and IBM cards. Bad people will never value human rights; getting privacy from bad people requires you to make it actually difficult to implement the spying, rather than just lie about it. That's why the slogan says, "cypherpunks write code."
  - Grustaf 5y ago
    No, they could enact repressive measures because they had almost absolute power. The form that oppression took would obviously rely on the then current technology, but the fundamental enabler of oppression was power, not technology.
    - wizzwizz4 5y ago
      > but the fundamental enabler of oppression was power, not technology. But bad people will always have power.
    - vimacs2 5y ago
      Power which was extended through the use of said technology. I'm fine with focusing on regressive hierarchical systems as the fundamental root of oppression but pretending like the information and mechanical technology introduced in the late 19th century to early 20th century did not massively increase the reach of the state is just being ridiculous. Prior to motorised vehicles and the telegraph for example, borders were almost impossible to enforce on the general population without significant leakage. Ironically in spite of powered flight, freedom of movement is in a sense significantly less than even 200 years ago when it comes to movement between countries that do not already have diplomatic ties to one another allowing for state authorized travel. Incidentally, this is part of why many punishments in the medieval age were so extreme - the actual ability of the state to enforce the law was so pathetically limited by today's standards that they needed a strong element of fear to attempt to control the population.
strogonoff 5y ago
> Apple is already saying "We'll take this database of illegal image hashes provided by the government and use it to scan your phone." This is incorrect. Apple has been saying—since 2019![0]—that they can scan for any potentially illegal content, not just images, not just CSAM, and not even strictly illegal. That's what should be opposed. CSAM is a red herring. [0] https://www.macobserver.com/analysis/apple-scans-uploaded-content/ https://www.macobserver.com/analysis/apple-scans-uploaded-co...
- noduerme 5y ago
  The difference is that's scanning in the cloud, on their servers, of things people choose to upload. It's not scanning on the user's private device, and currently they have no way to scan what's on a private device.
  - strogonoff 5y ago
    The phrase is “pre-screening” of uploaded content, which is what is happening. I'm pretty sure this change in ToS was made to enable this CSAM feature.
spacedcowboy 5y ago
There is so much disinformation about this with people not being informed. Apple have put out clear documentation, it would be a good idea to read it before fear-mongering. 1) The list of CSAM hashes is that provided by the relevant US government agency, that is it. 2) The project is not targeted to roll out anywhere other than the USA. 3) The hashes are baked into the OS, there is no capability to update them other than on signed and delivered Apple OS updates. 4) Apple has exactly the same leverage with EvilGov as at any other time. They can refuse to do as asked, and risk being ejected from the country. Their stated claim is that this is what will happen. We will see.
- benmmurphy 5y ago
  Regarding 3: it’s very easy to make a mistake in the protocol that would allow apple to detect hashes outside the CSAM list. Without knowing exactly how their protocol works it’s difficult to know whether it is correct. https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni... For example here is a broken PSI protocol in terms of point 3. I don’t think normally in PSI this is considered broken because the server knows the value so it is part of its private set. Server computes M_s = g . H(m) . S_s where g is a generator of an elliptic curve, H(m) is the neural hash of the image and S_s is the server blinding secret. The client computes M_sc = M_s . S_c where S_c is the client ephemeral secret. This M_sc value is the shared key. The client also computes M_c = g . H(m) . S_c and sends the M_c value to the server. The server can now compute M_cs = M_c . S_s = M_sc since they both used the same H(m) values. This allows the server and client to share a key based on the shared image. However, what happens if the client does it’s step using the ‘wrong’ image. If 3) is to hold it should not be possible for the server to compute the key. Client computes: M_sc = M_s . S_c M_c = g. H(m’) . S_c The clients final key share is: M_sc = g . H(m) . S_c . S_s Now server computes: M_cs = M_c . S_s = g . H(m’) . S_c . S_s The secret shares don’t match. But if the server knows H(m’) it can compute: M_cs’ = M_cs . inv(H(m’)) . H(m) and this secret share will match Normally this client side list in PSI is just used to speed up the protocol so the server does not have to do a crypto operation for every element in its set. It is not a pre-commitment from the server. Also, maybe the way I’m doing it here is just normally broken because it is not robust against low entropy inputs to the hash function. I've also reversed some of apple's non-public crypto that is used in some of it's services and they have made dubious design decisions in the past they have created weird weaknesses. Without knowing exactly what they are doing I would not try and infer properties that might not exist or trust their implementation.
OneLeggedCat 5y ago
> just Winnie the Pooh memes I think it's hilarious that one of the world' great nations would scan for exactly this.

kelnos 5y ago

There's a difference -- often a big one -- between being theoretically able to build some feature into a phone (plus the server-side infra and staffing to support it), and actually having built it. That's the capability angle. However, the difference between having a feature enabled based on the value of a seemingly-unrelated user-controlled setting, versus having that feature enabled all the time... is basically zero. Additionally, extending this feature to encompass other kinds of content is a matter of data entry, not engineering. That's the policy angle. When you don't yet have a capability, it might take a lot of work and commitment to develop it. But on the contrary, policy can be changed with the flip of a switch.