7 ms·

I think you're underestimating the degree to which companies have their own internal logic that has nothing to do with the external reality. That tendency incre

I think you're underestimating the degree to which companies have their own internal logic that has nothing to do with the external reality. That tendency increases the larger a company grows. If you're in a one-room house, you can always see outside. If you're inside a giant office building, mostly you see the building. Did somebody raise the concern of push-back? I'm sure. But the moral questions around CSAM are something that was settled long ago internally. When I was at Twitter fighting abuse, the CSAM stuff was a separate group. My boss called them The Department of Mysteries because we almost never saw them or spoke to them. It was led by a serious person, an ex-FBI agent or something like that. They did what they did and we were all ok with it and grateful for it, because that shit is horrific and we didn't want it on our platform and we didn't want to have to deal with it ourselves. My cousin was a PO for sex offenders, and one of our regular discussion topics at family reunions was how sex offenders were way more technologically savvy than a state parole department. How they really needed more help in making sure offenders weren't reoffending while on parole, while also not forcing them to just not use computers and phones altogether. If even I've heard this, I'm sure that Apple execs have heard it from law enforcement a zillion times. It's also clear Apple put a lot of thought into addressing the privacy concerns for this. Technologically, it's sophisticated, impressive. So I can easily believe the people at Apple said, "Sure, there are reasonable concerns, but we think we have addressed them." And that they're surprised by the level of sustained pushback.

dmitryminkovsky 5y ago
Thank you for this perspective. I've never worked at an organization of this magnitude, so I am definitely lacking some perspective. > It's also clear Apple put a lot of thought into addressing the privacy concerns for this. Technologically, it's sophisticated, impressive. I'm not sure about this. How is a perceptual hash sophisticated and impressive given that it can be abused by governments demanding Apple scan for political content, etc?
- gjsman-1000 5y ago
  Apple says their protection against authoritarian governments is that an image must appear in two or more government CSAM databases to be scanned. So if a corrupt politician adds something to the database, it won't be scanned unless a different government adds the same image. Now, will China follow this? Probably not. But Apple's defense there is that China could have directly ordered them to build this scanning tool at anytime in the last decade anyway. It's not like China has a magical new tool for invading privacy when they could have (and actually have) just ordered tools to be built as desired.
  - wpietri 5y ago
    For sure. It's not clear to me whether Apple's approach is truly sufficient to preserve the privacy and safety of iPhone users that whole governments are out to get. But it definitely looks to me like they thought hard about it and have given it a go.
  - ArtDev 5y ago
    Maybe Apple already built the tool and was like "let's use this tool we already have but just targets specific child porn images". That actually makes them sound more reasonable.. kinda.
  - cge 5y ago
    >Now, will China follow this? Probably not. It's quite possible that countries would do the same thing interested parties on both sides are accused of doing with membership and voting in the International Whaling Commission: find countries that have no interest in the matter either way, and buy their support (or in this case, their addition of images to databases), which can be quite cheap when the matters don't actually affect the country.
  - hu3 5y ago
    Was cooperation between two contries considered?
  - sfifs 5y ago
    And of course it's absolutely unprecedented for two governments to ever collaborate /s "Five Eyes" ring a bell?
- matwood 5y ago
  > I'm not sure about this. How is a perceptual hash sophisticated and impressive given that it can be abused by governments demanding Apple scan for political content, etc? As it's setup today it can't be. The list is the intersection of at least 2 jurisdictions, and is only checked when items are going to iCloud. Of course this could change, but it's important to recognize what it is now vs. speculation about what it could become.
  - deleted 5y ago
    [deleted]
  - deleted 5y ago
    [deleted]
- shadowfiend 5y ago
  A perceptual hash by itself is neither sophisticated nor impressive. Apple could have just applied a client-side perceptual hash and auto-reported you and disabled your device. That is not what they have done, and pretending the solution's sophistication is limited to the sophistication of a percetual hash seems either incurious or disingenuous—I'm assuming the former. Either way, it makes it difficult to engage with your argument.
  - dmitryminkovsky 5y ago
    A searing and possibly legit indictment... what can I say... > incurious Hurts to read this. But maybe you're right: my mind kind of shut off when I read the abstract (client side scanning). I can't say I've looked into the details.
- cvwright 5y ago
  If you're not impressed by the perceptual hash, then you should at least be impressed by their implementation of the private set intersection. That's something that not a lot of companies could pull off. It doesn't make the whole system any less fundamentally flawed, but I think the technical "Wow!" factor certainly helped many people be OK with it. In this view, the shiny new PSI system is a trojan horse for the totally unaccountable and opaque hash set that they've feeding into it.
- Closi 5y ago
  > How is a perceptual hash sophisticated and impressive given that it can be abused by governments demanding Apple scan for political content, etc? It's pretty sophisticated when you look at everything implemented and also consider the infrastructure / review pipelines that are required. See the link below: https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model... It's based on perceptual hashing, but the whole end-to-end system is clearly sophisticated when operating on Apple's scale.
A4ET8a8uTh0 5y ago
This. I will add to this a little more. A big company is its own country. It has its own customs. It has its own ruling class. It has its own language. All those things can make things interesting and sometimes misfire in a form of an idea that is not received that well in the real world.
phire 5y ago
It's worth looking into the privacy concerns that Apple was actually trying to fix here: 1. Photos (and other documents) are currently uploaded to iCloud un-encrypted 2. These photos are already scanned for CSAM after upload 3. Because the photos are not encrypted, at any point, any government can file a court order to release those photos. 4. The court order can require Apply to not notify the user, or the public in general. 5. [Speculation] Such orders might already exist and be somewhat common within Apple Apple wanted to fix this and introduce end-to-end encryption on all photos uploaded to iCloud, but scanning for CSAM was non-negotiable (due to internal or external politics?). They must keep doing it. So they implemented this big mess of a workaround to scan for CSAM before upload and attach a cert with a decryption key only to photos that match so that they could later human verify once a user had enough matches and weed out false positives (which Apple acknowledge will happen) before notifying law enforcement. Because of the direction that Apple came from, and how much effort they put into designing this system to maximize privacy, they saw this solution as a large privacy win over the existing situation. It's not surprising Apple might have been blinded to the privacy concerns of doing AI scanning of photos on user devices, they were looking at it from the wrong angle.
- Unklejoe 5y ago
  > Apple wanted to fix this and introduce end-to-end encryption on all photos uploaded to iCloud Apple has not come out and said this. This may be what people hope is the motivation, but I would think they would be immediately coming out to say this if it were the case, as it could be used as ammunition against some of the criticism they've been getting recently.
  - phire 5y ago
    True. But I'll point out their technical summary[1] explictly talks about attaching decryption keys to positive matches, which you don't need to do if there is no end-to-end encryption. [1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
- deleted 5y ago
  [deleted]
timbaboon 5y ago
Having worked for a large corporate I frequently saw decisions that did not always make sense, with the justification being that the company was an innovator and market leader, and that they wanted to be like Apple and tell the customers what they wanted. Market research and such were not considered useful.
- deleted 5y ago
  [deleted]
- wpietri 5y ago
  I am not sure whether to laugh or cry at that. "We want to tell customers what they want. And it turns out what they want is this half-assed thing that will advance the career of the executive pushing for it." I have my issues with Apple. But they have such a deep tradition of user research and customer focus. When they tell customers what they want, it's often because they have really good data indicating it's what customers do want but don't totally know it yet.

dmitryminkovsky 5y ago

Thank you for this perspective. I've never worked at an organization of this magnitude, so I am definitely lacking some perspective. > It's also clear Apple put a lot of thought into addressing the privacy concerns for this. Technologically, it's sophisticated, impressive. I'm not sure about this. How is a perceptual hash sophisticated and impressive given that it can be abused by governments demanding Apple scan for political content, etc?

gjsman-1000 5y ago
Apple says their protection against authoritarian governments is that an image must appear in two or more government CSAM databases to be scanned. So if a corrupt politician adds something to the database, it won't be scanned unless a different government adds the same image. Now, will China follow this? Probably not. But Apple's defense there is that China could have directly ordered them to build this scanning tool at anytime in the last decade anyway. It's not like China has a magical new tool for invading privacy when they could have (and actually have) just ordered tools to be built as desired.
- wpietri 5y ago
  For sure. It's not clear to me whether Apple's approach is truly sufficient to preserve the privacy and safety of iPhone users that whole governments are out to get. But it definitely looks to me like they thought hard about it and have given it a go.
- ArtDev 5y ago
  Maybe Apple already built the tool and was like "let's use this tool we already have but just targets specific child porn images". That actually makes them sound more reasonable.. kinda.
- cge 5y ago
  >Now, will China follow this? Probably not. It's quite possible that countries would do the same thing interested parties on both sides are accused of doing with membership and voting in the International Whaling Commission: find countries that have no interest in the matter either way, and buy their support (or in this case, their addition of images to databases), which can be quite cheap when the matters don't actually affect the country.
- hu3 5y ago
  Was cooperation between two contries considered?
- sfifs 5y ago
  And of course it's absolutely unprecedented for two governments to ever collaborate /s "Five Eyes" ring a bell?
matwood 5y ago
> I'm not sure about this. How is a perceptual hash sophisticated and impressive given that it can be abused by governments demanding Apple scan for political content, etc? As it's setup today it can't be. The list is the intersection of at least 2 jurisdictions, and is only checked when items are going to iCloud. Of course this could change, but it's important to recognize what it is now vs. speculation about what it could become.
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
shadowfiend 5y ago
A perceptual hash by itself is neither sophisticated nor impressive. Apple could have just applied a client-side perceptual hash and auto-reported you and disabled your device. That is not what they have done, and pretending the solution's sophistication is limited to the sophistication of a percetual hash seems either incurious or disingenuous—I'm assuming the former. Either way, it makes it difficult to engage with your argument.
- dmitryminkovsky 5y ago
  A searing and possibly legit indictment... what can I say... > incurious Hurts to read this. But maybe you're right: my mind kind of shut off when I read the abstract (client side scanning). I can't say I've looked into the details.
cvwright 5y ago
If you're not impressed by the perceptual hash, then you should at least be impressed by their implementation of the private set intersection. That's something that not a lot of companies could pull off. It doesn't make the whole system any less fundamentally flawed, but I think the technical "Wow!" factor certainly helped many people be OK with it. In this view, the shiny new PSI system is a trojan horse for the totally unaccountable and opaque hash set that they've feeding into it.
Closi 5y ago
> How is a perceptual hash sophisticated and impressive given that it can be abused by governments demanding Apple scan for political content, etc? It's pretty sophisticated when you look at everything implemented and also consider the infrastructure / review pipelines that are required. See the link below: https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model... It's based on perceptual hashing, but the whole end-to-end system is clearly sophisticated when operating on Apple's scale.

A4ET8a8uTh0 5y ago

This. I will add to this a little more. A big company is its own country. It has its own customs. It has its own ruling class. It has its own language. All those things can make things interesting and sometimes misfire in a form of an idea that is not received that well in the real world.

phire 5y ago

It's worth looking into the privacy concerns that Apple was actually trying to fix here: 1. Photos (and other documents) are currently uploaded to iCloud un-encrypted 2. These photos are already scanned for CSAM after upload 3. Because the photos are not encrypted, at any point, any government can file a court order to release those photos. 4. The court order can require Apply to not notify the user, or the public in general. 5. [Speculation] Such orders might already exist and be somewhat common within Apple Apple wanted to fix this and introduce end-to-end encryption on all photos uploaded to iCloud, but scanning for CSAM was non-negotiable (due to internal or external politics?). They must keep doing it. So they implemented this big mess of a workaround to scan for CSAM before upload and attach a cert with a decryption key only to photos that match so that they could later human verify once a user had enough matches and weed out false positives (which Apple acknowledge will happen) before notifying law enforcement. Because of the direction that Apple came from, and how much effort they put into designing this system to maximize privacy, they saw this solution as a large privacy win over the existing situation. It's not surprising Apple might have been blinded to the privacy concerns of doing AI scanning of photos on user devices, they were looking at it from the wrong angle.

Unklejoe 5y ago
> Apple wanted to fix this and introduce end-to-end encryption on all photos uploaded to iCloud Apple has not come out and said this. This may be what people hope is the motivation, but I would think they would be immediately coming out to say this if it were the case, as it could be used as ammunition against some of the criticism they've been getting recently.
- phire 5y ago
  True. But I'll point out their technical summary[1] explictly talks about attaching decryption keys to positive matches, which you don't need to do if there is no end-to-end encryption. [1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
deleted 5y ago
[deleted]

timbaboon 5y ago

Having worked for a large corporate I frequently saw decisions that did not always make sense, with the justification being that the company was an innovator and market leader, and that they wanted to be like Apple and tell the customers what they wanted. Market research and such were not considered useful.

deleted 5y ago
[deleted]
wpietri 5y ago
I am not sure whether to laugh or cry at that. "We want to tell customers what they want. And it turns out what they want is this half-assed thing that will advance the career of the executive pushing for it." I have my issues with Apple. But they have such a deep tradition of user research and customer focus. When they tell customers what they want, it's often because they have really good data indicating it's what customers do want but don't totally know it yet.