2 ms·
Why does the CCP even need to talk to Apple? They have a database of CSAM, and can modify an image in this set to collide with a special image they're looking
by jrockway 5y ago
Why does the CCP even need to talk to Apple? They have a database of CSAM, and can modify an image in this set to collide with a special image they're looking for on people's phones. They then share their new modified cache of CSAM with other countries ("hey, China is helping us! that's great!") and it gets added to Apple's database for the next iOS release, because it looks to humans like CSAM. Only the CCP knows that it collides with something special they're looking for.
Now that we know that collisions are not just easy -- but happen by themselves with no human input (as evidenced by the Imagenet collisions), we know this system can't work. Apple has two remaining safeguards: a second set of hashes (they say), and human reviewers.
The human reviewers are likely trained to prefer false-positives when unsure, and so while a thorough human review would clearly indicate "not CSAM" for the images the malicious collisions match, it doesn't feel like much of a safeguard to me. (Remember the leaked memo from the US's organization -- they called our objections "screeching voices". I'm sure the actual reviewers think similarly.)
I assume the people reviewing CSAM for the CCP will be in China, so they can be in on the whole scheme. (In the US, we have slightly better checks and balances. Eventually the image will be in front of a court and a jury of your peers, and it's not illegal to have photos that embarrass the CCP, so you'll personally be fine modulo the stress of a criminal investigation. But that dissident in China, probably not going to get a fair trial -- despite the image not being CSAM, it's still illegal. And Apple handed it directly to the government for you.)
I don't know, I just find this thing exceedingly worrying. When faced with a government-level adversary, this doesn't sound like a very good system. I think if we're okay with this, we might as well mandate CCTV in everyone's home, so we can catch real child abusers in the act.