6 ms·

It is possible to detect CSAM in end-to-end encrypted messaging by doing it on-device on the client side before it is encrypted and/or on the receiving client s

It is possible to detect CSAM in end-to-end encrypted messaging by doing it on-device on the client side before it is encrypted and/or on the receiving client side after it is decrypted. iMessage already does this in the latest release. Most smart phones have AI-enabled chips that would be able to run images/videos against classification algorithms. The tricky part would be enforcing that users use clients which do this, so the task becomes regulating allowed messaging clients, which might be impossible. That being said, one could probably knock out 80% of the problem by legally forcing the hand of all the major platforms to do client-side scanning. At that point, only the truly dedicated would move their comms to another platform. Mobile users are very susceptible to nudges.

andersa 3y ago
If the app is processing data in a way you do not want before sending it, especially one that will no doubt be constantly argued to scan for more and more things they find objectionable, then it defeats whole point of end to end encryption, and you might as well just not do it at all.
aaomidi 3y ago
Please don’t AI detect CSAM. That is a disaster waiting to happen.
- vacuity 3y ago
  It works great! Until it doesn't, and then someone gets put into jail and the court doesn't bother reviewing the AI's verdict because how could it go wrong?
  - Freak_NL 3y ago
    It's not that you would end up in jail (although granted, that level of fuck-uppery is possible), it's just that anyone flagged would go through hell with real-world consequences before everything is cleared up.
  - figglestar 3y ago
    Even if no one charges you it still creates a mess for the accused. We already saw how google handled a verified false positive scan that the police determined was innocuous.
- unaindz 3y ago
  And do not hash detect it either. If you mass scan eventually you are gonna hit a collision .
- cwoolfe 3y ago
  [flagged]
majke 3y ago
I think the problem is different. This is not about a technical possibility. This is about who will submit and manage the CSAM fingerprints. With the technology it will be trivial for oppressive governments to query which citizens have a copy of a particular image on their phone. Consider a scenario: - you use whatsapp, end-to-end encrypted by an USA company - someone, perhaps a random stranger, sends you an image in which there is a criticism to a ruling party in Dabujistan - your whatsapp has turned on the feature to save images on your phone - you happen to transfer via Dabujistan With CSAM tech, and fingerprints managed by the governments, you might be subject to jail.
- JCharante 3y ago
  The unfortunate part about a centralized CSAM list is that now you need a committee to review every CSAM addition (not just the hashes but the source material to confirm it's CSAM and not political) and the committee has to have enough people to make sure the committee isn't all in kahoots
- mordae 3y ago
  > This is about who will submit and manage the CSAM fingerprints. Not it's not. It's about "Why the fuck should I be under constant surveillance?" Our computers are practically brain extensions at this point. I don't want anyone wiretapping my thoughts or coming close to it by wiretapping my private conversations with my family. Fuck off.
4bpp 3y ago
Do you imagine that a significant proportion of CSAM is shared by users who are so casual about it that they would be "very susceptible to nudges"?
- cool_dude85 3y ago
  I suspect that a large portion of CSAM is nude selfies that might be susceptible to "nudges" in the behavioral economics sense.
  - salawat 3y ago
    To hell with your "nudges". It It's a cutesy way of saying "flex of State power", and everyone effing knows it. No. It's not appropriate.
  - 4bpp 3y ago
    This may well be true according to the legal definition, though a system that matches against a fixed list of hashes, perceptual or otherwise, rather than working on magical AI, would not catch those. However, if this is all that the measure would be effective against then this is rather at odds with the proponents' narrative which makes this out to be about pictures produced and shared by adults with criminal intent.
- bloopernova 3y ago
  I think that it's only going to get faster, easier, and more accurate to generate sexualized pictures and videos of someone younger than 18. Whether you or I consider that "real" CSAM, is moot. What matters is that more and more girls are going to be deepfaked into content they didn't consent to (and in fact can't consent to due to their age. The proliferation of that content will lead to new laws driven by parents and politicians eager to "think of the children". However impractical, I could see image generators being treated like drugs or firearms and being highly restricted or banned.
- seanp2k2 3y ago
  I’m not sure if it’s a happy or sad thing how unaware most people, even the technically-savvy crowd, are about the extent of human trafficking and slavery in 2023. Is it proof that civilization is working how so many don’t need to concern themselves with hunting and gathering, fighting wars, fighting crime, or fighting in general to survive, and are free to spend their time on recreation and the pursuit of happiness instead of survival?
AJ007 3y ago
A greater point here is that end to end encryption in this context is almost entirely a lie.
stephen_g 3y ago
iMessage does not do this in any release, that plan was abandoned due to legitimate privacy concerns. The feature that is implemented is a parental control that can be enabled on the devices of minors, that blurs any detected nude images with a warning 'do you want to see this'. It's not possible to detect CSAM without breaking the security model of the system for everyone, if the system can make any kind of report of detected content to an outside entity. If the system scans messages client side and then sends anything to do with the message to authorities, that is breaking the end to end encryption. This is either done by 'AI', which means huge numbers of false positives that are actually legal, private and probably intimate get sent to authorities (hence breaking the end-to-end security model for that legal content), or by a secret, un-auditable collection of hashes - which are still fuzzy matched, so there would definitely be false positives (again, breaking the security model for legal content), but potentially also Governments could start to add other, legal material to the database, such as certain political material to detect dissidents. Since it's un-auditable, there is no way to know whether it's actually just illegal material in the database. So whether it's AI or PhotoDNA or whatever, there's no way to do it that doesn't break the security model and cause more danger to innocent people, including destroying the privacy of children as well as adult, law-abiding citizens.
- cwoolfe 3y ago
  [flagged]
- sneak 3y ago
  It is worth noting that due to endpoint key escrow on by default in the non-e2ee iCloud Backup, iMessage is already presently broken and not e2ee as Apple has endpoint keys available to them to decrypt almost all iMessages that transit the service (Messages in iCloud) in realtime. Even if you enable iCloud backup E2EE, or disable iCloud entirely, iMessage is still not E2EE as your conversation partners are escrowing (to Apple) the keys from the other endpoint.
Freak_NL 3y ago
Why should my phone be acting like I'm a criminal? That's like searching the bags of everyone who exits the supermarket, or police doing door-to-door house searches "just in case". What's next? Verify every photo I take with my phone?
- buildbot 3y ago
  Yeah that's what this person wants, to scan every photo you take...ick.
- seanp2k2 3y ago
  Or make everyone go through metal detectors and millimeter wave scanners at the airport, take their shoes off, treat any liquid over whatever the small limit is like it’s bomb-making material (but still let that person fly; checkmate, would-be bomber!)…it’s all security theater. Notice how they don’t release their testing rates on how many of the test weapons and threats make it through TSA security, because then it would be too easy to point to that and rightfully assert that they utterly fail at their mission while wasting untold lifetimes worth of human time and make travel that much more miserable. https://www.schneier.com/blog/archives/2015/06/reassessing_air.html https://www.schneier.com/blog/archives/2015/06/reassessing_a... (2015) is a good read on that if you don’t already know. My point is that when politicians do something “to save the children”, it’s rarely about actually helping anyone but themselves.
razakel 3y ago
How do you plan to do that without having a database of known CSAM on the device?
didntcheck 3y ago
What classification algorithm can detect CSAM, given that humans cannot reliably determine someone's age? [1] [2] [1] https://www.eldia.com/nota/2010-4-23-actriz-porno-salva-a-fan-de-ir-a-la-carcel https://www.eldia.com/nota/2010-4-23-actriz-porno-salva-a-fa... [ES] [2] anyone who still gets ID'd for alcohol despite being well over 18, and often over 25 too
- seanp2k2 3y ago
  Yeah, no one who’s actually had to work with this on a technical level would seriously suggest our current GPT-4 level of AI for detection. It’s hash-based with some sugar on top.

andersa 3y ago

If the app is processing data in a way you do not want before sending it, especially one that will no doubt be constantly argued to scan for more and more things they find objectionable, then it defeats whole point of end to end encryption, and you might as well just not do it at all.

aaomidi 3y ago

Please don’t AI detect CSAM. That is a disaster waiting to happen.

vacuity 3y ago
It works great! Until it doesn't, and then someone gets put into jail and the court doesn't bother reviewing the AI's verdict because how could it go wrong?
- Freak_NL 3y ago
  It's not that you would end up in jail (although granted, that level of fuck-uppery is possible), it's just that anyone flagged would go through hell with real-world consequences before everything is cleared up.
- figglestar 3y ago
  Even if no one charges you it still creates a mess for the accused. We already saw how google handled a verified false positive scan that the police determined was innocuous.
unaindz 3y ago
And do not hash detect it either. If you mass scan eventually you are gonna hit a collision .
cwoolfe 3y ago
[flagged]

majke 3y ago

I think the problem is different. This is not about a technical possibility. This is about who will submit and manage the CSAM fingerprints. With the technology it will be trivial for oppressive governments to query which citizens have a copy of a particular image on their phone. Consider a scenario: - you use whatsapp, end-to-end encrypted by an USA company - someone, perhaps a random stranger, sends you an image in which there is a criticism to a ruling party in Dabujistan - your whatsapp has turned on the feature to save images on your phone - you happen to transfer via Dabujistan With CSAM tech, and fingerprints managed by the governments, you might be subject to jail.

JCharante 3y ago
The unfortunate part about a centralized CSAM list is that now you need a committee to review every CSAM addition (not just the hashes but the source material to confirm it's CSAM and not political) and the committee has to have enough people to make sure the committee isn't all in kahoots
mordae 3y ago
> This is about who will submit and manage the CSAM fingerprints. Not it's not. It's about "Why the fuck should I be under constant surveillance?" Our computers are practically brain extensions at this point. I don't want anyone wiretapping my thoughts or coming close to it by wiretapping my private conversations with my family. Fuck off.

4bpp 3y ago

Do you imagine that a significant proportion of CSAM is shared by users who are so casual about it that they would be "very susceptible to nudges"?

cool_dude85 3y ago
I suspect that a large portion of CSAM is nude selfies that might be susceptible to "nudges" in the behavioral economics sense.
- salawat 3y ago
  To hell with your "nudges". It It's a cutesy way of saying "flex of State power", and everyone effing knows it. No. It's not appropriate.
- 4bpp 3y ago
  This may well be true according to the legal definition, though a system that matches against a fixed list of hashes, perceptual or otherwise, rather than working on magical AI, would not catch those. However, if this is all that the measure would be effective against then this is rather at odds with the proponents' narrative which makes this out to be about pictures produced and shared by adults with criminal intent.
bloopernova 3y ago
I think that it's only going to get faster, easier, and more accurate to generate sexualized pictures and videos of someone younger than 18. Whether you or I consider that "real" CSAM, is moot. What matters is that more and more girls are going to be deepfaked into content they didn't consent to (and in fact can't consent to due to their age. The proliferation of that content will lead to new laws driven by parents and politicians eager to "think of the children". However impractical, I could see image generators being treated like drugs or firearms and being highly restricted or banned.
seanp2k2 3y ago
I’m not sure if it’s a happy or sad thing how unaware most people, even the technically-savvy crowd, are about the extent of human trafficking and slavery in 2023. Is it proof that civilization is working how so many don’t need to concern themselves with hunting and gathering, fighting wars, fighting crime, or fighting in general to survive, and are free to spend their time on recreation and the pursuit of happiness instead of survival?

AJ007 3y ago

A greater point here is that end to end encryption in this context is almost entirely a lie.

stephen_g 3y ago

iMessage does not do this in any release, that plan was abandoned due to legitimate privacy concerns. The feature that is implemented is a parental control that can be enabled on the devices of minors, that blurs any detected nude images with a warning 'do you want to see this'. It's not possible to detect CSAM without breaking the security model of the system for everyone, if the system can make any kind of report of detected content to an outside entity. If the system scans messages client side and then sends anything to do with the message to authorities, that is breaking the end to end encryption. This is either done by 'AI', which means huge numbers of false positives that are actually legal, private and probably intimate get sent to authorities (hence breaking the end-to-end security model for that legal content), or by a secret, un-auditable collection of hashes - which are still fuzzy matched, so there would definitely be false positives (again, breaking the security model for legal content), but potentially also Governments could start to add other, legal material to the database, such as certain political material to detect dissidents. Since it's un-auditable, there is no way to know whether it's actually just illegal material in the database. So whether it's AI or PhotoDNA or whatever, there's no way to do it that doesn't break the security model and cause more danger to innocent people, including destroying the privacy of children as well as adult, law-abiding citizens.

cwoolfe 3y ago
[flagged]
sneak 3y ago
It is worth noting that due to endpoint key escrow on by default in the non-e2ee iCloud Backup, iMessage is already presently broken and not e2ee as Apple has endpoint keys available to them to decrypt almost all iMessages that transit the service (Messages in iCloud) in realtime. Even if you enable iCloud backup E2EE, or disable iCloud entirely, iMessage is still not E2EE as your conversation partners are escrowing (to Apple) the keys from the other endpoint.

Freak_NL 3y ago

Why should my phone be acting like I'm a criminal? That's like searching the bags of everyone who exits the supermarket, or police doing door-to-door house searches "just in case". What's next? Verify every photo I take with my phone?

buildbot 3y ago
Yeah that's what this person wants, to scan every photo you take...ick.
seanp2k2 3y ago
Or make everyone go through metal detectors and millimeter wave scanners at the airport, take their shoes off, treat any liquid over whatever the small limit is like it’s bomb-making material (but still let that person fly; checkmate, would-be bomber!)…it’s all security theater. Notice how they don’t release their testing rates on how many of the test weapons and threats make it through TSA security, because then it would be too easy to point to that and rightfully assert that they utterly fail at their mission while wasting untold lifetimes worth of human time and make travel that much more miserable. https://www.schneier.com/blog/archives/2015/06/reassessing_air.html https://www.schneier.com/blog/archives/2015/06/reassessing_a... (2015) is a good read on that if you don’t already know. My point is that when politicians do something “to save the children”, it’s rarely about actually helping anyone but themselves.

razakel 3y ago

How do you plan to do that without having a database of known CSAM on the device?

didntcheck 3y ago

What classification algorithm can detect CSAM, given that humans cannot reliably determine someone's age? [1] [2] [1] https://www.eldia.com/nota/2010-4-23-actriz-porno-salva-a-fan-de-ir-a-la-carcel https://www.eldia.com/nota/2010-4-23-actriz-porno-salva-a-fa... [ES] [2] anyone who still gets ID'd for alcohol despite being well over 18, and often over 25 too

seanp2k2 3y ago
Yeah, no one who’s actually had to work with this on a technical level would seriously suggest our current GPT-4 level of AI for detection. It’s hash-based with some sugar on top.