14 ms·

It's scary, because CSAM (child sexual abuse material) is very, very broad, and quite vague in its definition¹. My three year old son stayed with my parents for

It's scary, because CSAM (child sexual abuse material) is very, very broad, and quite vague in its definition¹. My three year old son stayed with my parents for a week, and he was being taught to swim in their swimming pool. Of course, I get sent pictures and clips of his progress by a proud grandma, and of course, him being a toddler, he felt swimming trunks to be completely optional (and of course, they are at his age in a private pool). That's not CSAM right? But who knows how the AI will mark those happy pics? Some quirk in its programming and training leading to them being pushed way up whatever ranking is used? It's a nude child after all, and that is most likely one of the few things such an algorithm can detect quite reliably. The automated filter won't care about the context though, and if any of the recent failures of algorithms ruining peoples lives are anything to go by (the Dutch Toeslagenaffaire comes to mind), being flagged by the CSAM filter is a high risk event — even if you can clear your name later by human intervention — because now you are on a list. 1: People tend to conjure up horrific images of underage children getting raped by adults when they hear 'CSAM' or 'child pornography', but when you read up on the legal definitions used it becomes very vague, quite fast. A seventeen year old boy sending a dick pic to his 18 year old lover is producing child pornography and can in many jurisdictions be sentenced as such, and nudism of any minor can be seen as CSAM if the pose they strike can be construed as 'erotic' — yet there is no clear definition of what this means, falling squarely into “I'll know it when I see it” territory.

vidarh 4y ago
I almost think these efforts will need to succeed in become applied in a way too authoritarian way before there's a sufficient backlash. Unfortunately that means there's likely to be a lot of collateral damage.
- seanw444 4y ago
  Seems that's the only way people end up caring to fix things these days.
stef25 4y ago
You're probably not the first person to ask the question of "but what about parents taking pictures of their naked kids in a pool", I'm sure they plan on having some way of dealing with it if anything to lighten their workload. That doesn't mean it will be perfect but no matter how jaded we all are, nobody wants to start falsely accusing innocent people.
- BlargMcLarg 4y ago
  >nobody wants to start falsely accusing innocent people. If anything, the 21st century has proven quite a few people are more than willing to do so when they deem the lack of true positives to be the greater evil. Or even just doing it for their own benefit.
  - stef25 4y ago
    Can you give some EU specific examples that can be considered representative and not huge outliers? Shit happens everywhere but where am I in Europe if anything there's a lack of law enforcement in the sense that prison terms always seem to be considered too low, people with sentences under X months don't actually go to jail because of overcrowding, police is constantly frustrated that people they pick off the street are immediately released again by judges etc.
    - BlargMcLarg 4y ago
      The starting comment gave you an example to start off with. The Netherlands, Toeslagenaffaire. You can't go telling "yeah we really care about innocent people" when you let an algorithm wreak havoc producing false positives and then take ages to fix it while the families are still dealing with the aftermath. That's the opposite, caring for true positives despite the potential false positives produced. I'd also ask you to take a few trips down some ideologies. If you believe "EU is so nice it can do no wrong", you'll probably be cured of it soon when you realize extremists are spread all around the world, and have no qualms exercising their sense of justice even if it hurts innocent people.
zosima 4y ago
Don't worry, as this is not about child sex abuse anyway. The legislators know, from experience, that it's hard to argue against measures to protect children from predators, and so they are free to launch controversial and unacceptable measures. Once in place, it will be extended to other purposes. The same as always.
- chinathrow 4y ago
  > The same as always. Do you have a guess about who is behind that lasting global push against private communications?
  - deleted 4y ago
    [deleted]
  - K0balt 4y ago
    I’d love to know, specifically
  - emteycz 4y ago
    Why does it have to be a somebody? It's almost every politician since the invention of privacy. It's not a cabal, they just have common goals. Edit: can't reply. The endgame is control over the population - taxing them, imprisoning them, etc. Why? I don't know, some people just like to be in (absolute) power.
    - chinathrow 4y ago
      And what are these common goals then? I really don't see the endgame here.
    - eru 4y ago
      It's the government version of instrumental convergence. https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence
  - hulitu 4y ago
Cthulhu_ 4y ago
I mean that's cute and all, but why are your parents taking pictures of naked children and publishing them online? That's making assumptions about the channel used to send them and the phone itself being secure. Better to not risk it - not because of this legislation, but because of anyone else that might be listening in.
- Freak_NL 4y ago
  Sending pictures to me via WhatsApp is not 'publishing things online'. It is meant as a private communication channel. The current proposal wants to listen in on those channels. And obviously, I personally, can instruct anyone handling my child to not do that when such laws come into effect, but that doesn't change the fact that for many people such actions are completely normal (as it should be).
- nahnahno 4y ago
  This is how free speech ends. If the parent doesn’t think the image is inappropriate or would damage the child, they should be free to share what they want. Full stop.
- tokai 4y ago
  yeah OP why ain't you letting fear and unjust laws control your whole life? \s
- Dave3of5 4y ago
  The message here is a private message. The title of this post contains the words "private communication" same for the title in the actual post. Have you not actually read anything about the post ?
BlargMcLarg 4y ago
It's funny because in this regard, The Netherlands is actually trying to do better and change the laws to allow mutually consenting teens to do this stuff (still tentative) and punish blackmail, prostitution (edit: of minors), minor-adult sexual interaction etc. Our general mentality also seems to be a lot more lax and supportive of kids being victims of their own mistakes rather than going "yeah, that's illegal, let's throw a fine on top". Almost as if we acknowledge teens are going to experiment.
- vanderZwan 4y ago
  It probably helps that we were complaining about and even making fun of[0] our government for using child pornography as an excuse for pushing through draconian populist laws a decade ago already. We may just happen to have a little head-start here basically. Edit: for the non-Dutch, the link goes to an article by the Dutch equivalent of The Onion and the headline says "Opstelten [our minister of Security and Justice at the time] says new regulations are needed or else child pornography". [0] https://speld.nl/2012/10/24/opstelten-%E2%80%98nieuwe-maatregelen-nodig-want-anders-kinderporno%E2%80%99/ https://speld.nl/2012/10/24/opstelten-%E2%80%98nieuwe-maatre...
  - tluyben2 4y ago
    You need awareness that things most people don’t understand fully are absurd. Humour can be a tool for that.
  - pc86 4y ago
    The Netherlands seems great as an outsider. I've been to Amsterdam, which was too touristy for me, but still beautiful. And we took a train to Utrecht and the surrounding areas which was great. Even struck up a conversation with a group of (local) software developers on the train. Even with the lockdown protest issues it seems like it should be on the shortlist if you want to move to the EU as an American.
    - eru 4y ago
      Keep in mind that comp for Dutch software developers is probably an order of magnitude lower than what you can get in the US.
marsRoverDev 4y ago
My understanding of this technology was it was usually a hash or fingerprint of some form of the material, to be compared against a database of hashes of known-bad material. I doubt EU countries have the infrastructure or budget to run models across everything, no?
- GordonS 4y ago
  An article about this was posted yesterday, and the EU wants to take it further that detecting known CSAM content, using AI to detect new content too. Of course, it's all a pretence for getting access to read all of your data. It might be a language thing, but the wording of the report was such that they were almost stating this explicitly. Honestly, what.the.hell are these clowns thinking?!
  - matheusmoreira 4y ago
    The article says they already have technology that detects "grooming", whatever that means. That's seriously frightening, who knows what else they can detect? Maybe a few years from now we'll be reading articles about people getting arrested for expressing prohibited thoughts, subversive ideas, political opposition in what they thought was a private communications channel.
- matt123456789 4y ago
  That’s the implementation used by Apple for client-side scanning of iCloud uploads. NeuralHash or something. The EU may (and likely will) build an unrelated system, although it may share ideas with prior work.
- Freak_NL 4y ago
  Even with a database of hashes the risk remains. If someone gets off on nude children, then there is nothing stopping them from collecting innocent photographs from social media¹, reposting these on seedy or oniony websites, and have the hash ending up in the huge black box of CSAM hashes by an automated scraper. I am not under the impression that such a database is curated too closely. 1: This is not something I'm personally at risk of, because my parents are tech savvy enough not to post nude pics of a toddler anywhere but in a private chat with his parents (i.e., me and my wife).
londons_explore 4y ago
> I get sent pictures and clips of his progress Those pictures will be seen by you and grandma, but also maybe by employees of whoever you use to backup the pictures, whatever security services have a need to view the images, whichever police forensics department may need to look through your or grandmas phone if it were to be confiscated, etc. Those employees need a good work environment, and can't be expected to look at your naked three year old.
- Freak_NL 4y ago
  Having an actual person see those pics is totally fine. No one in their right mind would see anything but normal child raising there, and the judicial process has checks to prevent abuse by any one actor. It's algorithms I'm concerned about.
- kevincox 4y ago
  I'm sorry. The chance of someone needing to look at these pictures should be near 0. The backup company doesn't need to look at them. They should actually be end-to-end encrypted so that only me and grandma can see. In exceptional cases the police may need to see them. But even in that case there is nothing wrong with these pictures. Pictures of gore and literal shits are legal and while I agree that it can make this job very taxing. But a child swimming naked is not the problem here.
- dvdkon 4y ago
  We seem to have very different priorities. You are concerned about the welfare of employees of privacy-violating organisations. I'm concerned about citizens' privacy and say those organisations shouldn't exist.
peoplefromibiza 4y ago
> The automated filter won't care about the context though That's why we have a justice system. Specifically not a common law based one, which is kinds scary per se. People are innocent until proven guilty and the burden of proof's on the prosecution. We are talking about EU here, not Sudan.
- pc86 4y ago
  1. What makes you think Sudan (or NK, or Russia, or China) won't get their hands on this technology? 2. What makes you think every government in the entire EU will always have the people's best interests at heart? 3. What makes you think #2 is true even today? It seems like this technology could be very easy to abuse, to great public detriment. Is the juice worth the squeeze?
  - peoplefromibiza 4y ago
    > 1. What makes you think Sudan (or NK, or Russia, or China) won't get their hands on this technology? their problem! they have more pressing issues already! and some of them have already implemented worse ways of surveilling their citizens. In this regards, even USA is far worse than EU! > 2. What makes you think every government in the entire EU will always have the people's best interests at heart? I don't, that's why we constantly check on them and change them if we don't like them > 3. What makes you think #2 is true even today? Because I live in EU and everywhere else I look the situation is far worse. > It seems like this technology could be very easy to abuse, to great public detriment. Is the juice worth the squeeze? Like any technology! Funny that guns are abused daily in US but the biggest preoccupation of HN is potential abuse of CSAM filters in EU... Which would also be illegal! If the filter has to be used for CSAM content, it can't be used for anything else.
    - pc86 4y ago
      > > 3. What makes you think #2 is true even today? > Because I live in EU and everywhere else I look the situation is far worse. "It's worse other places" doesn't answer the question. The EU's government isn't perfect and I don't see any safeguards in this proposal. It's ripe for abuse. > Funny that guns are abused daily in US Whataboutism nonsense, nobody is talking about guns except you. > If the filter has to be used for CSAM content, it can't be used for anything else. Utterly fanciful. Once this exists and is in place, it can and will be expanded to other things. Eventually a government you don't like will be elected in your country, and they'll be able to expand this. This tool in the hands of an authoritarian government is as dangerous as any firearm because it would allow them to track dissent.
kevincox 4y ago
The biggest problem is that these images are CSAM. Or at least a bit-for-bit identical image is in a different context. If it is family members are sharing an image of their child having fun or even sending an image to your doctor then there is no problem here. But those exact images in a different context could be a problem. Of course this is why they have human moderation before conviction. But I absolutely don't want my private images being reviewed because they may be CSAM. This is basically assuming that I have likely done something wrong and intruding my privacy because of that.
paganel 4y ago
At this point I would never send nor have nude photos with my kids (whom I don't have, but hypothetically speaking) on my phone or on any other electronic device, doing otherwise is just asking for trouble. Maybe it's not the perfect situation for parents who want to share said photos with the kid's grandparents thousands of kilometres away but it is what it is.
- pfortuny 4y ago
  Oh but OP is talking about his parents (the granparents) sending the photos. And you already have received them!
oauea 4y ago
> and nudism of any minor can be seen as CSAM if the pose they strike can be construed as 'erotic' So we're now expected to teach our kids what erotic poses are so they know not to accidentally do that during family pics, lest daddy or mommy wind up in jail? The future is fucked up.
- kingcharles 4y ago
  If you look at the history of litigation of nudism magazines in the 80s and 90s it is exactly that. Some magazines were allowed in, some were banned and the owners imprisoned because a judge said that some of the photos weren't candid or casual enough.
kingcharles 4y ago
I personally know someone who spent 13 years in prison because at 18 his 16 year old girlfriend sent him a nude selfie. Her Mom was in her phone and saw that she had sent the photos, so she called the police on her daughter's boyfriend. They arrested him, seized his phone and found the photo.
babyshake 4y ago
It would actually be great if iPhone/Android detected and offered to blur genitals with a click. As creepy as that may seem those of us with young children may only incidentally capture material that could be mistaken for CSAM and would be fine blurring out those details.

vidarh 4y ago

I almost think these efforts will need to succeed in become applied in a way too authoritarian way before there's a sufficient backlash. Unfortunately that means there's likely to be a lot of collateral damage.

seanw444 4y ago
Seems that's the only way people end up caring to fix things these days.

stef25 4y ago

You're probably not the first person to ask the question of "but what about parents taking pictures of their naked kids in a pool", I'm sure they plan on having some way of dealing with it if anything to lighten their workload. That doesn't mean it will be perfect but no matter how jaded we all are, nobody wants to start falsely accusing innocent people.

BlargMcLarg 4y ago
>nobody wants to start falsely accusing innocent people. If anything, the 21st century has proven quite a few people are more than willing to do so when they deem the lack of true positives to be the greater evil. Or even just doing it for their own benefit.
- stef25 4y ago
  Can you give some EU specific examples that can be considered representative and not huge outliers? Shit happens everywhere but where am I in Europe if anything there's a lack of law enforcement in the sense that prison terms always seem to be considered too low, people with sentences under X months don't actually go to jail because of overcrowding, police is constantly frustrated that people they pick off the street are immediately released again by judges etc.
  - BlargMcLarg 4y ago
    The starting comment gave you an example to start off with. The Netherlands, Toeslagenaffaire. You can't go telling "yeah we really care about innocent people" when you let an algorithm wreak havoc producing false positives and then take ages to fix it while the families are still dealing with the aftermath. That's the opposite, caring for true positives despite the potential false positives produced. I'd also ask you to take a few trips down some ideologies. If you believe "EU is so nice it can do no wrong", you'll probably be cured of it soon when you realize extremists are spread all around the world, and have no qualms exercising their sense of justice even if it hurts innocent people.

zosima 4y ago

Don't worry, as this is not about child sex abuse anyway. The legislators know, from experience, that it's hard to argue against measures to protect children from predators, and so they are free to launch controversial and unacceptable measures. Once in place, it will be extended to other purposes. The same as always.

chinathrow 4y ago
> The same as always. Do you have a guess about who is behind that lasting global push against private communications?
- deleted 4y ago
  [deleted]
- K0balt 4y ago
  I’d love to know, specifically
- emteycz 4y ago
  Why does it have to be a somebody? It's almost every politician since the invention of privacy. It's not a cabal, they just have common goals. Edit: can't reply. The endgame is control over the population - taxing them, imprisoning them, etc. Why? I don't know, some people just like to be in (absolute) power.
  - chinathrow 4y ago
    And what are these common goals then? I really don't see the endgame here.
  - eru 4y ago
    It's the government version of instrumental convergence. https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence
- hulitu 4y ago

Cthulhu_ 4y ago

I mean that's cute and all, but why are your parents taking pictures of naked children and publishing them online? That's making assumptions about the channel used to send them and the phone itself being secure. Better to not risk it - not because of this legislation, but because of anyone else that might be listening in.

Freak_NL 4y ago
Sending pictures to me via WhatsApp is not 'publishing things online'. It is meant as a private communication channel. The current proposal wants to listen in on those channels. And obviously, I personally, can instruct anyone handling my child to not do that when such laws come into effect, but that doesn't change the fact that for many people such actions are completely normal (as it should be).
nahnahno 4y ago
This is how free speech ends. If the parent doesn’t think the image is inappropriate or would damage the child, they should be free to share what they want. Full stop.
tokai 4y ago
yeah OP why ain't you letting fear and unjust laws control your whole life? \s
Dave3of5 4y ago
The message here is a private message. The title of this post contains the words "private communication" same for the title in the actual post. Have you not actually read anything about the post ?

BlargMcLarg 4y ago

It's funny because in this regard, The Netherlands is actually trying to do better and change the laws to allow mutually consenting teens to do this stuff (still tentative) and punish blackmail, prostitution (edit: of minors), minor-adult sexual interaction etc. Our general mentality also seems to be a lot more lax and supportive of kids being victims of their own mistakes rather than going "yeah, that's illegal, let's throw a fine on top". Almost as if we acknowledge teens are going to experiment.

vanderZwan 4y ago
It probably helps that we were complaining about and even making fun of[0] our government for using child pornography as an excuse for pushing through draconian populist laws a decade ago already. We may just happen to have a little head-start here basically. Edit: for the non-Dutch, the link goes to an article by the Dutch equivalent of The Onion and the headline says "Opstelten [our minister of Security and Justice at the time] says new regulations are needed or else child pornography". [0] https://speld.nl/2012/10/24/opstelten-%E2%80%98nieuwe-maatregelen-nodig-want-anders-kinderporno%E2%80%99/ https://speld.nl/2012/10/24/opstelten-%E2%80%98nieuwe-maatre...
- tluyben2 4y ago
  You need awareness that things most people don’t understand fully are absurd. Humour can be a tool for that.
- pc86 4y ago
  The Netherlands seems great as an outsider. I've been to Amsterdam, which was too touristy for me, but still beautiful. And we took a train to Utrecht and the surrounding areas which was great. Even struck up a conversation with a group of (local) software developers on the train. Even with the lockdown protest issues it seems like it should be on the shortlist if you want to move to the EU as an American.
  - eru 4y ago
    Keep in mind that comp for Dutch software developers is probably an order of magnitude lower than what you can get in the US.

marsRoverDev 4y ago

My understanding of this technology was it was usually a hash or fingerprint of some form of the material, to be compared against a database of hashes of known-bad material. I doubt EU countries have the infrastructure or budget to run models across everything, no?

GordonS 4y ago
An article about this was posted yesterday, and the EU wants to take it further that detecting known CSAM content, using AI to detect new content too. Of course, it's all a pretence for getting access to read all of your data. It might be a language thing, but the wording of the report was such that they were almost stating this explicitly. Honestly, what.the.hell are these clowns thinking?!
- matheusmoreira 4y ago
  The article says they already have technology that detects "grooming", whatever that means. That's seriously frightening, who knows what else they can detect? Maybe a few years from now we'll be reading articles about people getting arrested for expressing prohibited thoughts, subversive ideas, political opposition in what they thought was a private communications channel.
matt123456789 4y ago
That’s the implementation used by Apple for client-side scanning of iCloud uploads. NeuralHash or something. The EU may (and likely will) build an unrelated system, although it may share ideas with prior work.
Freak_NL 4y ago
Even with a database of hashes the risk remains. If someone gets off on nude children, then there is nothing stopping them from collecting innocent photographs from social media¹, reposting these on seedy or oniony websites, and have the hash ending up in the huge black box of CSAM hashes by an automated scraper. I am not under the impression that such a database is curated too closely. 1: This is not something I'm personally at risk of, because my parents are tech savvy enough not to post nude pics of a toddler anywhere but in a private chat with his parents (i.e., me and my wife).

londons_explore 4y ago

> I get sent pictures and clips of his progress Those pictures will be seen by you and grandma, but also maybe by employees of whoever you use to backup the pictures, whatever security services have a need to view the images, whichever police forensics department may need to look through your or grandmas phone if it were to be confiscated, etc. Those employees need a good work environment, and can't be expected to look at your naked three year old.

Freak_NL 4y ago
Having an actual person see those pics is totally fine. No one in their right mind would see anything but normal child raising there, and the judicial process has checks to prevent abuse by any one actor. It's algorithms I'm concerned about.
kevincox 4y ago
I'm sorry. The chance of someone needing to look at these pictures should be near 0. The backup company doesn't need to look at them. They should actually be end-to-end encrypted so that only me and grandma can see. In exceptional cases the police may need to see them. But even in that case there is nothing wrong with these pictures. Pictures of gore and literal shits are legal and while I agree that it can make this job very taxing. But a child swimming naked is not the problem here.
dvdkon 4y ago
We seem to have very different priorities. You are concerned about the welfare of employees of privacy-violating organisations. I'm concerned about citizens' privacy and say those organisations shouldn't exist.

peoplefromibiza 4y ago

> The automated filter won't care about the context though That's why we have a justice system. Specifically not a common law based one, which is kinds scary per se. People are innocent until proven guilty and the burden of proof's on the prosecution. We are talking about EU here, not Sudan.

pc86 4y ago
1. What makes you think Sudan (or NK, or Russia, or China) won't get their hands on this technology? 2. What makes you think every government in the entire EU will always have the people's best interests at heart? 3. What makes you think #2 is true even today? It seems like this technology could be very easy to abuse, to great public detriment. Is the juice worth the squeeze?
- peoplefromibiza 4y ago
  > 1. What makes you think Sudan (or NK, or Russia, or China) won't get their hands on this technology? their problem! they have more pressing issues already! and some of them have already implemented worse ways of surveilling their citizens. In this regards, even USA is far worse than EU! > 2. What makes you think every government in the entire EU will always have the people's best interests at heart? I don't, that's why we constantly check on them and change them if we don't like them > 3. What makes you think #2 is true even today? Because I live in EU and everywhere else I look the situation is far worse. > It seems like this technology could be very easy to abuse, to great public detriment. Is the juice worth the squeeze? Like any technology! Funny that guns are abused daily in US but the biggest preoccupation of HN is potential abuse of CSAM filters in EU... Which would also be illegal! If the filter has to be used for CSAM content, it can't be used for anything else.
  - pc86 4y ago
    > > 3. What makes you think #2 is true even today? > Because I live in EU and everywhere else I look the situation is far worse. "It's worse other places" doesn't answer the question. The EU's government isn't perfect and I don't see any safeguards in this proposal. It's ripe for abuse. > Funny that guns are abused daily in US Whataboutism nonsense, nobody is talking about guns except you. > If the filter has to be used for CSAM content, it can't be used for anything else. Utterly fanciful. Once this exists and is in place, it can and will be expanded to other things. Eventually a government you don't like will be elected in your country, and they'll be able to expand this. This tool in the hands of an authoritarian government is as dangerous as any firearm because it would allow them to track dissent.

kevincox 4y ago

The biggest problem is that these images are CSAM. Or at least a bit-for-bit identical image is in a different context. If it is family members are sharing an image of their child having fun or even sending an image to your doctor then there is no problem here. But those exact images in a different context could be a problem. Of course this is why they have human moderation before conviction. But I absolutely don't want my private images being reviewed because they may be CSAM. This is basically assuming that I have likely done something wrong and intruding my privacy because of that.

paganel 4y ago

At this point I would never send nor have nude photos with my kids (whom I don't have, but hypothetically speaking) on my phone or on any other electronic device, doing otherwise is just asking for trouble. Maybe it's not the perfect situation for parents who want to share said photos with the kid's grandparents thousands of kilometres away but it is what it is.

pfortuny 4y ago
Oh but OP is talking about his parents (the granparents) sending the photos. And you already have received them!

oauea 4y ago

> and nudism of any minor can be seen as CSAM if the pose they strike can be construed as 'erotic' So we're now expected to teach our kids what erotic poses are so they know not to accidentally do that during family pics, lest daddy or mommy wind up in jail? The future is fucked up.

kingcharles 4y ago
If you look at the history of litigation of nudism magazines in the 80s and 90s it is exactly that. Some magazines were allowed in, some were banned and the owners imprisoned because a judge said that some of the photos weren't candid or casual enough.

kingcharles 4y ago

I personally know someone who spent 13 years in prison because at 18 his 16 year old girlfriend sent him a nude selfie. Her Mom was in her phone and saw that she had sent the photos, so she called the police on her daughter's boyfriend. They arrested him, seized his phone and found the photo.

babyshake 4y ago

It would actually be great if iPhone/Android detected and offered to blur genitals with a click. As creepy as that may seem those of us with young children may only incidentally capture material that could be mistaken for CSAM and would be fine blurring out those details.