18 ms·

I'm really conflicted about this. For context, I deeply hate the abuse of children and I've worked on a contract before that landed 12 human traffickers in cus

I'm really conflicted about this. For context, I deeply hate the abuse of children and I've worked on a contract before that landed 12 human traffickers in custody that were smuggling sex slaves across boarders. I didn't need to know details about the victims in question, but it's understood that they're often teenagers or children. So my initial reaction when reading this Twitter thread was "let's get these bastards" but on serious reflection I think that impulse is wrong. Unshared data shouldn't be subject to search. Once it's shared, I can make several cases for an automated scan, but a cloud backup of personal media should be kept private. Our control of our own privacy matters. Not for the slippery slope argument or for the false positive argument, but for its own sake. We shouldn't be assuming the worst of people without cause or warrant. That said, even though I feel this way a not-small-enough part of me will be pleased if it is deployed because I want these people arrested. It's the same way I feel when terrorists get captured even if intelligence services bent or broke the rules. I can be happy at the outcome without being happy at the methods, and I can feel queasy about my own internal, conflicted feelings throughout it all.

drvdevd 5y ago
Since you worked on an actual contract catching these sorts of people you are perhaps in a unique position to answer the question: will this sort of blanket surveillance technique in general but also in iOS specifically - actually work to help catch them?
- jsjohnst 5y ago
  I have direct knowledge of examples of where individuals were arrested and convicted of sharing CP online and they were identified because a previous employer I worked for used PhotoDNA analysis on all user uploaded images. So yeah, this type of thing can catch bad people. I’m still not convinced Apple doing this is a good thing, especially on private media content without a warrant, even though the technology can help catch criminals.
  - disiplus 5y ago
    now im afraid, i have two young children < 5 years old. i have occasionally took pictures of them naked with some bumps on the skin or mosquito bite and sent them to my wife over whatsapp to look at and decide do we need to send them to doctor, do i have to fear now that i will be marked as distributing CP.
    - laumars 5y ago
      It’s not just you. I have pictures of my kids playing in the bath. No genitals are in shot and it’s just kids innocently playing with bubbles. The photos aren’t even shared but they’d still get scanned by this tool. This kind of thing isn’t even unusual either. I know my parents have pictures of myself and my siblings playing in the bath (obviously taken on film rather than digital photography) and I know friends have pictures of their kids too. While the difference between innocent images and something explicit easy for a human to identify, I’m not sure I’d trust AI to understand that nuance.
      - Closi 5y ago
        > While the difference between innocent images and something explicit easy for a human to identify, I’m not sure I’d trust AI to understand that nuance. In this case it’s not AI that’s understanding the nuance, it’s authorities that identify the exact pictures they want to track and then this tool lets them identify what phones/accounts have that photo (or presumably took it). If ‘AI’ is used here it is to detect if one photo contains all/part of another photo, rather than to determine if the photo is abusive or not. Although there is a legitimate slippery slope argument to be had here.
        nitrogen 5y ago
        Is there some way of verifying that the fingerprints in this database will never match sensitive documents on their way from a whistleblower to journalists, or anything else that isn't strictly illegal? How will this tech be repurposed over time once it's in place?
- deleted 5y ago
  [deleted]
- fvold 5y ago
  Just as being banned from one social media platform for bad behavior pushes people to a different social media platform, this might very well push the exactly wrong sort of people from iOS to Android. If Android then implements something similar, they have the option to simply run different software, as Android lets you run whatever you want so long as you sign the wavier. "You're using Android?! What do you have to hide?" -- Apple ad in 2030, possibly
- 3pt14159 5y ago
  I'm the person you're responding to, and I think so? My contract was on data that wasn't surveilled, it was willingly supplied in bad faith. Fake names, etc. And there was cause / outside evidence to look into it. I can't really go into more details than that, but it wasn't for an intelligence agency. It was for another party that wanted to hand something over to the police after they found out what was happening.
  - drvdevd 5y ago
    I see. I was responding to you, yes. And in this case I was more curious about your opinion - based on your previous knowledge - on the viability of Apple’s technology here, rather than the specific details of your work. In my (uninformed) opinion - this looks like more of a bad faith move on Apples part that will maybe catch some bad actors but will be a net harmful result for apple’s users and society, as expressed in the Twitter thread. Others who responded here though also seem to think it’ll be a viable technique.
0xy 5y ago
The NCMEC database that Apple is likely using to match hashes, contains countless non-CSAM pictures that are entirely legal not only in the U.S. but globally. This should be reason enough for you to not support the idea. From day 1, it's matching legal images and phoning home about them. Increasing the scope of scanning is barely a slippery slope, they're already beyond the stated scope of the database.
- wbl 5y ago
  Could you say more about these legal photos? That's a pretty big difference from what I thought was contained in the DB.
  - 0xy 5y ago
    If there are pictures recovered alongside CSAM but are not CSAM themselves they can be included in the database. The thing I can publicly say is that the database is not strictly for illegal or even borderline imagery. NCMEC try to keep the contents, processes and access to the database under wraps for obvious reasons.
    - oh_sigh 5y ago
      So are we talking about images which are not actually CSAM but a reasonable person would consider to be CSAM if they encountered them? Or is it just the README.txt for whatever popular bittorrent client?
      - 0xy 5y ago
        A reasonable person would call it a normal picture if they encountered it in isolation. Like I said, content which is not even borderline are included. By both a legal and moralistic standard they're not CSAM. Not even nearly. Of course, this is a minority of the content in the database. But even 1 such image is gross neglect of stated purpose in my book.
        5y ago
MikeUt 5y ago
> Unshared data shouldn't be subject to search. Once it's shared, I can make several cases for an automated scan, but a cloud backup of personal media should be kept private. Our control of our own privacy matters. Not for the slippery slope argument or for the false positive argument, but for its own sake. We shouldn't be assuming the worst of people without cause or warrant. I have a much simpler rule: Your device should never willingly* betray you. *With a warrant, police can attempt to plant a bug, but your device should not help them do so.
- ChrisKnott 5y ago
  I don't think this rule makes any sense, because it just abstracts all the argument into the word "betray". The vast majority of iPhone users won't consider it a betrayal that they can't send images of child abuse, any more than they consider it a betrayal that it doesn't come jailbroken. The victims of child abuse depicted in these images may well have considered it a betrayal by Apple that they allowed their privacy to be so flagrantly violated on their devices up until now.
  - MikeUt 5y ago
    > The vast majority of iPhone users won't consider it a betrayal that they can't send images of child abuse Probably neither would child abusers, since as soon as they send an image of child abuse, they're much more likely to be caught than if it had stayed on their phone. > a betrayal by Apple that they allowed their privacy to be so flagrantly violated on their devices up until now. "Their" devices? Once Apple sells an iPhone, it no longer belongs to Apple. Taking "betrayal" to mean "didn't plant backdoors on other people's computers to catch your abusers" is stretching that word far beyond reason.
  - feanaro 5y ago
    I don't think you read your ancestor post carefully enough. I at least don't see any room for ambiguity. The rule is that your (note the emphasis) device won't ever willingly betray you. There's nothing here that implicates the majority in any way. Simply, your own device should never work against you. This actually sounds like a great rule to prevent this kind of authoritarian scope creep.
arthur_sav 5y ago
- Don't you want to get the terrorists? - Yea yeah - Great. Give me access to every part of your life so i know you're not a terrorist.
- andy_ppp 5y ago
  “If you want a vision of the future, imagine a boot stamping on a human face - forever.” I always think about this Orwell quote and think it’s up to us to try to fight for what is good, but we were too busy doom-scrolling on Twitter to do anything about it.
hippari 5y ago
If what kills my enemies also kills my friends, then I don't want it.
shoto_io 5y ago
Is this confirmed yet or just something someone believes will happen? How credible are the sources of this twitter account?
mshroyer 5y ago
I'm not at all conflicted about this. Obviously CSAM is bad and should be stopped, but it is inevitable this new feature will become a means for governments to attack material of far more debatable 'harm'.
croes 5y ago
This scanning doesn't prevent the actual abuse and all this surveillance doesn't get to the root of the problem but can be misused by authoritarian governments. It's a pandoras box. You wouldn't allow the regular search of your home in real life.
oauea 5y ago
If you truly want to "protect the children" you should have no issue for the police to visit and inspect your, and all of your neighbors houses. Every few days. Unannounced, of course. And if you were to resist, you MUST be a pedophile who is actively abusing children in their basement.
- amelius 5y ago
  I'm actually more OK with unannounced inspections of my basement (within reason) than with some government agents reading through my files all the time.
  - oauea 5y ago
    Why just your basement? While they're there they might kindly ask you to unlock that computer or hand over that phone. Just to make sure, of course.
ChrisMarshallNY 5y ago
Having known many victims of sexual violence and trafficking, I feel for the folks that honestly want that particular kind of crime to stop. Humans can be complete scum. Most folks in this community may think they know how low we can go, but you are likely being optimistic. That said, law enforcement has a nasty habit of having a rather "binary" worldview. People are either cops, or uncaught criminals. ..and they wonder why they have so much trouble making non-cop friends (DISCLAIMER: I know a number of cops). With that worldview, it can be quite easy to "blur the line" between child sex traffickers, and parking ticket violators. I remember reading a The Register article, about how anti-terrorism statutes are being abused by local town councils to do things like find zoning violations (for example, pools with no CO). Misapplied laws can be much worse than letting some criminals go. This could easily become a nightmare, if we cede too much to AI. And that isn't even talking about totalitarian regimes, run by people of the same ilk as child sex traffickers (only wearing Gucci, and living in palaces). ”Any proposal must be viewed as follows. Do not pay overly much attention to the benefits that might be delivered were the law in question to be properly enforced, rather one needs to consider the harm done by the improper enforcement of this particular piece of legislation, whatever it might be.” -Lyndon B. Johnson
- junon 5y ago
  > People are either cops, or uncaught criminals. ..and they wonder why they have so much trouble making non-cop friends (DISCLAIMER: I know a number of cops). Ehh let's not make a habit of asserting anecdote as fact, please. Saying you know cops is like saying you know black people and that somehow it affords you some privilege others do not possess. This is a weak and ad-hom argument.
  - freshdonut 5y ago
    > is like saying you know black people and that somehow it affords you some privilege others do not possess. Of course it does. Interacting with black people (or any race) affords you insight into their life experiences, struggles, worldview etc... Of course sociological discourse is highly subjective but this attitude on HN that anecdotal data has no value whatsoever is silly. Do you seriously expect every fact of every people to be published in some infallible academic journal?
sho 5y ago
> I'm really conflicted about this. I'm not. I am very unambiguously against this and I think if word gets out Apple could have a real problem. I would like to think I am against child porn as any well-adjusted adult. That does not mean I wish for all my files to be scanned without my consent or even knowledge for compliance, for submission to who knows where matching to who knows what reporting to, well, who knows. That's crossing a line. You are now reading my private files, interpreting them, and doing something based on that interpretation. That is surveillance. I am very not OK with this.
slumdev 5y ago
Let me help you, then, because you shouldn't be conflicted at all. "Think of the children" is how they force changes that would otherwise be unconscionable. They've done it with encryption and anonymity for years. Now they're doing it with the hardware in your pocket.
orangegreen 5y ago
The road to hell is paved with good intentions...
suizi 5y ago
There have been child abuse victims who have openly condemned this sort of intrusion on privacy, although they obviously don't speak for them all.
suizi 5y ago
Well, it's a lot like everything. No one wants abusers, murderers, and others out and about. But then, we can't search everyone's homes all of the time for dead bodies, or other crimes. We would all be better off without these things happening, and anyone would want less of it to happen.
Grustaf 5y ago
> Unshared data shouldn't be subject to search Since they are only searching for _known_ abusive content, by definition they can only detect data that has been shared, which I think is the important point here.

drvdevd 5y ago

Since you worked on an actual contract catching these sorts of people you are perhaps in a unique position to answer the question: will this sort of blanket surveillance technique in general but also in iOS specifically - actually work to help catch them?

jsjohnst 5y ago
I have direct knowledge of examples of where individuals were arrested and convicted of sharing CP online and they were identified because a previous employer I worked for used PhotoDNA analysis on all user uploaded images. So yeah, this type of thing can catch bad people. I’m still not convinced Apple doing this is a good thing, especially on private media content without a warrant, even though the technology can help catch criminals.
- disiplus 5y ago
  now im afraid, i have two young children < 5 years old. i have occasionally took pictures of them naked with some bumps on the skin or mosquito bite and sent them to my wife over whatsapp to look at and decide do we need to send them to doctor, do i have to fear now that i will be marked as distributing CP.
  - laumars 5y ago
    It’s not just you. I have pictures of my kids playing in the bath. No genitals are in shot and it’s just kids innocently playing with bubbles. The photos aren’t even shared but they’d still get scanned by this tool. This kind of thing isn’t even unusual either. I know my parents have pictures of myself and my siblings playing in the bath (obviously taken on film rather than digital photography) and I know friends have pictures of their kids too. While the difference between innocent images and something explicit easy for a human to identify, I’m not sure I’d trust AI to understand that nuance.
    - Closi 5y ago
      > While the difference between innocent images and something explicit easy for a human to identify, I’m not sure I’d trust AI to understand that nuance. In this case it’s not AI that’s understanding the nuance, it’s authorities that identify the exact pictures they want to track and then this tool lets them identify what phones/accounts have that photo (or presumably took it). If ‘AI’ is used here it is to detect if one photo contains all/part of another photo, rather than to determine if the photo is abusive or not. Although there is a legitimate slippery slope argument to be had here.
      - nitrogen 5y ago
        Is there some way of verifying that the fingerprints in this database will never match sensitive documents on their way from a whistleblower to journalists, or anything else that isn't strictly illegal? How will this tech be repurposed over time once it's in place?
deleted 5y ago
[deleted]
fvold 5y ago
Just as being banned from one social media platform for bad behavior pushes people to a different social media platform, this might very well push the exactly wrong sort of people from iOS to Android. If Android then implements something similar, they have the option to simply run different software, as Android lets you run whatever you want so long as you sign the wavier. "You're using Android?! What do you have to hide?" -- Apple ad in 2030, possibly
3pt14159 5y ago
I'm the person you're responding to, and I think so? My contract was on data that wasn't surveilled, it was willingly supplied in bad faith. Fake names, etc. And there was cause / outside evidence to look into it. I can't really go into more details than that, but it wasn't for an intelligence agency. It was for another party that wanted to hand something over to the police after they found out what was happening.
- drvdevd 5y ago
  I see. I was responding to you, yes. And in this case I was more curious about your opinion - based on your previous knowledge - on the viability of Apple’s technology here, rather than the specific details of your work. In my (uninformed) opinion - this looks like more of a bad faith move on Apples part that will maybe catch some bad actors but will be a net harmful result for apple’s users and society, as expressed in the Twitter thread. Others who responded here though also seem to think it’ll be a viable technique.

0xy 5y ago

The NCMEC database that Apple is likely using to match hashes, contains countless non-CSAM pictures that are entirely legal not only in the U.S. but globally. This should be reason enough for you to not support the idea. From day 1, it's matching legal images and phoning home about them. Increasing the scope of scanning is barely a slippery slope, they're already beyond the stated scope of the database.

wbl 5y ago
Could you say more about these legal photos? That's a pretty big difference from what I thought was contained in the DB.
- 0xy 5y ago
  If there are pictures recovered alongside CSAM but are not CSAM themselves they can be included in the database. The thing I can publicly say is that the database is not strictly for illegal or even borderline imagery. NCMEC try to keep the contents, processes and access to the database under wraps for obvious reasons.
  - oh_sigh 5y ago
    So are we talking about images which are not actually CSAM but a reasonable person would consider to be CSAM if they encountered them? Or is it just the README.txt for whatever popular bittorrent client?
    - 0xy 5y ago
      A reasonable person would call it a normal picture if they encountered it in isolation. Like I said, content which is not even borderline are included. By both a legal and moralistic standard they're not CSAM. Not even nearly. Of course, this is a minority of the content in the database. But even 1 such image is gross neglect of stated purpose in my book.
      - 5y ago

MikeUt 5y ago

> Unshared data shouldn't be subject to search. Once it's shared, I can make several cases for an automated scan, but a cloud backup of personal media should be kept private. Our control of our own privacy matters. Not for the slippery slope argument or for the false positive argument, but for its own sake. We shouldn't be assuming the worst of people without cause or warrant. I have a much simpler rule: Your device should never willingly* betray you. *With a warrant, police can attempt to plant a bug, but your device should not help them do so.

ChrisKnott 5y ago
I don't think this rule makes any sense, because it just abstracts all the argument into the word "betray". The vast majority of iPhone users won't consider it a betrayal that they can't send images of child abuse, any more than they consider it a betrayal that it doesn't come jailbroken. The victims of child abuse depicted in these images may well have considered it a betrayal by Apple that they allowed their privacy to be so flagrantly violated on their devices up until now.
- MikeUt 5y ago
  > The vast majority of iPhone users won't consider it a betrayal that they can't send images of child abuse Probably neither would child abusers, since as soon as they send an image of child abuse, they're much more likely to be caught than if it had stayed on their phone. > a betrayal by Apple that they allowed their privacy to be so flagrantly violated on their devices up until now. "Their" devices? Once Apple sells an iPhone, it no longer belongs to Apple. Taking "betrayal" to mean "didn't plant backdoors on other people's computers to catch your abusers" is stretching that word far beyond reason.
- feanaro 5y ago
  I don't think you read your ancestor post carefully enough. I at least don't see any room for ambiguity. The rule is that your (note the emphasis) device won't ever willingly betray you. There's nothing here that implicates the majority in any way. Simply, your own device should never work against you. This actually sounds like a great rule to prevent this kind of authoritarian scope creep.

arthur_sav 5y ago

- Don't you want to get the terrorists? - Yea yeah - Great. Give me access to every part of your life so i know you're not a terrorist.

andy_ppp 5y ago
“If you want a vision of the future, imagine a boot stamping on a human face - forever.” I always think about this Orwell quote and think it’s up to us to try to fight for what is good, but we were too busy doom-scrolling on Twitter to do anything about it.

hippari 5y ago

If what kills my enemies also kills my friends, then I don't want it.

shoto_io 5y ago

Is this confirmed yet or just something someone believes will happen? How credible are the sources of this twitter account?

mshroyer 5y ago

I'm not at all conflicted about this. Obviously CSAM is bad and should be stopped, but it is inevitable this new feature will become a means for governments to attack material of far more debatable 'harm'.

croes 5y ago

This scanning doesn't prevent the actual abuse and all this surveillance doesn't get to the root of the problem but can be misused by authoritarian governments. It's a pandoras box. You wouldn't allow the regular search of your home in real life.

oauea 5y ago

If you truly want to "protect the children" you should have no issue for the police to visit and inspect your, and all of your neighbors houses. Every few days. Unannounced, of course. And if you were to resist, you MUST be a pedophile who is actively abusing children in their basement.

amelius 5y ago
I'm actually more OK with unannounced inspections of my basement (within reason) than with some government agents reading through my files all the time.
- oauea 5y ago
  Why just your basement? While they're there they might kindly ask you to unlock that computer or hand over that phone. Just to make sure, of course.

ChrisMarshallNY 5y ago

Having known many victims of sexual violence and trafficking, I feel for the folks that honestly want that particular kind of crime to stop. Humans can be complete scum. Most folks in this community may think they know how low we can go, but you are likely being optimistic. That said, law enforcement has a nasty habit of having a rather "binary" worldview. People are either cops, or uncaught criminals. ..and they wonder why they have so much trouble making non-cop friends (DISCLAIMER: I know a number of cops). With that worldview, it can be quite easy to "blur the line" between child sex traffickers, and parking ticket violators. I remember reading a The Register article, about how anti-terrorism statutes are being abused by local town councils to do things like find zoning violations (for example, pools with no CO). Misapplied laws can be much worse than letting some criminals go. This could easily become a nightmare, if we cede too much to AI. And that isn't even talking about totalitarian regimes, run by people of the same ilk as child sex traffickers (only wearing Gucci, and living in palaces). ”Any proposal must be viewed as follows. Do not pay overly much attention to the benefits that might be delivered were the law in question to be properly enforced, rather one needs to consider the harm done by the improper enforcement of this particular piece of legislation, whatever it might be.” -Lyndon B. Johnson

junon 5y ago
> People are either cops, or uncaught criminals. ..and they wonder why they have so much trouble making non-cop friends (DISCLAIMER: I know a number of cops). Ehh let's not make a habit of asserting anecdote as fact, please. Saying you know cops is like saying you know black people and that somehow it affords you some privilege others do not possess. This is a weak and ad-hom argument.
- freshdonut 5y ago
  > is like saying you know black people and that somehow it affords you some privilege others do not possess. Of course it does. Interacting with black people (or any race) affords you insight into their life experiences, struggles, worldview etc... Of course sociological discourse is highly subjective but this attitude on HN that anecdotal data has no value whatsoever is silly. Do you seriously expect every fact of every people to be published in some infallible academic journal?

sho 5y ago

> I'm really conflicted about this. I'm not. I am very unambiguously against this and I think if word gets out Apple could have a real problem. I would like to think I am against child porn as any well-adjusted adult. That does not mean I wish for all my files to be scanned without my consent or even knowledge for compliance, for submission to who knows where matching to who knows what reporting to, well, who knows. That's crossing a line. You are now reading my private files, interpreting them, and doing something based on that interpretation. That is surveillance. I am very not OK with this.

slumdev 5y ago

Let me help you, then, because you shouldn't be conflicted at all. "Think of the children" is how they force changes that would otherwise be unconscionable. They've done it with encryption and anonymity for years. Now they're doing it with the hardware in your pocket.

orangegreen 5y ago

The road to hell is paved with good intentions...

suizi 5y ago

There have been child abuse victims who have openly condemned this sort of intrusion on privacy, although they obviously don't speak for them all.

suizi 5y ago

Well, it's a lot like everything. No one wants abusers, murderers, and others out and about. But then, we can't search everyone's homes all of the time for dead bodies, or other crimes. We would all be better off without these things happening, and anyone would want less of it to happen.

Grustaf 5y ago

> Unshared data shouldn't be subject to search Since they are only searching for _known_ abusive content, by definition they can only detect data that has been shared, which I think is the important point here.