6 ms·

My opinion about: 1. Every pedophile know about the existence of this system, so I don't think it will be useful to fight those monster, maybe only marginally;

My opinion about: 1. Every pedophile know about the existence of this system, so I don't think it will be useful to fight those monster, maybe only marginally; 2. Anyway, is that legal ? Even if some crazy store material on his Apple hardware isn't that illegal search non usable in law courts ? 3. Child abuse is often used as Trojan horse to introduce questionable practice. What if: - the system is used to looking for dissidents: I look for people that have a photo of Tiananmen Square protests on their pc, for example; - for espionage: I have the hash of some documents of interest, so all the PCs with that kind of documents could be a valuable target; - profiling people: you have computer virus sample on your PC -> security researcher/hacker; I think that the system is prone to all kind of privacy abuse. 4. this could be part of the previous point, but, because I think it's the final and real reason for the existence of that system, I give to this point its own section: piracy fight. I think that the one of the real reason is to discourage the exchange of illigal multimedia material to enforce copyrighs. For the listed reasons, I think that is a bad idea. Let me know what are you thinking about.

aborsy 5y ago
I agree!
simonh 5y ago
Of course it would be possible to implement content search, profiling and reporting mechanisms for such content, but this seems to be a singularly bad platform for that sort of search. The image profiles are part of the OS so there's no mechanism to deliver image profiles separately for different countries. Also when the threshold number of matching images is reached, the matches are reported to a manual reviewer at Apple not a government. It only checks images on upload to iCloud photo storage. So of course each of these limitations of the system could be changed, but you'd really need to change all of them and at that point you've created a completely different system. There's no simple change to this system that would suddenly turn it into a snitch for e.g. China or Saudi Arabia. I've seen exactly the same objections raised every time any kind of device content search has become mainstream. Back in the 90s it was virus checking (Do you trust the AV company? What if they were bribed by the content companies?), full device indexing and search (Do you trust the OS vendor? What if they're in league with the government?). I'm very surprised this didn't blow up when Apple implemented ubiquitous image text recognition. Maybe it did. AV and device indexing mechanisms, which are ubiquitous, seem like a far more vulnerable target for such requirements. So I don't really buy the slippery slope argument. In theory any government could pass a law requiring any company operating in it's jurisdiction to do anything, with an implementation suitable to that actual purpose. Of course this mechanism is motivated by laws in the US so it's a perfect example of exactly that, and it's a completely new system not a slippery slope subversion of an existing one. The real slippery slope here is legislative, not technical and I think that should be far, far more concerning. I do think the legal and moral questions about this mechanism are legitimate. I think it would make more sense for Apple to scan photos in their cloud storage on the cloud storage rather than on upload. I understand there are theoretical privacy benefits to users from this implementation but the optics of having user's devices snitch on them are all wrong.
- aNoob7000 5y ago
  Apple should just scan the pictures that are in iCloud (their servers). They just assumed that if you have the iCloud option enabled on your device that it gave them the right to do the scan on your phone/computer. I want to also point out that A/V companies never said they were going to scan for child abuse images on your computer and report you if they found any. Like you said, the optics are terrible.
  - deleted 5y ago
    [deleted]
  - nicce 5y ago
    > Apple should just scan the pictures that are in iCloud (their servers). They just assumed that if you have the iCloud option enabled on your device that it gave them the right to do the scan on your phone/computer. End result is the same. Difference is, that now Apple has very limited access to your images. You can only trust in closed systems. When you step into the Apple ecosystem, you are giving a lot of trust. > I want to also point out that A/V companies never said they were going to scan for child abuse images on your computer and report you if they found any. Why would they say, if it is perfectly legal to do anyway. They literally scan every file, so no need to mention anything specific which could lead only for negative PR.
    - zionic 5y ago
      Apple has unlimited access now, all that stands in their way is the thinnest of policy lines.
      - nicce 5y ago
        > Apple has unlimited access now Always been. You don't own your iOS based device which is very closed source and mostly unusable for any other operating systems.
- luckyandroid 5y ago
  >Back in the 90s it was virus checking (Do you trust the AV company? What if they were bribed by the content companies?), full device indexing and search (Do you trust the OS vendor? What if they're in league with the government?) These are examples of companies choosing to do something as a selling point of their software as a benefit to the end user, and people worrying that it could aid the government down the line if they change their mind. Apple's content review change is explicitly FOR reporting people to police in a way that can be expanded beyond it's currently set purpose (child porn) later. >I'm very surprised this didn't blow up when Apple implemented ubiquitous image text recognition. I'm personally not a fan of that stuff anyway, but personally if it's only my local device I don't tend to care about image recognition, it's only when it involves communicating information from MY hardware to THEIR servers that I get antsy.
lovelyviking 5y ago
Perhaps we should also rise again the question about who controls own computer because any abuse starts there. Only full control over own device can prevent abuses. Especially when device comes any close to definition of being personal. You should be able to install own software on the personal device. Including os and bios/firmware.
- villgax 5y ago
  *laughs in Minix land
  - _trampeltier 5y ago
    An inside joke :-)
larschdk 5y ago
5. The system can easily be abused by governments or malicious actors to frame innocent people. People merely suspected of keeping such images are de-facto punished and stripped of rights even without standing before a judge or getting a conviction.
- GeekyBear 5y ago
  Google has been scanning your entire account for kiddie porn for the past decade. >a man [was] arrested on child pornography charges, after Google tipped off authorities about illegal images found in the Houston suspect's Gmail account https://techcrunch.com/2014/08/06/why-the-gmail-scan-that-led-to-a-mans-arrest-for-child-porn-was-not-a-privacy-violation/ https://techcrunch.com/2014/08/06/why-the-gmail-scan-that-le... Their system can easily be abused by governments or malicious actors to frame innocent people.
  - Unklejoe 5y ago
    Once again, this scan takes place on _their_ servers on data that is stored on _their_ servers. It does not take place on the device itself, which is the case with this new Apple thing.
    - GeekyBear 5y ago
      Once again, the notion that everything on the device is scanned under Apple's system was never true. Only photos you attempt to upload to Apple's iCloud are scanned. If you turn off iCloud photos, NOTHING is scanned. >Q: So if iCloud Photos is disabled, the system does not work, which is the public language in the FAQ. I just wanted to ask specifically, when you disable iCloud Photos, does this system continue to create hashes of your photos on device, or is it completely inactive at that point? A: If users are not using iCloud Photos, NeuralHash will not run https://techcrunch.com/2021/08/10/interview-apples-head-of-privacy-details-child-abuse-detection-and-messages-safety-features/ https://techcrunch.com/2021/08/10/interview-apples-head-of-p...
      - 5y ago
tsherr 5y ago
As I understand it, in Canada, having a picture of your child in a bathtub is child porn. So I can see this going horribly wrong.
- 3pt14159 5y ago
  Citation needed. Many parents here have photos of their kids in bathtubs.
  - jackson1way 5y ago
    Many people steal bicycles and are not in handcuffs or even prosecuted. Doesn‘t mean it‘s legal, right?
  - newbamboo 5y ago
    Selective enforcement should concern anyone who cares about civil liberties.
    - 3pt14159 5y ago
      I still don't see a citation, and I'm not exactly about to google for one and click around.
- swiley 5y ago
  The Canadian government has really lost their minds, that's where ignoring principles gets you.
deleted 5y ago
[deleted]
janeroe 5y ago
> What if: the system is used to looking for dissidents ... / espionage ... / profiling ... Not if. When.
- zionic 5y ago
  Lets go one step further, Apple is a content producer with multiple streaming services now. Using this system to look for unlicensed content will be irresistible to them.
snarf21 5y ago
I think this is a bad move by Apple even if the point is to set up E2EE later. However, one thing that everyone seems to forget is that all these pictures were already being sent un-encrypted to iCloud. ALL of the same issues already completely exist today and were already being scanned and we have heard no outcry. ALL of the same loopholes and unreasonable warrants can be used against you today with all of the un-encrypted data they have on their server right now. The one thing that occurred to me is that this is almost seems like this is a cya, Section 230 protection in disguise. There has been more discussions about Big Tech and 230, and this is one way to say "Look, we are compliant on our platform. Don't remove our protections or break us up, we are your friend!" It also shouldn't be too surprising given how Apple has behaved in China. They will only push back against the government up until the point it starts to affect profits.
- jackTheMan 5y ago
  Even if it is possible in other ways as well today, this is a black pattern of going down step-by-step. When it will be when people will say no? These are all small steps only.
  - nicce 5y ago
    > this is a black pattern of going down step-by-step. This is very hard to argue. Functionality like spying all your files is trivial to add, and technically we haven't really moved anywhere. "Now technology is there", is not valid argument since it has always been there. Scanning your files and send some metadata is the feature which requires least effort to make from everything that Apple has released. It might feel bad, when your device scans your to-be-uploaded-cloud images now, but iOS has never been yours. It is very closed system, a part of Apple-ecosystem. Only a guy who as has access for whole iOS source code knows what is actually happening in there. On Apple-ecosystem, only the final result matters in reality and what they say. Since your device is not really yours, you should think like that you are just using Apple-ecosystem, being part of it. If you don't want that, you should have switched into some Linux phone already. You can speculate all-day what else it might do in hidden in the future. Speculation about hidden features is as valid now as it was yesterday or will be tomorrow. In reality, we can only be really worried when they publicly say something, which finally makes the end results worse. This did not happen yet. Actually the opposite happened, but here we go. We have been trusting Apple for quite some time, and they really haven't got caught on doing something else than they have said, so what has changed?
Bodell 5y ago
I just want to add to your number 1. Not only would this be marginal it also wouldn’t necessarily be catching the real “monsters”. I don’t think if you find someone with old already known about images that it would necessarily equate to someone that actually abuses children. I think about this in a similar way (not exactly) as I do with drugs, just because a person gets busted with drugs doesn’t mean they are a drug dealer or a maker of drugs. This is not to say that perhaps there are some more active real-time stuff in these databases that maybe with enough searching could make its way back to the perpetrator and indeed maybe even find a victim. It’s just seems that that would be far more marginal and is generally what I’m concerned about when it comes to these issues. For me it’s more important to protect children than it is to bust some weirdos for looking at the wrong porn (these can both be related as well and I do understand that I just think it’s not as cut and dry as we believe it is), further if it keeps said weirdo from actually harming a child then let them have it. We allow these databases to exist for, presumably, the same reason, with the idea that we can stop future victims from happening.
eplanit 5y ago
Today they look for child porn, tomorrow it will be "Covid Misinformation" (which is today's terrorism). A blind man with a stick can see it coming.
- stalkersyndrome 5y ago
  The honest among of us, and perpetrators, have seen this coming. It’s the paid shills and unpaid fools who do bidding of masters they are unaware of exist that you have to be on the look out for. HN has been under massive astroturfing/disinformation campaigns that the honest people here can’t separate reality from fiction. I don’t even want to think about how badly larger entities are dealing with it.
- docmars 5y ago
  I can't reply to or upvote ~stalkersyndrome's response to add my applause, but I'll do it here instead. He's right. It's inconvenient, and people don't like reading that, which is why it's downvoted to dead, but alas, this is the same reason people avoid getting counseling, because they haven't worked up the courage to be honest with themselves yet. We'll get there, I hope. https://news.ycombinator.com/item?id=28156934 https://news.ycombinator.com/item?id=28156934
nzealand 5y ago
> 2. Anyway, is that legal ? Even if some crazy store material on his Apple hardware isn't that illegal search non usable in law courts ? Yes, it's considered legal. Apple reviews the content first. Courts say this means it is not an illegal government search. It's a search by a private party, who then manually decides to notify the government.
- cookiengineer 5y ago
  > Yes, it's considered legal. No, it's not. At least not here in Germany. By law, even police officers are not allowed to look at child porn. The only institution explicitely allowed to do so is the BSI. The rest of the population implicitely incriminates themselves when they look at (not own) child porn, including Apple's legal entity or employees. See [1] for 184b Strafgesetzbuch I'm trying to point out that with this action Apple bluntly decided to ignore a whole lot of countries and their federal laws, which is not something I would embrace - even when they had good intentions. [1] https://www.gesetze-im-internet.de/stgb/__184b.html https://www.gesetze-im-internet.de/stgb/__184b.html
  - nzealand 5y ago
    I think they are only planning to scan US phones, so I am not sure how German law applies here.

aborsy 5y ago

I agree!

simonh 5y ago

Of course it would be possible to implement content search, profiling and reporting mechanisms for such content, but this seems to be a singularly bad platform for that sort of search. The image profiles are part of the OS so there's no mechanism to deliver image profiles separately for different countries. Also when the threshold number of matching images is reached, the matches are reported to a manual reviewer at Apple not a government. It only checks images on upload to iCloud photo storage. So of course each of these limitations of the system could be changed, but you'd really need to change all of them and at that point you've created a completely different system. There's no simple change to this system that would suddenly turn it into a snitch for e.g. China or Saudi Arabia. I've seen exactly the same objections raised every time any kind of device content search has become mainstream. Back in the 90s it was virus checking (Do you trust the AV company? What if they were bribed by the content companies?), full device indexing and search (Do you trust the OS vendor? What if they're in league with the government?). I'm very surprised this didn't blow up when Apple implemented ubiquitous image text recognition. Maybe it did. AV and device indexing mechanisms, which are ubiquitous, seem like a far more vulnerable target for such requirements. So I don't really buy the slippery slope argument. In theory any government could pass a law requiring any company operating in it's jurisdiction to do anything, with an implementation suitable to that actual purpose. Of course this mechanism is motivated by laws in the US so it's a perfect example of exactly that, and it's a completely new system not a slippery slope subversion of an existing one. The real slippery slope here is legislative, not technical and I think that should be far, far more concerning. I do think the legal and moral questions about this mechanism are legitimate. I think it would make more sense for Apple to scan photos in their cloud storage on the cloud storage rather than on upload. I understand there are theoretical privacy benefits to users from this implementation but the optics of having user's devices snitch on them are all wrong.

aNoob7000 5y ago
Apple should just scan the pictures that are in iCloud (their servers). They just assumed that if you have the iCloud option enabled on your device that it gave them the right to do the scan on your phone/computer. I want to also point out that A/V companies never said they were going to scan for child abuse images on your computer and report you if they found any. Like you said, the optics are terrible.
- deleted 5y ago
  [deleted]
- nicce 5y ago
  > Apple should just scan the pictures that are in iCloud (their servers). They just assumed that if you have the iCloud option enabled on your device that it gave them the right to do the scan on your phone/computer. End result is the same. Difference is, that now Apple has very limited access to your images. You can only trust in closed systems. When you step into the Apple ecosystem, you are giving a lot of trust. > I want to also point out that A/V companies never said they were going to scan for child abuse images on your computer and report you if they found any. Why would they say, if it is perfectly legal to do anyway. They literally scan every file, so no need to mention anything specific which could lead only for negative PR.
  - zionic 5y ago
    Apple has unlimited access now, all that stands in their way is the thinnest of policy lines.
    - nicce 5y ago
      > Apple has unlimited access now Always been. You don't own your iOS based device which is very closed source and mostly unusable for any other operating systems.
luckyandroid 5y ago
>Back in the 90s it was virus checking (Do you trust the AV company? What if they were bribed by the content companies?), full device indexing and search (Do you trust the OS vendor? What if they're in league with the government?) These are examples of companies choosing to do something as a selling point of their software as a benefit to the end user, and people worrying that it could aid the government down the line if they change their mind. Apple's content review change is explicitly FOR reporting people to police in a way that can be expanded beyond it's currently set purpose (child porn) later. >I'm very surprised this didn't blow up when Apple implemented ubiquitous image text recognition. I'm personally not a fan of that stuff anyway, but personally if it's only my local device I don't tend to care about image recognition, it's only when it involves communicating information from MY hardware to THEIR servers that I get antsy.

lovelyviking 5y ago

Perhaps we should also rise again the question about who controls own computer because any abuse starts there. Only full control over own device can prevent abuses. Especially when device comes any close to definition of being personal. You should be able to install own software on the personal device. Including os and bios/firmware.

villgax 5y ago
*laughs in Minix land
- _trampeltier 5y ago
  An inside joke :-)

larschdk 5y ago

5. The system can easily be abused by governments or malicious actors to frame innocent people. People merely suspected of keeping such images are de-facto punished and stripped of rights even without standing before a judge or getting a conviction.

GeekyBear 5y ago
Google has been scanning your entire account for kiddie porn for the past decade. >a man [was] arrested on child pornography charges, after Google tipped off authorities about illegal images found in the Houston suspect's Gmail account https://techcrunch.com/2014/08/06/why-the-gmail-scan-that-led-to-a-mans-arrest-for-child-porn-was-not-a-privacy-violation/ https://techcrunch.com/2014/08/06/why-the-gmail-scan-that-le... Their system can easily be abused by governments or malicious actors to frame innocent people.
- Unklejoe 5y ago
  Once again, this scan takes place on _their_ servers on data that is stored on _their_ servers. It does not take place on the device itself, which is the case with this new Apple thing.
  - GeekyBear 5y ago
    Once again, the notion that everything on the device is scanned under Apple's system was never true. Only photos you attempt to upload to Apple's iCloud are scanned. If you turn off iCloud photos, NOTHING is scanned. >Q: So if iCloud Photos is disabled, the system does not work, which is the public language in the FAQ. I just wanted to ask specifically, when you disable iCloud Photos, does this system continue to create hashes of your photos on device, or is it completely inactive at that point? A: If users are not using iCloud Photos, NeuralHash will not run https://techcrunch.com/2021/08/10/interview-apples-head-of-privacy-details-child-abuse-detection-and-messages-safety-features/ https://techcrunch.com/2021/08/10/interview-apples-head-of-p...
    - 5y ago

tsherr 5y ago

As I understand it, in Canada, having a picture of your child in a bathtub is child porn. So I can see this going horribly wrong.

3pt14159 5y ago
Citation needed. Many parents here have photos of their kids in bathtubs.
- jackson1way 5y ago
  Many people steal bicycles and are not in handcuffs or even prosecuted. Doesn‘t mean it‘s legal, right?
- newbamboo 5y ago
  Selective enforcement should concern anyone who cares about civil liberties.
  - 3pt14159 5y ago
    I still don't see a citation, and I'm not exactly about to google for one and click around.
swiley 5y ago
The Canadian government has really lost their minds, that's where ignoring principles gets you.

deleted 5y ago

[deleted]

janeroe 5y ago

> What if: the system is used to looking for dissidents ... / espionage ... / profiling ... Not if. When.

zionic 5y ago
Lets go one step further, Apple is a content producer with multiple streaming services now. Using this system to look for unlicensed content will be irresistible to them.

snarf21 5y ago

I think this is a bad move by Apple even if the point is to set up E2EE later. However, one thing that everyone seems to forget is that all these pictures were already being sent un-encrypted to iCloud. ALL of the same issues already completely exist today and were already being scanned and we have heard no outcry. ALL of the same loopholes and unreasonable warrants can be used against you today with all of the un-encrypted data they have on their server right now. The one thing that occurred to me is that this is almost seems like this is a cya, Section 230 protection in disguise. There has been more discussions about Big Tech and 230, and this is one way to say "Look, we are compliant on our platform. Don't remove our protections or break us up, we are your friend!" It also shouldn't be too surprising given how Apple has behaved in China. They will only push back against the government up until the point it starts to affect profits.

jackTheMan 5y ago
Even if it is possible in other ways as well today, this is a black pattern of going down step-by-step. When it will be when people will say no? These are all small steps only.
- nicce 5y ago
  > this is a black pattern of going down step-by-step. This is very hard to argue. Functionality like spying all your files is trivial to add, and technically we haven't really moved anywhere. "Now technology is there", is not valid argument since it has always been there. Scanning your files and send some metadata is the feature which requires least effort to make from everything that Apple has released. It might feel bad, when your device scans your to-be-uploaded-cloud images now, but iOS has never been yours. It is very closed system, a part of Apple-ecosystem. Only a guy who as has access for whole iOS source code knows what is actually happening in there. On Apple-ecosystem, only the final result matters in reality and what they say. Since your device is not really yours, you should think like that you are just using Apple-ecosystem, being part of it. If you don't want that, you should have switched into some Linux phone already. You can speculate all-day what else it might do in hidden in the future. Speculation about hidden features is as valid now as it was yesterday or will be tomorrow. In reality, we can only be really worried when they publicly say something, which finally makes the end results worse. This did not happen yet. Actually the opposite happened, but here we go. We have been trusting Apple for quite some time, and they really haven't got caught on doing something else than they have said, so what has changed?

Bodell 5y ago

I just want to add to your number 1. Not only would this be marginal it also wouldn’t necessarily be catching the real “monsters”. I don’t think if you find someone with old already known about images that it would necessarily equate to someone that actually abuses children. I think about this in a similar way (not exactly) as I do with drugs, just because a person gets busted with drugs doesn’t mean they are a drug dealer or a maker of drugs. This is not to say that perhaps there are some more active real-time stuff in these databases that maybe with enough searching could make its way back to the perpetrator and indeed maybe even find a victim. It’s just seems that that would be far more marginal and is generally what I’m concerned about when it comes to these issues. For me it’s more important to protect children than it is to bust some weirdos for looking at the wrong porn (these can both be related as well and I do understand that I just think it’s not as cut and dry as we believe it is), further if it keeps said weirdo from actually harming a child then let them have it. We allow these databases to exist for, presumably, the same reason, with the idea that we can stop future victims from happening.

eplanit 5y ago

Today they look for child porn, tomorrow it will be "Covid Misinformation" (which is today's terrorism). A blind man with a stick can see it coming.

stalkersyndrome 5y ago
The honest among of us, and perpetrators, have seen this coming. It’s the paid shills and unpaid fools who do bidding of masters they are unaware of exist that you have to be on the look out for. HN has been under massive astroturfing/disinformation campaigns that the honest people here can’t separate reality from fiction. I don’t even want to think about how badly larger entities are dealing with it.
docmars 5y ago
I can't reply to or upvote ~stalkersyndrome's response to add my applause, but I'll do it here instead. He's right. It's inconvenient, and people don't like reading that, which is why it's downvoted to dead, but alas, this is the same reason people avoid getting counseling, because they haven't worked up the courage to be honest with themselves yet. We'll get there, I hope. https://news.ycombinator.com/item?id=28156934 https://news.ycombinator.com/item?id=28156934

nzealand 5y ago

> 2. Anyway, is that legal ? Even if some crazy store material on his Apple hardware isn't that illegal search non usable in law courts ? Yes, it's considered legal. Apple reviews the content first. Courts say this means it is not an illegal government search. It's a search by a private party, who then manually decides to notify the government.

cookiengineer 5y ago
> Yes, it's considered legal. No, it's not. At least not here in Germany. By law, even police officers are not allowed to look at child porn. The only institution explicitely allowed to do so is the BSI. The rest of the population implicitely incriminates themselves when they look at (not own) child porn, including Apple's legal entity or employees. See [1] for 184b Strafgesetzbuch I'm trying to point out that with this action Apple bluntly decided to ignore a whole lot of countries and their federal laws, which is not something I would embrace - even when they had good intentions. [1] https://www.gesetze-im-internet.de/stgb/__184b.html https://www.gesetze-im-internet.de/stgb/__184b.html
- nzealand 5y ago
  I think they are only planning to scan US phones, so I am not sure how German law applies here.