5 ms·

Despite that Apple scanning our images is a horrible privacy practice, I don't get why 𝚜̶𝚘̶ ̶𝚖̶𝚊̶𝚗̶𝚢̶ some people think this is an ineffective idea. Sure

Despite that Apple scanning our images is a horrible privacy practice, I don't get why 𝚜̶𝚘̶ ̶𝚖̶𝚊̶𝚗̶𝚢̶ some people think this is an ineffective idea. Surely you can easily fabricate innocent images whose NeuralHash matches the database. But in what way are you going to send them to victims and convince them to save them to their photo library? The moment you send it via WhatsApp FB will stop you because (they think) it is a problematic image. And Even if the image did land, it has to look like some cats and dogs or the receiver will just ignore. (Even worse, the receiver may report you.) And even if your image does look like cats and dogs, it has to pass another automatic test at the server side that uses another obfuscated, constantly-updating algorithm. After that, even more tests if Apple really wants to. That means your image needs to collide ≥ three times, one open, one obfuscated, and one Turing. Gmail scans your attachments and most people are cool with it. I highly doubt that Apple has any reason to withdraw this.

istingray 5y ago
I believe being downvoted because this is thoroughly covered in the thread. Suggest you read it all again.
- ThinBold 5y ago
  I did my best skimming comments the algorithm showed me, including related posts' comments. But man am I bad at reading comprehension. If I miss anything then surely I am willing to be corrected. But so far I don't see comments that show us how to penetrate the four-layer system (local hash check, semantic check by user, on-server hash check, and human reviewer).
  - istingray 5y ago
    Ah ok! Here's the relevant part of the thread for that. https://news.ycombinator.com/item?id=28229832 https://news.ycombinator.com/item?id=28229832
    - ThinBold 5y ago
      That NSFW picture is, in honor of Sean Lock, a challenging wank
copperx 5y ago
Almost nobody is arguing the effectiveness of the idea. That would be missing the point entirely.
- ThinBold 5y ago
  The first comment algorithm choses to show me contains substring "and this system is transparently broken". Isn't that a remark on the effectiveness of the idea or I misread? (In case you have problem with me saying "so many", that I can fix.)
  - sennight 5y ago
    > I don't get why 𝚜̶𝚘̶ ̶𝚖̶𝚊̶𝚗̶𝚢̶ some people think this is an ineffective idea. Because such architectural flaws become absolute train wrecks when scaled. Remember the Clipper Chip? This is like that: cryptographers pointing out fundamental flaws that may seem like minor issues to most of the users who were going to be compelled to use it - but at scale those flaws result in the direct opposite of the stated objectives. It feels weird having to explain scalability on HN... everyone here should know that if your little scheme is struggling pre-rollout then trying to power through will only magnify your troubles. So it is hard to account for that blind spot that defenders of this thing seem to have.
    - ThinBold 5y ago
      Unless Apple implements this with a backdoor on iCloud, the worst case scenario here is that they receive millions of false positives per day and terminate the program after two weeks. The scalability issue seems to work in our favor because, perhaps, the normal usage will overwhelm the human reviewers Apple prepared and we don't even need to send troll images. And for the entire time, our data remains untouched.
      - sennight 5y ago
        So Apple customers go from paying for a status symbol luxury item to paying for the privilege of participating in this, at best, pointless exercise wherein they rely upon the goodwill and competence of Apple's employees to not get them swatted. The mental gymnastics needed in order to guard one's ego on this issue can't be healthy either.
t-writescode 5y ago
Some people sync iMessages. That might even be the default.
asxd 5y ago
I really appreciate this comment, as anytime a new security issue creates a fuss, I feel like I'm the only one wondering what the real attack vector is. I'm genuinely glad people are so thoroughly investigating this new Apple policy, but at the same time I feel like I'm the only one dumb enough not to understand what I should be actually concerned about.
- zo1 5y ago
  It's all theater. We need to stop talking about it and get trusted security researchers picked at random to be deployed to Apple for an audit. They sign an NDA, they work in a clean room with no way to exfiltrate data, they get full access to all the algorithms, source code, trained networks, all test data, access to Apples infra to test as they please, etc. And then they need to be the definitive authority on this matter to give us info and suggestions. Not Apple, not PR departments, not the EU parliament, not the US gov. And most certainly not us.
undecisive 5y ago
It's a good question, and unfortunately you're probably right. It boils down to this: If you can prevent [some organisation] from potentially destroying civilisation, how much effort would be too much effort, and how much uncertainty is too much uncertainty? For most, there's a trade off. If someone believes that the technology is sufficient for any country to implement a brutal civil rights destruction campaign, and that this is 50% likely, and all you need to do is upload some harmless images to your icloud to thwart it, why wouldn't you? For example, maybe a certain political party could regain power in a massive way and start doing away with gay rights, locking up anyone with pictures of men kissing each other on their phones. Of course, other countries have other types of extremists that look like they might take over government, or have already taken over governments. In those countries, tools like this are already in place, and this new one could be very powerful. So if you could upload some innocuous images and save hundreds of lives in 20 or more countries in the world, even if you don't fear for your own country, would you? Assuming you said yes, the question is, could everyone who agrees give Apple enough false positives, force enough human moderators to have to inspect the images, that the whole scheme becomes financially un-viable on Apple's end? Of course, the only thing we could do here is slow this "progress" down. Maybe we can use that extra time to share the message that "this kind of technology is not ok" and that being naive about what this tech will be used for is almost guaranteed to kill more people than it saves. But while so few people are thinking of it in these terms, or if everyone believes the attempt to be futile, it can't work. It's like anything - by the time people realise there's a problem, it's often too late to fix the problem.
- ThinBold 5y ago
  save hundreds of lives, would you? Sure, why not. But let me ask some questions, because at this point I am not sure if people want Apple's system to be robust or jammable. If our fear is that Apple will tune the system to detect pictures of two men kissing, wouldn't an easily jammable system works in our favor because we can DDoS it or threaten to do so anytime we want?
  - undecisive 5y ago
    It's a tricky one, certainly. There is probably a graph somewhere showing how much effort to fix is too much effort, vs how much bad will in the community this project is inducing, vs how much value this project has. So no, Apple's system being jammable is a great "booby prize" right up to the point where Apple fix the algorithms, or the chinese government start reporting these false positives as bugs and saying that Apple must fix the bugs before their devices can be sold in China. And so, one has to assume that if the algorithms can be fixed, they will be fixed. If we can DDoS Apple's human checking capability while it's still young, we might be able to prevent more resource being sunk into it - though I agree, that's unlikely. If we can do all that AND make it clear that this is going to cause nothing but bad will, and if we can get enough governments to regulate against it, then there may be hope. So your question was, why do people think this software isn't going to work, and the answer is we really hope it isn't going to work and really hope that enough people get onboard with the efforts to subvert it and really hope that the message sinks in to Apple that this was a doomed project that they should abandon for good. But with enough money and time, there is no problem that can't be overcome. So even if it is a currently jammable child protection system, in the future is will almost definitely become a robust human rights violation system. In the space between those two points there is hope. Slim hope, but hope nonetheless.

istingray 5y ago

I believe being downvoted because this is thoroughly covered in the thread. Suggest you read it all again.

ThinBold 5y ago
I did my best skimming comments the algorithm showed me, including related posts' comments. But man am I bad at reading comprehension. If I miss anything then surely I am willing to be corrected. But so far I don't see comments that show us how to penetrate the four-layer system (local hash check, semantic check by user, on-server hash check, and human reviewer).
- istingray 5y ago
  Ah ok! Here's the relevant part of the thread for that. https://news.ycombinator.com/item?id=28229832 https://news.ycombinator.com/item?id=28229832
  - ThinBold 5y ago
    That NSFW picture is, in honor of Sean Lock, a challenging wank

copperx 5y ago

Almost nobody is arguing the effectiveness of the idea. That would be missing the point entirely.

ThinBold 5y ago
The first comment algorithm choses to show me contains substring "and this system is transparently broken". Isn't that a remark on the effectiveness of the idea or I misread? (In case you have problem with me saying "so many", that I can fix.)
- sennight 5y ago
  > I don't get why 𝚜̶𝚘̶ ̶𝚖̶𝚊̶𝚗̶𝚢̶ some people think this is an ineffective idea. Because such architectural flaws become absolute train wrecks when scaled. Remember the Clipper Chip? This is like that: cryptographers pointing out fundamental flaws that may seem like minor issues to most of the users who were going to be compelled to use it - but at scale those flaws result in the direct opposite of the stated objectives. It feels weird having to explain scalability on HN... everyone here should know that if your little scheme is struggling pre-rollout then trying to power through will only magnify your troubles. So it is hard to account for that blind spot that defenders of this thing seem to have.
  - ThinBold 5y ago
    Unless Apple implements this with a backdoor on iCloud, the worst case scenario here is that they receive millions of false positives per day and terminate the program after two weeks. The scalability issue seems to work in our favor because, perhaps, the normal usage will overwhelm the human reviewers Apple prepared and we don't even need to send troll images. And for the entire time, our data remains untouched.
    - sennight 5y ago
      So Apple customers go from paying for a status symbol luxury item to paying for the privilege of participating in this, at best, pointless exercise wherein they rely upon the goodwill and competence of Apple's employees to not get them swatted. The mental gymnastics needed in order to guard one's ego on this issue can't be healthy either.

t-writescode 5y ago

Some people sync iMessages. That might even be the default.

asxd 5y ago

I really appreciate this comment, as anytime a new security issue creates a fuss, I feel like I'm the only one wondering what the real attack vector is. I'm genuinely glad people are so thoroughly investigating this new Apple policy, but at the same time I feel like I'm the only one dumb enough not to understand what I should be actually concerned about.

zo1 5y ago
It's all theater. We need to stop talking about it and get trusted security researchers picked at random to be deployed to Apple for an audit. They sign an NDA, they work in a clean room with no way to exfiltrate data, they get full access to all the algorithms, source code, trained networks, all test data, access to Apples infra to test as they please, etc. And then they need to be the definitive authority on this matter to give us info and suggestions. Not Apple, not PR departments, not the EU parliament, not the US gov. And most certainly not us.

undecisive 5y ago

It's a good question, and unfortunately you're probably right. It boils down to this: If you can prevent [some organisation] from potentially destroying civilisation, how much effort would be too much effort, and how much uncertainty is too much uncertainty? For most, there's a trade off. If someone believes that the technology is sufficient for any country to implement a brutal civil rights destruction campaign, and that this is 50% likely, and all you need to do is upload some harmless images to your icloud to thwart it, why wouldn't you? For example, maybe a certain political party could regain power in a massive way and start doing away with gay rights, locking up anyone with pictures of men kissing each other on their phones. Of course, other countries have other types of extremists that look like they might take over government, or have already taken over governments. In those countries, tools like this are already in place, and this new one could be very powerful. So if you could upload some innocuous images and save hundreds of lives in 20 or more countries in the world, even if you don't fear for your own country, would you? Assuming you said yes, the question is, could everyone who agrees give Apple enough false positives, force enough human moderators to have to inspect the images, that the whole scheme becomes financially un-viable on Apple's end? Of course, the only thing we could do here is slow this "progress" down. Maybe we can use that extra time to share the message that "this kind of technology is not ok" and that being naive about what this tech will be used for is almost guaranteed to kill more people than it saves. But while so few people are thinking of it in these terms, or if everyone believes the attempt to be futile, it can't work. It's like anything - by the time people realise there's a problem, it's often too late to fix the problem.

ThinBold 5y ago
save hundreds of lives, would you? Sure, why not. But let me ask some questions, because at this point I am not sure if people want Apple's system to be robust or jammable. If our fear is that Apple will tune the system to detect pictures of two men kissing, wouldn't an easily jammable system works in our favor because we can DDoS it or threaten to do so anytime we want?
- undecisive 5y ago
  It's a tricky one, certainly. There is probably a graph somewhere showing how much effort to fix is too much effort, vs how much bad will in the community this project is inducing, vs how much value this project has. So no, Apple's system being jammable is a great "booby prize" right up to the point where Apple fix the algorithms, or the chinese government start reporting these false positives as bugs and saying that Apple must fix the bugs before their devices can be sold in China. And so, one has to assume that if the algorithms can be fixed, they will be fixed. If we can DDoS Apple's human checking capability while it's still young, we might be able to prevent more resource being sunk into it - though I agree, that's unlikely. If we can do all that AND make it clear that this is going to cause nothing but bad will, and if we can get enough governments to regulate against it, then there may be hope. So your question was, why do people think this software isn't going to work, and the answer is we really hope it isn't going to work and really hope that enough people get onboard with the efforts to subvert it and really hope that the message sinks in to Apple that this was a doomed project that they should abandon for good. But with enough money and time, there is no problem that can't be overcome. So even if it is a currently jammable child protection system, in the future is will almost definitely become a robust human rights violation system. In the space between those two points there is hope. Slim hope, but hope nonetheless.