7 ms·

So if I understand correctly, they want to scan all your photos, stored on your private phone, that you paid for, and they want to check if any of the hashes ar

So if I understand correctly, they want to scan all your photos, stored on your private phone, that you paid for, and they want to check if any of the hashes are the same as hashes of child porn? So... all your hashes will be uploaded to the cloud? How do you prevent them from scanning other stuff (memes, leaked documents, trump-fights-cnn-gif,... to profile the users)? Or will a huge hash database of child porn hashes be downloaded to the phone? Honestly, i think it's one more abuse of terrorism/child porn to take away privacy of people, and mark all oposing the law as terrorists/pedos. ...also, as in the thread from the original url, making false positives and spreading them around (think 4chan mass e-mailing stuff) might cause a lot of problems too.

willis936 5y ago
>So... all your hashes will be uploaded to the cloud? That isn't how I interpret "client-side". The privacy implications are far more subtle.
- dannyw 5y ago
  It's still really, really bad. It always starts with child porn, and in a few years the offline Notes app will be phoning home if you write speech criticising the government in China. This technology inevitably leads to the sueveillance, suppression and murder of activists and journalists. It always starts with protecting the kids or terrorism. Perceptual hashes like what Apple is using are already used in WeChat to detect memes that critique the CCP. What happens on local end user devices must be off limits. It is unacceptable that Apple is actively implementing machine learning systems that surveil and snitch on local content.
  - srswtf123 5y ago
    I agree with you 100% — the only solution I’ve found workable is limiting my use of the technology itself as much as possible.
    - ajsnigrutin 5y ago
      Or regulation saying that this is illegal, because it invades users privacy too much.
      - bigiain 5y ago
        Which'd be fine if we had one global government with world wide jurisdiction. Or technology choices from companies which couldn't be pressured by governments outside your personal regulation jurisdiction. I wouldn't hold your breath waiting for those regulations to become law in, say, Chine or Turkey or Saudi Arabia. I'd bet even Israel won't pass them, surely NSO have enough political lobbying swing (and probably also suitable blackmail material on sitting politicians).
        aaronmdjones 5y ago
        They sell software designed to break into people's phones to oppressive regimes all over the planet. They definitely have the capability and requisite lack of ethics to compromise their own politicians; there's no probably about it.
        TeMPOraL 5y ago
        But we don't have a global government, so the next best thing is for individual countries to pass such regulation, which would prevent products violating privacy like this from being offered and sold in those countries. Think of GDPR, which is essentially each member of the EU saying in unison, "your product/service must comply with these data protection laws, or you can't legally do business with any of our citizens". Come to think of it, I wonder if this Apple thing would even fly under GDPR?
        ben_w 5y ago
        > Come to think of it, I wonder if this Apple thing would even fly under GDPR? Possibly? I’m not a lawyer, but if this is about compliance with a legal obligation, and they’re under that category of pressure? I think GDPR would allow that? Certainly seems more likely allowed than the stuff Facebook complained Apple was preventing them from doing.
  - judge2020 5y ago
    > It's still really, really bad. The OP still addresses the inaccurate statement (presented in the form of a question for plausible deniability).
  - toxik 5y ago
    > in a few years the offline Notes app will be phoning home if you write speech criticising the government in China. A totalitarian autocracy like China does not need this technology to search for wrongspeech, sadly. You are of course aware that all Chinese iCloud users get their data stored in a special set of datacenters that Apple actually doesn't control.
    - ajsnigrutin 5y ago
      The problem is, that this will be done in other, (currently) freer countries. Eg. Reddit has removed a lot of anti-china posts in the last few years. And of course, local leaders will use this to find anti-local-leader stuff on their citizens phones too.
    - consumer451 5y ago
      This seems to be done voluntarily by NVIDIA, at least partially. While in Seattle I set up geo-blocking on my LAN as an experiment. Later when I tried to create an NVIDIA account I couldn't because it was attempting to store my PII at nvidia.cn. When I changed the url to nvidia.com everything worked just fine. I've always wondered what non-evil reasons one could use to explain that choice by NVIDIA. Ping was at least 2x longer to .cn.
zepto 5y ago
No, your hashes are not uploaded to the cloud, yes, hashes are downloaded to your phone. Yes, it will be interesting to see if it gets spammed with false positives, although it seems as though that can easily be identified silently to the user.
- google234123 5y ago
  How hard would it be to create a valid image that matches some 128bit hahs
  - zepto 5y ago
    If it’s a cryptographic hash - very hard.
    - azalemeth 5y ago
      But the probability is still not zero, and the number of iPhones in the world is large. A hash collision is possible, however unlikely.
      - zepto 5y ago
        The probability of Apple and all its devices winking out of existence due to quantum fluctuations is not zero. ‘Not zero’ is effectively zero if the number is small enough.
      - ben_w 5y ago
        128 bit hashes are “expect your first collision when each human buys 2305 iPhones, all with one terabyte storage, and then fills them up with photos that are an average of 1MB in file size” https://en.wikipedia.org/wiki/Birthday_attack https://en.wikipedia.org/wiki/Birthday_attack http://www.wolframalpha.com/input/?i=2%5E64%20%2F%20%288e9%2A1e6%29 http://www.wolframalpha.com/input/?i=2%5E64%20%2F%20%288e9%2...
    - account42 5y ago
      The tweets talk about perceptual hashes, not cryptographic hashes.
echelon 5y ago
That document you downloaded that is critical of the party will land you and your family in jail. Enjoy your iPhone. Seriously, folks, we shouldn't celebrate Apple's death grip over their platform. It's dangerous for all of us. The more of you that use it, the more it creates a sort of "anti-herd immunity" towards totalitarian control. Apple talks "privacy", but jfc they're nothing of the sort. Apple gives zero shits about your privacy. They're staking more ground against Facebook and Google, trying to take their beachheads. You're just a pawn in the game for long term control. Apple cares just as much for your privacy as they do your "freedom" to run your own (un-taxed) software or repair your devices (for cheaper). And after Tim Cook is replaced with a new regime, you'll be powerless to stop the further erosion of your liberties. It'll be too late. Stop. Using. Apple.
- bigiain 5y ago
  > Stop. Using. Apple. But is there a realistically better alternative? Pinephone with a personally audited Linux distro? A jailbroken Android device with a non-stock firmware that you built yourself? A homebuilt RaspberryPi based device? A paper notepad and a film camera and an out of print street map?
  - echelon 5y ago
    Not really, and I'm not going to sway anyone deeply into the ecosystem. My hope is that those of you that share my viewpoint will call your legislators and demand regulations or a break up. There are forces of good within the DOJ that are putting together an antitrust case against Apple, and the more of us that lend our voices, the louder and more compelling the argument. The DOJ is really the last lever we have, and that's pretty good measure for the power Apple wields.
    - dylan604 5y ago
      Wouldn't you think Apple is doing something like this at the behest of DoJ?
    - wmf 5y ago
bigiain 5y ago
> and they want to check if any of the hashes are the same as hashes of child porn? ... without any technical guarantee or auditability that any of the hashes they're alerting on are actually of child porn. How much would you bet against law enforcement to abuse their ability to use this, and add hashes to find out who's got anti government memes or police committing murder images on their phones? And that's just in "there land of the free", how much worst will the abuse of this be in countries who, say, bonesaw journalists to pieces while they are alive?
- hda2 5y ago
  I remember the story where some large gaming company permanently banned someone because they had a file with a hash that matched a "hacking tool". Turns out the hash was for an empty file. This will end badly for humanity.
  - ddtaylor 5y ago
    Path of Exile has in the past looked at DNS cache information on the system and will ban players who have contacted known bad sites.
- bagacrap 5y ago
  they don't check the unhashed bytes against the child porn bytes after a hash match?
0xy 5y ago
That's the stated purpose, but keep in mind that these databases (NCMEC's in particular, which is used by FB and very likely Apple) contain legal images that are NOT child porn.
- SrslyJosh 5y ago
  Source for that info?
  - mousepilot 5y ago
    Think of it this way, take a regular, legal set of adult pornographic pictures. While legal, we'd still classify this set of pictures as known porn if we were tracking it. Now the first few images might be the model completely clothed and not even be porn, maybe there's a picture of her lounging around a pool, then another picture of the pool itself. Still its part of a set of pictures that is known porn. Heck most porn starts off with actors being clothed (so I hear lol).
onetimeusename 5y ago
I agree, I would add that people have generated legal images that match the hashes. So I want to ask what happens if you have a photo that is falsely identified as one in question and then an automated mechanism flags you and reports you to the FBI without you even knowing. Can they access your phone at that point to investigate? Would they come to your office and ask about it? Would that be enough evidence to request a wiretap or warrant? Would they alert your neighbors? How do you clear your name after that happens? edits: yes, the hash database is downloaded to the phone and matches are checked on your phone. Another point is that these photos used to generate the fingerprints are really legal black holes that the public is not allowed to inspect I assume. No one wants to be involved in looking at them, no one wants to be known as someone who looks at them. It could even be legally dangerous requesting to find out what has been put into the image database I assume.
- SilverRed 5y ago
  >I would add that people have generated legal images that match the hashes. That seems like a realistic attack. Since the hash list is public (has to be for client side scanning), you could likely set your computer to grind out a matching image hash but of some meme which you then distribute.
  - google234123 5y ago
    Might be hard if they use a huge hash.
    - SilverRed 5y ago
      One thing to note is these are not typical cryptographic hashes because they have to be able to find recompressed/cropped/edited versions as well. Perhaps a hash is not an accurate way to describe it. There have been a number of cases where people have found ways to trick CV programs in to seeing something that no human would ever see. If you were sufficiently malicious I imagine it would be possible to do with this system as well.
  - shuckles 5y ago
sebyx07 5y ago
your phone phones the phone manufacturer to phone the police to iphone you
lynndotpy 5y ago
No need to upload every hash or download a huge database with very hash. If I were building this system, I'd make a bloom-filter of hashes. This means O(1) space and time checking of a hash match, with a risk of false positives. I'd only send hashes to check against a full database.
- deleted 5y ago
  [deleted]
headmelted 5y ago
This is the big one right here. A malware will definitely be created, almost immediately, that will download files that are intentionally made to match CP - either for the purposes of extortion or just watching the world burn. I'm usually sticking my neck out in defence of more government access to private media than most on HN because of the need to stop CP, but this plan is so naive, and so incredibly irresponsible, that I can't see how anyone with any idea of how easy it would be to manipulate would ever stand behind it.
- dredmorbius 5y ago
  Signal famously implemented, or at least claimed to implement, a rather similar-sounding feature as a countermeasure against the Cellebrite forensics tool: https://signal.org/blog/cellebrite-vulnerabilities/ https://signal.org/blog/cellebrite-vulnerabilities/
  - nceqs3 5y ago
    What is file that they have installed?
    - CRConrad 5y ago
      If they told that, people would (try to) remove it. The whole point is that you can't know which (if any) of the hundreds of thousands of files on your device it is. So they aren't telling. Could be they have (or at least claim to have) written their system so it chooses files at random; I think that's what I would do (or claim to have done).
- andrewmd5 5y ago
  If you can recreate a file so it’s hash matches known CP then that file is CP my dude. The probability of just two hashes accidentally colliding is approximately: 4.3*10-60 Even if you do a content aware hash where you break the file into chunks and hash each chunk, you still wouldn’t be able to magically recreate the hash of a CP file without also producing part of the CP.
deleted 5y ago
[deleted]
WhyNotHugo 5y ago
> So... all your hashes will be uploaded to the cloud? No, it'll be done on-device. > How do you prevent them from scanning other stuff (memes, leaked documents, trump-fights-cnn-gif Nothing. Given that it's only done on their closed-source messaging platform though, nothing is preventing them from reading your messages already. But yes, it could potentially be used to detect images that the current political party doesn't like.
SergeAx 5y ago
No-no-no. It's not your phone. If it was your phone - you would have a root access to it. It's their phone. And it's their photos. They just don't like when there's something illegal on their photos, so they will scan it, just in case.

willis936 5y ago

>So... all your hashes will be uploaded to the cloud? That isn't how I interpret "client-side". The privacy implications are far more subtle.

dannyw 5y ago
It's still really, really bad. It always starts with child porn, and in a few years the offline Notes app will be phoning home if you write speech criticising the government in China. This technology inevitably leads to the sueveillance, suppression and murder of activists and journalists. It always starts with protecting the kids or terrorism. Perceptual hashes like what Apple is using are already used in WeChat to detect memes that critique the CCP. What happens on local end user devices must be off limits. It is unacceptable that Apple is actively implementing machine learning systems that surveil and snitch on local content.
- srswtf123 5y ago
  I agree with you 100% — the only solution I’ve found workable is limiting my use of the technology itself as much as possible.
  - ajsnigrutin 5y ago
    Or regulation saying that this is illegal, because it invades users privacy too much.
    - bigiain 5y ago
      Which'd be fine if we had one global government with world wide jurisdiction. Or technology choices from companies which couldn't be pressured by governments outside your personal regulation jurisdiction. I wouldn't hold your breath waiting for those regulations to become law in, say, Chine or Turkey or Saudi Arabia. I'd bet even Israel won't pass them, surely NSO have enough political lobbying swing (and probably also suitable blackmail material on sitting politicians).
      - aaronmdjones 5y ago
        They sell software designed to break into people's phones to oppressive regimes all over the planet. They definitely have the capability and requisite lack of ethics to compromise their own politicians; there's no probably about it.
      - TeMPOraL 5y ago
        But we don't have a global government, so the next best thing is for individual countries to pass such regulation, which would prevent products violating privacy like this from being offered and sold in those countries. Think of GDPR, which is essentially each member of the EU saying in unison, "your product/service must comply with these data protection laws, or you can't legally do business with any of our citizens". Come to think of it, I wonder if this Apple thing would even fly under GDPR?
        ben_w 5y ago
        > Come to think of it, I wonder if this Apple thing would even fly under GDPR? Possibly? I’m not a lawyer, but if this is about compliance with a legal obligation, and they’re under that category of pressure? I think GDPR would allow that? Certainly seems more likely allowed than the stuff Facebook complained Apple was preventing them from doing.
- judge2020 5y ago
  > It's still really, really bad. The OP still addresses the inaccurate statement (presented in the form of a question for plausible deniability).
- toxik 5y ago
  > in a few years the offline Notes app will be phoning home if you write speech criticising the government in China. A totalitarian autocracy like China does not need this technology to search for wrongspeech, sadly. You are of course aware that all Chinese iCloud users get their data stored in a special set of datacenters that Apple actually doesn't control.
  - ajsnigrutin 5y ago
    The problem is, that this will be done in other, (currently) freer countries. Eg. Reddit has removed a lot of anti-china posts in the last few years. And of course, local leaders will use this to find anti-local-leader stuff on their citizens phones too.
  - consumer451 5y ago
    This seems to be done voluntarily by NVIDIA, at least partially. While in Seattle I set up geo-blocking on my LAN as an experiment. Later when I tried to create an NVIDIA account I couldn't because it was attempting to store my PII at nvidia.cn. When I changed the url to nvidia.com everything worked just fine. I've always wondered what non-evil reasons one could use to explain that choice by NVIDIA. Ping was at least 2x longer to .cn.

zepto 5y ago

No, your hashes are not uploaded to the cloud, yes, hashes are downloaded to your phone. Yes, it will be interesting to see if it gets spammed with false positives, although it seems as though that can easily be identified silently to the user.

google234123 5y ago
How hard would it be to create a valid image that matches some 128bit hahs
- zepto 5y ago
  If it’s a cryptographic hash - very hard.
  - azalemeth 5y ago
    But the probability is still not zero, and the number of iPhones in the world is large. A hash collision is possible, however unlikely.
    - zepto 5y ago
      The probability of Apple and all its devices winking out of existence due to quantum fluctuations is not zero. ‘Not zero’ is effectively zero if the number is small enough.
    - ben_w 5y ago
      128 bit hashes are “expect your first collision when each human buys 2305 iPhones, all with one terabyte storage, and then fills them up with photos that are an average of 1MB in file size” https://en.wikipedia.org/wiki/Birthday_attack https://en.wikipedia.org/wiki/Birthday_attack http://www.wolframalpha.com/input/?i=2%5E64%20%2F%20%288e9%2A1e6%29 http://www.wolframalpha.com/input/?i=2%5E64%20%2F%20%288e9%2...
  - account42 5y ago
    The tweets talk about perceptual hashes, not cryptographic hashes.

echelon 5y ago

That document you downloaded that is critical of the party will land you and your family in jail. Enjoy your iPhone. Seriously, folks, we shouldn't celebrate Apple's death grip over their platform. It's dangerous for all of us. The more of you that use it, the more it creates a sort of "anti-herd immunity" towards totalitarian control. Apple talks "privacy", but jfc they're nothing of the sort. Apple gives zero shits about your privacy. They're staking more ground against Facebook and Google, trying to take their beachheads. You're just a pawn in the game for long term control. Apple cares just as much for your privacy as they do your "freedom" to run your own (un-taxed) software or repair your devices (for cheaper). And after Tim Cook is replaced with a new regime, you'll be powerless to stop the further erosion of your liberties. It'll be too late. Stop. Using. Apple.

bigiain 5y ago
> Stop. Using. Apple. But is there a realistically better alternative? Pinephone with a personally audited Linux distro? A jailbroken Android device with a non-stock firmware that you built yourself? A homebuilt RaspberryPi based device? A paper notepad and a film camera and an out of print street map?
- echelon 5y ago
  Not really, and I'm not going to sway anyone deeply into the ecosystem. My hope is that those of you that share my viewpoint will call your legislators and demand regulations or a break up. There are forces of good within the DOJ that are putting together an antitrust case against Apple, and the more of us that lend our voices, the louder and more compelling the argument. The DOJ is really the last lever we have, and that's pretty good measure for the power Apple wields.
  - dylan604 5y ago
    Wouldn't you think Apple is doing something like this at the behest of DoJ?
  - wmf 5y ago

bigiain 5y ago

> and they want to check if any of the hashes are the same as hashes of child porn? ... without any technical guarantee or auditability that any of the hashes they're alerting on are actually of child porn. How much would you bet against law enforcement to abuse their ability to use this, and add hashes to find out who's got anti government memes or police committing murder images on their phones? And that's just in "there land of the free", how much worst will the abuse of this be in countries who, say, bonesaw journalists to pieces while they are alive?

hda2 5y ago
I remember the story where some large gaming company permanently banned someone because they had a file with a hash that matched a "hacking tool". Turns out the hash was for an empty file. This will end badly for humanity.
- ddtaylor 5y ago
  Path of Exile has in the past looked at DNS cache information on the system and will ban players who have contacted known bad sites.
bagacrap 5y ago
they don't check the unhashed bytes against the child porn bytes after a hash match?

0xy 5y ago

That's the stated purpose, but keep in mind that these databases (NCMEC's in particular, which is used by FB and very likely Apple) contain legal images that are NOT child porn.

SrslyJosh 5y ago
Source for that info?
- mousepilot 5y ago
  Think of it this way, take a regular, legal set of adult pornographic pictures. While legal, we'd still classify this set of pictures as known porn if we were tracking it. Now the first few images might be the model completely clothed and not even be porn, maybe there's a picture of her lounging around a pool, then another picture of the pool itself. Still its part of a set of pictures that is known porn. Heck most porn starts off with actors being clothed (so I hear lol).

onetimeusename 5y ago

I agree, I would add that people have generated legal images that match the hashes. So I want to ask what happens if you have a photo that is falsely identified as one in question and then an automated mechanism flags you and reports you to the FBI without you even knowing. Can they access your phone at that point to investigate? Would they come to your office and ask about it? Would that be enough evidence to request a wiretap or warrant? Would they alert your neighbors? How do you clear your name after that happens? edits: yes, the hash database is downloaded to the phone and matches are checked on your phone. Another point is that these photos used to generate the fingerprints are really legal black holes that the public is not allowed to inspect I assume. No one wants to be involved in looking at them, no one wants to be known as someone who looks at them. It could even be legally dangerous requesting to find out what has been put into the image database I assume.

SilverRed 5y ago
>I would add that people have generated legal images that match the hashes. That seems like a realistic attack. Since the hash list is public (has to be for client side scanning), you could likely set your computer to grind out a matching image hash but of some meme which you then distribute.
- google234123 5y ago
  Might be hard if they use a huge hash.
  - SilverRed 5y ago
    One thing to note is these are not typical cryptographic hashes because they have to be able to find recompressed/cropped/edited versions as well. Perhaps a hash is not an accurate way to describe it. There have been a number of cases where people have found ways to trick CV programs in to seeing something that no human would ever see. If you were sufficiently malicious I imagine it would be possible to do with this system as well.
- shuckles 5y ago

sebyx07 5y ago

your phone phones the phone manufacturer to phone the police to iphone you

lynndotpy 5y ago

No need to upload every hash or download a huge database with very hash. If I were building this system, I'd make a bloom-filter of hashes. This means O(1) space and time checking of a hash match, with a risk of false positives. I'd only send hashes to check against a full database.

deleted 5y ago
[deleted]

headmelted 5y ago

This is the big one right here. A malware will definitely be created, almost immediately, that will download files that are intentionally made to match CP - either for the purposes of extortion or just watching the world burn. I'm usually sticking my neck out in defence of more government access to private media than most on HN because of the need to stop CP, but this plan is so naive, and so incredibly irresponsible, that I can't see how anyone with any idea of how easy it would be to manipulate would ever stand behind it.

dredmorbius 5y ago
Signal famously implemented, or at least claimed to implement, a rather similar-sounding feature as a countermeasure against the Cellebrite forensics tool: https://signal.org/blog/cellebrite-vulnerabilities/ https://signal.org/blog/cellebrite-vulnerabilities/
- nceqs3 5y ago
  What is file that they have installed?
  - CRConrad 5y ago
    If they told that, people would (try to) remove it. The whole point is that you can't know which (if any) of the hundreds of thousands of files on your device it is. So they aren't telling. Could be they have (or at least claim to have) written their system so it chooses files at random; I think that's what I would do (or claim to have done).
andrewmd5 5y ago
If you can recreate a file so it’s hash matches known CP then that file is CP my dude. The probability of just two hashes accidentally colliding is approximately: 4.3*10-60 Even if you do a content aware hash where you break the file into chunks and hash each chunk, you still wouldn’t be able to magically recreate the hash of a CP file without also producing part of the CP.

deleted 5y ago

[deleted]

WhyNotHugo 5y ago

> So... all your hashes will be uploaded to the cloud? No, it'll be done on-device. > How do you prevent them from scanning other stuff (memes, leaked documents, trump-fights-cnn-gif Nothing. Given that it's only done on their closed-source messaging platform though, nothing is preventing them from reading your messages already. But yes, it could potentially be used to detect images that the current political party doesn't like.

SergeAx 5y ago

No-no-no. It's not your phone. If it was your phone - you would have a root access to it. It's their phone. And it's their photos. They just don't like when there's something illegal on their photos, so they will scan it, just in case.