67 ms·

One Bad Apple

• initplus 5y ago
  This reads like a failure of the NCMEC, and the legal system surrounding it. It is insane that using perceptual hashes is likely illegal. As the hashes are actually somewhat reversible and so possession of the hash is a criminal offence. It just shows how twisted up in itself the law is in this area. One independent image analysis service should not be beating reporting rates of major service providers. And NCMEC should not be acting like detection is a trade secret. Wider detection and reporting is the goal. And the law as setup prevents developing detection methods. You cannot legally check the results of your detection (which Apple are doing), as that involves transmitting the content to someone other than the NCMEC!
  • SV_BubbleTime 5y ago
    Doesn’t seem like it was an accident, rather protectionism written long ago to make sure they were the only “game” in town.
  • caeril 5y ago
    > This reads like a failure of the NCMEC Yes, it's a "failure" of a "private" Non(lol, technically, wink wink) Governmental Organization who works extremely closely with the FBI to put their camel-shaped nose under the very tent that the FBI happens to have been trying to breach for 20+ years. Come on. It's beyond gullibility, at this point, to believe that NCMEC isn't an arm of the Feeb. Specifically, it's an arm that isn't required to comply with FOIA requests, which is particularly convenient. Two years. At the current rate, you have approximately two years until the Feeb have full access to your iDevice. Though, I will admit, Apple's development of the SEP, their high-priced bug bounties, and their convincing play-acting at defying the FBI after the San Bernardino case definitely had me fooled. We probably should have been more keen after they failed to close the bugs that GreyKey et. al. exploited. But now we know. Everything they gave to China, they will give doubly so to their own corporate domicile.
  • kelnos 5y ago
    > And NCMEC should not be acting like detection is a trade secret. It feels to me like they want to hide their detection algorithms so people don't find out how bad they are.
• jolux 5y ago
  > To reiterate: scanning your device is not a privacy risk, but copying files from your device without any notice is definitely a privacy issue. Not a lawyer, but I believe this part about legality is inaccurate, because they aren’t copying your photos without notice. The feature is not harvesting suspect photos from a device, it is attaching data to all photos before they are uploaded to Apple’s servers. If you’re not using iCloud Photos, the feature will not be activated. Furthermore, they’re not knowingly transferring CSAM, because the system is designed only to notify them when a certain “threshold” of suspect images has been crossed. In this way it’s identical in practice to what Google and Facebook are already doing with photos that end up on their servers, they just run the check before the upload instead of after. I certainly have reservations about their technique here, but this argument doesn’t add up to me.
  • websites2023 5y ago
    This basic implementation fact has been misrepresented over and over and over again. Does anyone read anymore? I’m starting to get really concerned. The hacker community is where I’ve turned to be more informed, away from the clickbait. But I’m being let down.
    • dmitriid 5y ago
      What does "manual review" mean then and how are those images reported?
      • shuckles 5y ago
        Before: you would upload images to iCloud Photos. Apple can access your images in iCloud Photos, but it does not. Now: You upload images to iCloud Photos. When doing so, your device also uploads a separate safety voucher for the image. If there are enough vouchers for CSAM matched images in your library, Apple gains the ability to access the data in the vouchers for images matching CSAM. One of the data elements in the voucher is an “image derivative” (probably a thumbnail) which is manually reviewed. If the image derivative also looks like CSAM, Apple files a report with NCMEC’s CyberTip line. Apple can (for now) access the image you stored in iCloud, but it does not. All the data it needs is in the safety voucher. Lot of words spilled on this topic, yet I’d be surprised if a majority of people are even aware of these basic facts about the system operation.
• NKosmatos 5y ago
  Really nice explanation from someone who knows a thing or two about images/photos (Dr. Neal Krawetz is the creator of https://fotoforensics.com https://fotoforensics.com and specializes in computer forensics).
  • valparaiso 5y ago
    He wrongly interpreted CSAM scanning. He said that Apple will scan your photos and if finds something, it will send photo to Apple. Which is absolutely not how it works. Photo is only scanned before uploading to iCloud Photos. Apple already confirmed it to iMore and it’s clearly stated in Apple papers from press-release.
    • GauntletWizard 5y ago
      You're very clearly missing the forest for the trees. Right before uploading to icloud, "Apple will scan your photos and if finds something, it will send photo to Apple." This process is automated and turned on on most iPhones. Most iPhones will have automatic photo upload to icloud enabled, and that's when this scanning takes place.
      • 8bitsrule 5y ago
        "The problem with AI is that you don't know what attributes it finds important.... It determined that ... a guy with long hair is female." Lots of people had this problem back in the 60s. Funny - except that some guys were jailed or worse because of it.
      • judge2020 5y ago
        > Most iPhones will have automatic photo upload to icloud enabled, I highly doubt that with how each photo is 10+MB and Apple only gives an abysmal 5GB of free iCloud storage, which includes everything else. My iphone 6s backups got to 4.5gb on their own years ago, turning off photo storage was the only way to avoid paying for iCloud storage outright.
    • zionic 5y ago
• tgsovlerkhgsel 5y ago
  NCMEC has essentially shows that they have zero regard for privacy and called all privacy activists "screeching voices of the minority". At the same time, they're at the center point of a highly opaque, entrenched (often legally mandated) censorhip infrastructure that can and will get accounts shut down irrecoverably and possibly people's homes raided, on questionable data: In one of the previous discussions, I've seen claims about the NCMEC database containing a lot of harmless pictures misclassified as CSAM. This post confirms this again (ctrl-f "macaque") It also seems like the PhotoDNA hash algorithm is problematic (to the point where it may be possible to trigger false matches). Now NCMEC seem to be pushing for the development of a technology that would implant an informant in every single of our devices (mandating the inclusion of this technology is the logical next step that seems inevitable if Apple launches this). I'm surprised, and honestly disappointed, that the author seems to still play nice, instead of releasing the whitepaper. The NCMEC seems to have decided to position itself directly alongside other Enemies of the Internet, and while I can imagine that they're also doing a lot of important and good work, at this point, I don't think they're salvageable would like to see them disbanded. Really curious how this will play out. I expect attacks either sabotaging these scanning systems by flooding them with false positives, or exploiting them to get the accounts of your enemies shut down permanently by sending them a picture of a macaque.
  • lilSebastian 5y ago
    > It also seems like the PhotoDNA hash algorithm is problematic (to the point where it may be possible to trigger false matches A post here some days ago (since removed) linked to a google drive containing generated images (which displayed nonsense), the hashes of which matched those of genuine problems images.
  • gerash 5y ago
    I agree. NCMEC is the root cause. It's very much like the war on drugs. There are a lot of horrendous crimes out there and this level of overreach isn't deemed necessary for any of them. Child sex abuse seems to have become the excuse to create these heavy handed policies to terminate accounts with no recourse by Google and scanning locally stored photos by Apple. Even receiving a cartoon depicting child sex abuse can get you in trouble. Hopefully this will be a catalyst to reform the law.
• defaultname 5y ago
  Good article, however- "Due to how Apple handles cryptography (for your privacy), it is very hard (if not impossible) for them to access content in your iCloud account. Your content is encrypted in their cloud, and they don't have access. If Apple wants to crack down on CSAM, then they have to do it on your Apple device" I do not believe this is true. Maybe one day it will be true and Apple is planning for it, but right now iCloud service data is encrypted in the sense that they are stored encrypted at rest and in transit, however Apple holds the keys. We know this given that iCloud backups have been surrendered to authorities, and of course you can log into the web variants to view your photos, calendar, etc. Not to mention that Apple has purportedly been doing the same hash checking on their side for a couple of years. Thus far there has been no compelling answer as to why Apple needs to do this on device.
  • amelius 5y ago
    Probably because Apple wants to stay away from any suspicions that they sometimes actually use their keys to access private information.
  • tgsovlerkhgsel 5y ago
    > why Apple needs to do this on device Presumably to implement E2E encryption, while at the same time helping the NCMEC to push for legislation to make it illegal to offer E2E encryption without this backdoor. Apple users would be slightly better off than the status quo, but worse off than if Apple simply implemented real E2E without backdoors, and everyone else's privacy will be impacted by the backdoors that the NCMEC will likely push.
    • zepto 5y ago
      > make it illegal to offer E2E encryption without this backdoor. It isn’t a back door to E2E encryption. It can’t even be used to search for a specific image on a person’s device. It could be used possibly to find a collection of images that are not CSAM but are disliked by the state, assuming Apple is willing to enter into a conspiracy with NCMEC.
• Dah00n 5y ago
  >18 U.S.C. § 2258A is specific: the data can only be sent to NCMEC. (With 2258A, it is illegal for a service provider to turn over CP photos to the police or the FBI; you can only send it to NCMEC. Then NCMEC will contact the police or FBI.) What Apple has detailed is the intentional distribution (to Apple), collection (at Apple), and access (viewing at Apple) of material that they strongly have reason to believe is CSAM. As it was explained to me by my attorney, that is a felony. I'm not sure, after reading the article, who is/has the most insane system of Apple or NCMEC.
  • sbarre 5y ago
    This is the part that also caught my eye. Surely Apple's lawyers have also reviewed the same law, and if it's that clearly defined, how did they justify/explain their approach?
    • tdonovic 5y ago
      Because Apple (its employees) aren't actually viewing the images, nor transmitting them. They mention somewhere that it's a low res proxy of the image, or something similar.
      • heavyset_go 5y ago
        > They mention somewhere that it's a low res proxy of the image, or something similar. Perceptual hashes are just integer/byte encodings of images that were scaled down and had some transformations applied to them. If you convert a hash into an array of pixels and reverse the transformations, you'll get some of the original image that was scaled down and hashed.
        • xucheng 5y ago
          And if you then apply DLSS, you may even get the original image.
      • rdedev 5y ago
        But wouldn't a low res image of CP be still classified as CP ? I guess the manual verification will ve done as a joint venture by apple and the authorities
• netr0ute 5y ago
  To help fight back against false positives, why not just repeatedly trigger the code that sends the data to NCMEC (per the article's claimed legal requirements) and create a DoS attack?
  • deleted 5y ago
    [deleted]
  • cmsj 5y ago
    That code is not accessible to us. The report to NCMEC would be generated by Apple after they have manually reviewed the derived images included in the security vouchers your device submitted after it analysed your photos.
    • netr0ute 5y ago
      > That code is not accessible to us. Not to tools like IDA and Ghidra
      • kelnos 5y ago
        No, the tool that reports to NCMEC is on Apple employee workstations (or a private server). The stuff running in the iPhone essentially just flags things as possibly-CSAM, after which someone at Apple verifies it. Now, I suppose you could DDoS Apple's verification process. Either way, though, it's not like anyone who would do either of these things would win any points in the court of popular opinion. I can see the headlines now: "Hackers Disable Apple's CSAM Reporting Tools; Legitimate CSAM Reports Get Lost".
        • netr0ute 5y ago
          > "Hackers Disable Apple's CSAM Reporting Tools; Legitimate CSAM Reports Get Lost" What website are we on?
• antman 5y ago
  There are a lot of articles about Apples hadh algorithm and for me they are mostly irrelevant to the main problem. The main problem is that Apple has backdoored my device. More types of bad images or other files will be scanned since now apple does not have plausible deniablity to defend any of ghe government’x requests. In the future a false? positive that happened? to be of a political file that crept in the list can pin point people to the future dictator wannabe. It’s always about the children or terrorism.
  • baggy_trough 5y ago
    Exactly. The issue is that the iPhone is now a snitch AI for whatever purpose Apple deems fit.
    • sooheon 5y ago
      Drug dogs have a 50~80% false positive rate, because after they've been certified as capable of detecting drugs, they get that trained out of them the field to instead indicate whatever the handler wants indicated.
      • kfprt 5y ago
        Drug dogs are "probable cause on four legs". https://www.npr.org/2017/11/20/563889510/preventing-police-bias-when-handling-dogs-that-bite https://www.npr.org/2017/11/20/563889510/preventing-police-b...
  • ziml77 5y ago
    They could have done all that without telling you. And as long as the traffic was combined with normal traffic no one would ever notice (and in this case it would end up mixed with normal traffic since it only applies to images being uploaded to iCloud, so communication with Apples servers would be expected). What it looks like to me is that Apple is planning on releasing end-to-end encryption for iCloud. But they know that whenever E2EE comes up, people get mad that terrorists, child molesters, and mass shooters can hide their data and communications. Hell, they've been painted as the villain when they say they can't unlock iPhones for the FBI. This heads off those concerns for the most common out of those crimes.
• twoodfin 5y ago
  The author of this article purports to have done a ton of research into this system, but appears to have missed basic information that I’ve acquired from a few podcasts. Namely the “1-in-a-trillion” false positives per account per year is based on the likelihood of multiple photos matching the database (Apple doesn’t say how many are required to trip their manual screening threshold).
  • not2b 5y ago
    So they are assuming that the photos are independent. Common error with probabilistic reasoning. What if the same photo (or photos of the same scene taken seconds apart) gets uploaded more than once? The likelihood no longer multiplies. I don't believe the "one in a trillion" claim.
    • tehnub 5y ago
      They say that they address this issue through a mechanism outside of the cryptographic protocol, but don't say specifically how. The quote from the paper: A user might store multiple variants or near-duplicates of the same image on their client. In our language, this means that a single client could hold two triples (y,id,ad) and (y,id′,ad) that have same hash y, but different identifiers. This causes an issue that is addressed outside of the cryptographic protocol. Suppose a user copies a single image from a USB drive onto his or her device. The image will be assigned an identifier id. Later the user copies the same image from the USB drive onto a different client device. The new copy of the image will be assigned a new identifier id′ which is likely to be different from id. Because the two copies have different identifiers they will count twice towards the tPSI-AD threshold. In particular, the two triples will cause two distinct Shamir shares to be sent to the sever, even though they correspond to the same semantic image. Several solutions to this were considered, but ultimately, this issue is addressed by a mechanism outside of the cryptographic protocol. [0] [0]: https://www.apple.com/child-safety/pdf/Apple_PSI_System_Security_Protocol_and_Analysis.pdf https://www.apple.com/child-safety/pdf/Apple_PSI_System_Secu...
  • judge2020 5y ago
    Well, 3 photos or more are what puts you over the line for an affirmative defense against CSAM possession[0]. It's probably above that. 0: https://www.law.cornell.edu/uscode/text/18/2252A#:~:text=(d)Affirmative%20Defense https://www.law.cornell.edu/uscode/text/18/2252A#:~:text=(d)...
• cratermoon 5y ago
  Question: who is or will be making money on this deal? Answer that ("follow the money") and then I think we'll have a handle on what's really going on.
  • NonContro 5y ago
    Apple gets to maintain its 30% commission and monopoly abuse of the app store (which is becoming the de-facto place where all consumer software downloads and payments occur), in return for feeding the iCloud data of all users worldwide to US intelligence agencies. Presumably the quid pro quo outcome is that Apple is allowed to win the Epic vs Apple lawsuit.
    • cratermoon 5y ago
      Ah! This and the other comment suggest a method: Apple implements something for "THINK OF THE CHILDREN" in exchange for government calling off the regulation dogs. Cynical.
  • tgsovlerkhgsel 5y ago
    My guess: Apple is doing this to get regulators of its back while implementing end-to-end encryption.
• dmitryminkovsky 5y ago
  > However, nothing in the iCloud terms of service grants Apple access to your pictures for use in research projects, such as developing a CSAM scanner. (Apple can deploy new beta features, but Apple cannot arbitrarily use your data.) In effect, they don't have access to your content for testing their CSAM system. > If Apple wants to crack down on CSAM, then they have to do it on your Apple device. I don’t understand… Apple can’t change their TOS but they can install this scanning service on your device?
  • alfiedotwtf 5y ago
    > Apple can’t change their TOS Why not... has anyone actually successfully sued a company for changing their ToS from under them?
    • dmitryminkovsky 5y ago
      That’s what I don’t understand about that paragraph.
    • jdavis703 5y ago
      Yes, see Douglas v. Talk America.
      • alfiedotwtf 5y ago
        Thanks for the link. ... but I'm not sure that would apply here, especially if Apple has a pop-up that the user dismissed a long time ago.
        • jdavis703 5y ago
          It’s unclear what the legal limits are, but I’ve seen some sites do a bold print summary of the changes. Certainly if one wants to be sure their TOS change is enforceable they’ll make sure a judge and jury will agree the changes were prominently advertised.
    • jandrese 5y ago
• xvector 5y ago
  The engineers that worked on this should honestly be ashamed of themselves. We need some sort of oath of ethics in computer science.
• anonymouse008 5y ago
  This feels like missing the forest from the trees — Steve Jobs said many times to the effect ‘it doesn’t matter how any of this stuff happens, GigaHertz, Ram, Speeds, it only matters that the user gets what they want.’ Right now Apple’s biggest unhappy user is the DOJ. As it stands with the legislation coming down the pipe and both previous administrations building on a keenness to ‘get something done’ about big tech, Apple will do as they’ve done in China and ‘obey the laws in each jurisdiction.’ Right now there are a lot of unwritten laws that say Apple better play right or lose quite a bit more — So, how it’s getting done is a side show. That said, it wasn’t long ago that they stood toe to toe with the FBI —- but there also weren’t wonderfully strong ‘sanctions’ on the horizon.
  • fumar 5y ago
    Why do elected officials act as fake representatives to the people that elected them in the first place? Has it always been this way? It doesn’t matter left or right. The governing bodies should obey the people not the other way around.
    • closeparen 5y ago
      If the people had their way, those suspected of child sex crimes wouldn’t even get trials. Things like privacy and due process only exist to the extent that a ruling class has the power to impose their own values, contrary to popular will.
      • l33t2328 5y ago
        I’m one of the people and that’s not my way. I wonder where you got your data from.
        • reidjs 5y ago
          This made me come up with a thought experiment. If you sampled a thousand parents and asked them if they think sex offenders deserve a fair and just trial, what do you think the result would be?
• robertwt7 5y ago
  > Apple then manually reviews each report to confirm there is a match, This is always the terrifying part for me. They will access your personal photos or data without telling you. I’m surprised how is that even legal given all the law that are already available. Are they immune to those laws stated in thd blog? Also what happens when they launch this in EU, AU, etc with different privacy laws?
  • FabHK 5y ago
    I think the article got that wrong. Apple does manually review tagged images, but does not access the original image, but the security voucher containing metadata, including the NeuralHash and a "visual derivative" of it (see Apple's spec [1]). Also, this only applies to pictures you upload to iCloud. So, it's not like they're accessing your personal photos without telling you. [1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
    • GauntletWizard 5y ago
      Most iPhones have icloud backups enabled, which will trigger this detection automatically. Most iPhones are set up to upload every photo to icloud in case you lose your phone.
  • robertoandred 5y ago
    They do tell you. If your account is flagged, you are notified and have the opportunity to challenge it.
• blunte 5y ago
  Somewhat tangential, but people like this author amaze me in their deep knowledge AND ability to communicate it well. Also, none of this topic is something I would want to deal with.
• mlindner 5y ago
  That PhotoDNA is reversible is ridiculously shocking to me...
  • diebeforei485 5y ago
    Why? It's ancient, and hasn't been subject to open scrutiny from academics and researchers like open-source systems.
• Klonoar 5y ago
  I appreciate just about everything about this post, but this part keeps getting lost in everything I see written about it: >As noted, Apple says that they will scan your Apple device for CSAM material. If they find something that they think matches, then they will send it to Apple. The problem is that you don't know which pictures will be sent to Apple. It's iCloud Photos. Apple has explicitly said it's iCloud photos. If it's being synced to iCloud Photos, you know it's getting scanned one way or another (server side, currently, or client side, going forward). It notes privacy issues, but... iCloud syncs by default. You wouldn't do the kind of work they're talking about (e.g, investigation) and store that kind of material where it could be synced to a server to begin with. Everyone keeps proclaiming that Apple is scanning your entire device, but that's not what's happening with this change. It's not even comparable to A/V in this respect - it would be a very different story if that was the case. The wording and explanation matters.
  • kook_throwaway 5y ago
    The entire technical infrastructure to scan your entire device for arbitrary content is being built and deployed. The only change necessary to scan other files is changing a path and that's configuration that could even be silently done per-device.
    • zionic 5y ago
      I love that this is the (correct) response here. /r/Apple is full of morons that read the press release and honestly believe making this system scan the whole file system is impossible.
    • Klonoar 5y ago
      >The entire technical infrastructure to scan your entire device for arbitrary content is being built and deployed. It's a proprietary OS. This literally could exist already and you would have zero clue.
      • kook_throwaway 5y ago
        The difference between open and closed source is not lost to anyone on this site. There is a grand canyon sized gap between 'could already exist and you have no clue' and 'we have built and are deploying this system'.
• IronWolve 5y ago
  There was parents arrested over bath time and playing in the yard sprinklers, photos being processed at photo mats, will the same thing happen by apple mistakenly reporting parents? When I worked telecom, we had md5sum database to check for this type of content. If you emailed/sms/uploaded a file with the same md5sum, your account was flagged and sent to legal to confirm it. Also if a police was involved, the account was burned to dvd in the datacenter, and only a police officer would touch the dvd, no engineer touched or saw the evidence. (Chain of Evidence maintained) Prob changed since I haven't worked in telecom in 15 years, but one thing I've read for years, is the feds knew who these people are, where they hang out online, even ran the some of the honeypots. The problem is they leave these sites up to catch the ring leaders, the feds are aware, they have busts almost every month of rings of criminals. Twitter has had accounts reported, and they stay up for years. I dont think finding the criminals are the problem, seems like every time this happens, theres been people of interest for years, just not enough law enforcement dedicated to investigating this. All the defund the police, I think moving some police from traffic duty to Internet crimes would be more of an impact on actual cases being closed. Those crimes lead to racketeering and other organized crime anyways.
  • jolux 5y ago
    > There was parents arrested over bath time and playing in the yard sprinklers, photos being processed at photo mats, will the same thing happen by apple mistakenly reporting parents? No, because they’re not identifying content, they’re matching it against a set of already-known CSAM that NCMEC maintains. As you go on to say, telecoms and other companies already do this. Apple just advanced the state of the art when it comes to the security and privacy guarantees involved.
    • XorNot 5y ago
      It's a lossy hash match though. If it wasn't, then subtly re-encoding the image would hide it. So they're definitely going to be mistakingly matching images.
      • robertoandred 5y ago
• drivingmenuts 5y ago
  I think Apple may have really screwed the pooch with this move. They're going to catch hell for being able to take images from your device without warning or consent, and, they'll catch hell if they remove it. Worse, they've also opened the door to government censorship of images and content and propped that door wide open.
  • caeril 5y ago
    Yes, they screwed the pooch with us, representing: 1. The few who care about privacy. 2. That's about it. Maybe 1% if you're being generous. ( And subtract the portion of that 1% on this site who DO care about privacy but are being compensated $400k/yr by Apple and with Bay Area rents being what they are, they shouldn't rock the boat, and they really need to make good on their Tesla Roadster reservation. ) But go ask your mother-in-law, your dad, or any random normie friend. They're fighting PedoBear! And if they expand this further, well, they're fighting terrorists! And if they expand even further, well, I have nothing to hide! Do you? feh ~/memes/dennis_nedry_see_nobody_cares.gif
    • noptd 5y ago
      I disagree. Ask anyone you listed if they'd be comfortable with Apple reviewing their photos and possibly sending them to another organization without any warning or recourse and I highly doubt any of them would be okay with it.
  • mrshadowgoose 5y ago
    Unfortunately, we're in a bit of an echo chamber here. The average person is either unaware of these types of issues, doesn't care, or can be easily swayed with "think of the children"-type arguments. There's going to be absolutely no oversight or transparency into occasions when an image is removed from a device. Nobody will ever know when a non-CSAM image is accidentally pulled back from a device. All the public will ever see are headlines to the tune of "CSAM scanning system catches bad person once again". This is a really awful path that we're going down, and this is absolutely going to get abused by regimes around the world.
• fspacef 5y ago
  Great title.
• ksec 5y ago
  >Think of it this way: Your landlord owns your property, but in the United States, he cannot enter any time he wants. In order to enter, the landlord must have permission, give prior notice, or have cause. Any other reason is trespassing. Moreover, if the landlord takes anything, then it's theft. Apple's license agreement says that they own the operating system, but that doesn't give them permission to search whenever they want or to take content. Yes. That is an analogy only with Apple's software but not hardware. In Apple's view they are selling you the experience. So they are more like Hotels, You dont own the hotel room, the bed, the TV or anything inside that room. And in a Hotel, they can do Room Cleaning anytime they want.
• Vorh 5y ago
  https://web.archive.org/web/20210808233609/https://www.hackerfactor.com/blog/index.php?/archives/929-One-Bad-Apple.html https://web.archive.org/web/20210808233609/https://www.hacke... (I can't seem to access archive.is, so archive.org)
• g5095 5y ago
  I've been a FOSS dev for 25 years and I remember when everyone else I worked with were avid linux/freeBSD users because 'we didn't trust the big end of town'.. over the years I've watched the vast majority of devs move to apple devices for all sorts of 'just works', 'shinier' reasons that just boil down to 'convenience is more important than privacy'. Perhaps this is just the benefit of longevity but from my POV it was engineer early adoption and advocacy that made Apple, Google Search etc what they are, and it will be engineer early adoption and advocacy that dethrones these problematic companies from controlling the ecosystem.. Back 20 years ago, before the community filled with $_$ dollars-struck startup founders, software was built by people who wanted to use it.. rather than sell it. There are still some people doing this now, Look at Matrix network for instance. What will it take for a grass-roots software industry to start building privacy-first apps and systems that don't suck, based on decentralised, distributed principles? We have the skills to build highly polished alternatives to these things, but it takes a determination to step away from convenience for a period of time for the sake of privacy. How bad does it have to get before the dev community realise this? or are we in a frog boiling slowly scenario and it's hopeless?
• cromwellian 5y ago
  I don't see many people pushing back on the child pornography laws themselves that are the cause of this. I'm stepping into a hornets nest by even bringing this up, because any criticism of the laws on the books makes one look they're a pedo, so I'll preface by saying, child pornography (filmed with actual kids) is vile and disgusting, but it is the production of it that is evil to be fought and suppressed, not the possession of it. Remember in the 90s, we had to deal with the Communications Decency Act I & II, because every time they want to crack down on the internet, the excuse is always "it's for the children!" And pedophilia is the go-to excuse in a lot of circumstances. The CPPA had bans on virtual child porn (e.g. using look-alike adult actresses or CGI), that was overturned by SCOTUS, and then Congress responded with the PROTECT act which tightened up those provisions. These laws on possession are practically unenforceable with modern technology, peer to peer file sharing, onion routing, and encrypted hard drives. Thus, in order to make them enforcement, the government has to put surveillance at all egress and ingress points of our private/secure enclaves, whether it's at the point of storing it locally, or the point of uploading it to the cloud. While I agree with the goal of eliminating child porn, should it come at the cost of an omnipresent government surveillance system everywhere? One that could be used for future laws that restrict other forms of content? How about anti-vax imagery? Anti-Semitic imagery? And with other governments of the world watching, especially authoritarian governments, how long until China, which had a similar system with Jingwang Weishi (https://en.wikipedia.org/wiki/Jingwang_Weishi https://en.wikipedia.org/wiki/Jingwang_Weishi) starts asking: hey, can you extend this to Falun Gong, Islamic, Hong Kong resistance, and Tiananmen square imagery? What if Thailand passes a law that requires Apple to scan for images insulting to the Thai Monarch, does Apple comply? This is a very bad precedent. I liked the Apple that said no to the FBI instead of installing backdoors. I'd prefer if Apple get fined, and battle all the way to the Supreme Court to resist this.
  • api 5y ago
    Something else I haven’t heard anyone bring up: when child abusers are caught, sentences are often a joke. They often spend less time in prison than low to mid level drug offenders. CP is rape porn. People who produce or knowingly distribute actual rape porn (of children or adults) should be going to jail for 25+ years. I would not rule out life for extreme cases that also involve other forms of abuse. Yet as far as I know this isn’t what is being pushed for. Instead we get a dragnet to try to catch low level offenders who will probably barely see prison because sex crimes are just not punished much in our society. Secondly… look into some of the people who have set up fake CP honey pots. These have included both vigilantes and security researchers trying to get a sense of how much CP is on Tor. Turns out most of these people are imbeciles and catching them is easy. Why don’t police do this more? Because… well… as I said sex crimes are just not a priority. I really feel like the whole argument is in bad faith. If they really cared about the children there are so many other things that would be both easier and more effective.
• iJohnDoe 5y ago
  I haven’t read all the details, articles, and comments. My personal thoughts on the whole situation are the following. If you are a parent and lose a child then you would want every possible avenue taken to find your child. You would be going mad wanting to find them. If there is a way to match photos to known missing children then I say it should be at least tried. I equate this to Ring cameras. They are everywhere. You cannot go for a walk without showing up on dozens of cameras, which we know Amazon (god mode) and law enforcement abuse their access privileges. However, if a crime happened to you and a Ring camera captured it, then I know almost everyone would certainly want that footage reviewed. Would you ignore the Ring footage possibility just because you despise Ring cameras? Probably not. It’s all an invasion of privacy until you’re sitting on the other side of the table where you have a vested interest in getting access to the information.
  • vageli 5y ago
    > It’s all an invasion of privacy until you’re sitting on the other side of the table where you have a vested interest in getting access to the information. It is still an invasion of privacy regardless of the side on which you sit. This situation is unlike a ring camera capturing footage in a public space where individuals already have no expecation of privacy.
  • shrewduser 5y ago
    > If you are a parent and lose a child then you would want every possible avenue taken to find your child. You would be going mad wanting to find them. If there is a way to match photos to known missing children then I say it should be at least tried. the problem with this argument is that you can use it to justify basically anything.
  • chrismcb 5y ago
    And this is exactly why we have things like the fourth amendment in the US.
  • fragmede 5y ago
    If I lost my child, I would do anything and everything to get them back. I would shoot anyone who got in the way. This is fine because I am not the police and they won't give me guns. Kinda unrelated to that, is the 4th amendment, which protects the rights of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures by the government.
• kemayo 5y ago
  > To reiterate: scanning your device is not a privacy risk, but copying files from your device without any notice is definitely a privacy issue. I think the article is wrong about this. Or, right-but-situationally-irrelevant. As far as I can tell from Apple's statements, they're doing this only to photos which are being uploaded to iCloud Photos. So, any photo this is happening to is one that you've already asked Apple to copy to their servers. > In this case, Apple has a very strong reason to believe they are transferring CSAM material, and they are sending it to Apple -- not NCMEC. I also suspect this is a fuzzy area, and anything legal would depend on when they can actually be said to be certain there's illegal material involved. Apple's process seems to be: someone has uploaded photos to iCloud and enough of their photos have tripped this system that they get a human review; if the human agrees it's CSAM, they forward it on to law enforcement. There is a chance of false positives, so the human review step seems necessary... After all, "Apple has hooked up machine learning to automatically report you to the police for child pornograpy with no human review" would have been a much worse news week for Apple. :D
  • ajsnigrutin 5y ago
    So.. how well does "human review" work with copyright on youtube? This is basically fearmongering, and saying "if you're not a pedo, you have nothing to fear", installing the tech on all phones, and then using that tech to find the next wikileaks leaker (who was the first person with this photo), trump supporters (just add the trump-beats-cnn-gif to the hashes), anti-china protesters (winnie the pooh photos), etc. This is basically like forcing everyone to "voluntarily" record inside of their houses, AI on that camera would then recognise drugs, and only those recordings will be sent to the police.
    • kemayo 5y ago
      I was pointing out an inaccuracy in the article, not commenting about whether or not this tech is a good idea. I think if we're opposed to it, we should avoid misrepresenting it in arguments. On which note... it really does seem to be voluntary, as there's an easy opt-out of the process in the form of "not using iCloud Photos". Or Google Photos, or assorted other services, which apparently all do similar scanning. Yes, there's a slippery-slope argument here, but the actual service-as-it-exists-today really does seem scoped to a cautious examination of possible child porn that's being uploaded to Apple's servers.
• heavyset_go 5y ago
  As a fan of this blog for longer than I can remember, it's refreshing to hear this particular author's take on this issue, especially considering their background. I'm glad these issues were addressed in a much more elegant way than I would have put them: > Apple's technical whitepaper is overly technical -- and yet doesn't give enough information for someone to confirm the implementation. (I cover this type of paper in my blog entry, "Oh Baby, Talk Technical To Me" under "Over-Talk".) In effect, it is a proof by cumbersome notation. This plays to a common fallacy: if it looks really technical, then it must be really good. Similarly, one of Apple's reviewers wrote an entire paper full of mathematical symbols and complex variables. (But the paper looks impressive. Remember kids: a mathematical proof is not the same as a code review.) > Apple claims that there is a "one in one trillion chance per year of incorrectly flagging a given account". I'm calling bullshit on this.
  • q-rews 5y ago
    Regarding that rate, I’m no expert but my guess is that it’s the result of math, not actual testing of 1+ trillion images. This sounds like calling bullshit on “You have one trillion chance to win the lottery.”
    • heavyset_go 5y ago
      The author addresses this point: > Perhaps Apple is basing their "1 in 1 trillion" estimate on the number of bits in their hash? With cryptographic hashes (MD5, SHA1, etc.), we can use the number of bits to identify the likelihood of a collision. If the odds are "1 in 1 trillion", then it means the algorithm has about 40 bits for the hash. However, counting the bit size for a hash does not work with perceptual hashes. > With perceptual hashes, the real question is how often do those specific attributes appear in a photo. This isn't the same as looking at the number of bits in the hash. (Two different pictures of cars will have different perceptual hashes. Two different pictures of similar dogs taken at similar angles will have similar hashes. And two different pictures of white walls will be almost identical.) > With AI-driven perceptual hashes, including algorithms like Apple's NeuralHash, you don't even know the attributes so you cannot directly test the likelihood. The only real solution is to test by passing through a large number of visually different images. But as I mentioned, I don't think Apple has access to 1 trillion pictures. > What is the real error rate? We don't know. Apple doesn't seem to know. And since they don't know, they appear to have just thrown out a really big number. As far as I can tell, Apple's claim of "1 in 1 trillion" is a baseless estimate. In this regard, Apple has provided misleading support for their algorithm and misleading accuracy rates.
• easterncalculus 5y ago
  I'm just wondering how long it will take before average people, jokingly or otherwise, imply that buying an android phone makes you a criminal.
  • mrshadowgoose 5y ago
    Sadly, I'm not worried about that particular possibility. I'm fully expecting Google to roll out some similar BS within a year or two. "For the children" of course.
• charles_f 5y ago
  > So where else could they get 1 trillion pictures? That's a real kicker in my opinion. Unless they get training data from NCMEC I struggle to understand how they're training their model? Unless it's entirely algorithmic and not based on ML?
• borland 5y ago
  The "legal" section talks about local scanning, and possible transmission of CSAM from devices to Apple, in pursuit of verification, however Apple have made clear that the scanning happens only for files that have been uploaded to iCloud Photo Library -- in which case they are not deliberately transmitting the CSAM but rather flagging something which the user already sent them. Likewise the copyright issue; The user has already sent these files to Apple themselves by enabling iCloud photo library, and Apple are not making any additional copies that I am aware of. It also says "The problem is that you don't know which pictures will be sent to Apple." - but we do know exactly which pictures will and will not be sent to apple; the ones that are already sent by iCloud Photo library. [To be clear, I don't like the precedent/slippery slope that this kind of technique might lead towards in the future, but it doesn't seem like all the criticisms of it today are valid]
  • zionic 5y ago
    This is a short sighted and naive viewpoint. Apple did not build this entire system to only scan uploads hah. This was built to scan the entire file system.
• cletus 5y ago
  I'm honestly shocked that Apple is buying into this because it's one of those well-intentioned ideas that is just incredibly bad. It also goes to show you can justify pretty much anything by saying it fights terrorism or child exploitation. We went through this 20+ years ago when US companies then couldn't export "strong" encryption (being stronger than 40 bits if you can believe that). Even at the time that was ridiculously low. We then moved onto cryptographic back doors, which seem like a good idea but aren't for the obvious reason that if a backdoor exists, it will be exploited by someone you didn't intend or used by an authorized party in an unintended way (parallel construction anyone?). So these photos exist on Apple servers but what they're proposing, if I understand it correctly, is that that data will no longer be protected on their servers. That is, human review will be required in some cases. By definition that means the data can be decrypted. Of course it'll be by (or intended to be by) authorized individuals using a secured, audited system. But a backdoor now exists. Also, what controls exist on those who have to review the material? What if it's a nude photo of an adult celebrity? How confident are we that someone can't take a snap of that on their own phone and sell it or distribute it online? It doesn't have to be a celebrity either of course. Here's another issue: in some jurisdictions it's technically a case of distributing CSAM to have a naked photo of yourself (if you're underage) on your own phone. It's just another overly broad, badly written statute thrown together in the hysteria of "won't anybody think of the children?" but it's still a problem. Will Apple's system identify such photos and lead to people getting prosecuted for their own photos? What's next after this? Uploading your browsing history to see if you visit any known CSAM trafficking sites or view any such material? This needs to be killed.
  • fuzzybassoon 5y ago
    I don't know how I feel about all of this yet (still trying to understand better), but your post implies that you've made a lot of incorrect assumptions about how this system works. For example, the main system in discussion never sends the image to Apple, only a "visual proxy", and furthermore, it only aims to identify known (previously cataloged) CSAM. There's a [good primer of this on Daring Fireball](https://daringfireball.net/2021/08/apple_child_safety_initiatives_slippery_slope https://daringfireball.net/2021/08/apple_child_safety_initia...)
• robnewman7900 5y ago
  Wow, 20% of images being misclassified as CP would be terrifying. I never thought I'd say it but my next phone will definitely not be an iPhone or any device that is uploading my information to another server without my permission for any reason. Is there even any evidence that arresting people with the wrong bit pattern on the computer helps stop child rape/trafficking? If so, why aren't we also going after people who go to gore websites? There's tons of awful material out there easily accessible of people getting stabbed, shot, murdered, butchered, etc. Do we not want to find people who are keeping collections of this material on their computers? And if so, what about people who really like graphic horror movies like Saw or Hostel? Obviously it's not real violence, but it's definitely close enough, and if you like watching that stuff, maybe you should be on a list? If your neighbor to the left of you has videos of naked children, and your neighbor to the right has videos of people getting stabbed and tortured to death, only one should be arrested and put on a list? This is all not even taking into account that someone might not even realize they are in possession of CP because someone else put it on their device. I've heard there's tons of services marketing on the dark net where you pay someone $X00 in bitcoin and they remotely upload CP to any target's computer. It seems like we are going down a very scary and dangerous path.
  • robertoandred 5y ago
    You did agree to upload your information to another server when you turned on iCloud Photo Library.
    • robnewman7900 5y ago
      I don't have iCloud Photo Library
      • robertoandred 5y ago
        Then you don’t have to worry about it uploading anything.
        • crooked-v 5y ago
• gscott 5y ago
  Kim Dotcom did this and it lead to Riaa saying it obligated the file sharing service to look for all copyrighted materials. This opens pandoras box for Apple.
  • judge2020 5y ago
    No, it didn't, at least not legally. There is no federal requirement to scan for copyrighted content (which is an impossible task without $MM in capital and a media library of copyrighted content) - YouTube only does it with Content ID to appease the rights holders and not have them remove all their content from YT outright. The only requirement is to respond to DMCA takedown notices, which they were obligated to do before Kim started scanning for CSAM (which I can't find a story for).
• neilv 5y ago
  I'm concerned about personal security risks of this move by Apple (e.g., Swatting-like attacks). Just wasted the weekend on this, and now asking for options: https://news.ycombinator.com/item?id=28111995 https://news.ycombinator.com/item?id=28111995
• DethNinja 5y ago
  There is one particular thing I don’t understand about this Apple policy: You can buy a SIM card and send images to your enemies/competitors through WhatsApp, and these images automatically gets downloaded to iPhone and potentially uploaded to iCloud. What precautions are Apple taking against such actions? Or will it be some kind of exploitable implementation where you can easily swat any person you want and let them go to courts to prove their innocence?
  • celeritascelery 5y ago
    I know of no messaging app the “automatically” uploads to iCloud. And what sort of idiot would try and send CP using a messenger app (almost all of which CASM detection). That’s like trying to smuggle drugs past the TSA so you can put them in someone’s house to frame them.
    • vmladenov 5y ago
      The app doesn’t. The iOS camera roll uploads newly captured or saved images when it connects to Wi-Fi.
      • celeritascelery 5y ago
        If you choose to save it, then you no longer a “victim”. But someone else can’t force a save to iCloud on your device.
        • vmladenov 5y ago
          Sure, but some apps allow you to choose once and henceforth save all photos sent to you. That provides a vector for someone to inject things into your iCloud. However, I find this risk exaggerated because if someone actually does this, you can just block the app’s ability to access your camera roll in the privacy settings or stop using the app. And report the user to the service, of course.
    • kilroy123 5y ago
• celeritascelery 5y ago
  This made me think of a good point regarding apples error rate of “one in a trillion”. If they are so confident that there won’t be false positive’s why bother sending it to apple for manual review? Clearly they think the error rate will be high enough that a human has to double check everything. But that is not what they are saying.
  • ec109685 5y ago
    Apple doesn’t have access to the original CSAM images — only the hashes generated by the agency that produced them, so there could be hashes that don’t match anything illegal that trigger a false positive.
• judge2020 5y ago
  > As it was explained to me by my attorney, that is a felony. Apple could argue they were already going to receive the photo (since this algorithm only affects Photos destined for iCloud Photos) and thus "when in the upload process it was classified" is simply technological semantics. > This was followed by a memo leak, allegedly from NCMEC to Apple: Well, we certainly are the minority. If a majority of people knew and were mad we'd have protests in major cities.
• SPBS 5y ago
  > Apple's license agreement says that they own the operating system, but that doesn't give them permission to search whenever they want or to take content. If this is true, how has this gotten past Apple's legal team? Are they not aware it would be a flagrant violation of the law?
• a-dub 5y ago
  so maybe i'm confused, but i thought it worked like this: pre this thing: * before syncing photos to icloud, the device encrypts them with a device local key, so they sit on apple's servers encrypted at rest and apple cannot look at them unless they push an update to your device that sends them your key or uploads your photos unencrypted somewhere else after this thing: * before syncing photos to icloud, the device encrypts them, but there are essentially two keys. one on your device, and one that can be derived on their servers under special circumstances. the device also hashes the image, but using one of these fancy hashes that are invariant to crops, rotations, translations and noise (like shazam, but for pictures) * the encrypted photo is uploaded along with the hash (in a special crypto container) * their service scans all the hashes, but uses crypto magic that does the following: 1) it does some homeomorphic encryption thing where they don't actually see the hash, but they get something like a zero knowledge proof if the image's hash (uploaded along with the image in the "special crypto container") is in their list of bad stuff 2) if enough of these hit, then there's a key that pops out of this process that lets them decrypt the images that were hits 3) the images get added to a list where a room full of unfortunate human beings look at them and confirm that there's nothing good going on in those photos 4) they alert law enforcement a couple points of confusion to me: 1) i'm assuming they get their "one in a trillion" thing based on two factors. one being the known false positive rate of their perceptual hashing method and the other being their tunable number of hits necessary to trigger decryption. so they regularly benchmark their perceptual hash thing and compute a false positive rate, and then adjust the threshold to keep their overall system false positive probability where they want it? 2) all the user's photos are stored encrypted at rest, it seems that this thing isn't client side scanning, but rather assistance for server side scanning of end to end encrypted content put on their servers. first off i think it's actually pretty cool that a consumer products company offers end to end encrypted cloud backup of your photos. i don't think google does this, or anyone else. they can just scan images on the server. second off, this is some pretty cool engineering (if i understand it correctly). they're providing more privacy than their competition aaand they've given themselves a way to ensure that they're not in violation of the law by hosting CSAM for their customers. but i guess the big question is, if people don't like this, can't they just disable icloud?
  • upbeat_general 5y ago
• deleted 5y ago
  [deleted]
• eh9 5y ago
  I really HATE usage of “CP”, it’s abuse and has nothing to do with pornography.
• starchild_3001 5y ago
  Bad Apple it is. Also hypocrite Apple, with their fake (at the very least misleading) privacy stance against internet advertising.
• zmmmmm 5y ago
  There is so much focus on the technical aspects such as probability of mismatch etc. For me the risk is much more that through some mechanism outside my control real CSAM material becomes present on my device. Whether its a dodgy web site, a spam email, a successful hack attempt or something else like that, I feel like there's a significant chance some day I'll end up with this stuff injected onto my phone without me knowing. So I'm not at all concerned about the technical capacity to accurately match to CP etc. In fact I'm even more worried if its really accurate because then I know when this unfortunate event happens I face a huge risk of being immediately flagged before I even know about the content and then spending years extricating myself from a ruined reputation and a legal system that treats evidence like this with far more trust than it should have.
  • rovr138 5y ago
    I still have WhatsApp on my phone and I have had issues in the past with backups so I have it set to save the media directly to my photos app. I also have notifications off on it and check it when I need to. All this needs is someone forwarding me something that’s in the DB.
  • judge2020 5y ago
    The reason everyone is focusing on the technical aspects is because most people will evaluate it as it is planned to be, and not as what it could become. This is probably because, at any time, users can switch from Apple to Android in an upgrade cycle - so while perhaps 99% of users are fine with this CSAM scanner in its current state, if they expand its usage to something bad then those 99% that stayed can once again evaluate if they still value the hardware enough to keep it with the new status quo; therefore, evaluating it as it is in its current state will help people reading make the best personal decisions and the details will help with context if/when they do expand usage of this system.
  • randyrand 5y ago
    right? Phones are not at all secure. Dozens (hundreds?) of click-less exploits exist today that can do anything they want with “your” phone at the push of a button. My phone is not mine. nor is the data on it. nor is yours. That’s the real state of computer security today. all of this is ill conceived. The day someone chooses to mass release their worms on iPhone will be a wake up call.
• andrewmcwatters 5y ago
  1 in 1 trillion of what? https://i.imgur.com/E1YRsXQ.jpg https://i.imgur.com/E1YRsXQ.jpg
• ikhanhmai 5y ago
  cool
• ikhanhmai 5y ago
  awesome product!
• Little_John 5y ago
  I hear a Siren Outside Someone has a serious problem. Don't just shine it on.
• chris_wot 5y ago
  If they file even one false claim, then it ruins lives.
• hdjjhhvvhga 5y ago
  > If someone were to release code that reverses NCMEC hashes into pictures, then everyone in possession of NCMEC's PhotoDNA hashes would be in possession of child pornography. Please correct me if I'm wrong, but wouldn't it be more correct to say they "would be in possession of images recognized by PhotoDNA as child pornography" rather than actual CP?
  • raxxorrax 5y ago
    Technically not possible without severely grasping assumptions. It would be more likely you would create a collision and have calculated an image of a duck (south african whitenoise duck). Problem with all this is that the images of naked children made by their parents is CP in the eyes of its consumers. Perceptual AI is the best approach, but produces a certainty < 1. In my cryptography course I had a project about invisible watermarks and secret messages in imaging. The first hurdle was beating partial images and compression, so most early algorithms worked in the frequency domain. At that time it was basically an arms race between protection or deletion of said messages and I think that hasn't changed. Conventional file hashes can be beaten by randomizing meta data since a quality hash function would immediatly create a completely different hash. Never mind just flipping or probably just resaving them. If you create a polynomial approximation of frequencies or color histograms of an image, you have a relatively short key indicator. But you need a lot of those to even approach certainty. Could always be an image of a duck.
• therealmarv 5y ago
  If this is true and content of pictures can be unhashed to a 26x26 picture this is ethically a total nightmare. Nobody wants to carry a phone with child porn with them, discusting if you think about it. I don't even want to imagine what very religious people and countries will think about the iPhone then.
• ChrisMarshallNY 5y ago
  Here's the gist of a post I made a couple of days ago (I removed one sentence that someone considered inflammatory): Having known many victims of sexual violence and trafficking (Seriously. I deal with them, several times a week, and have, for decades), I feel for the folks that honestly want that particular kind of crime to stop. Humans can be complete scum. Most folks in this community may think they know how low we can go, but you are likely being optimistic. That said, law enforcement has a nasty habit of having a rather "binary" worldview. People are either cops, or uncaught criminals. With that worldview, it can be quite easy to "blur the line" between child sex traffickers, and traffic ticket violators. I remember reading a The Register article, about how anti-terrorism tools were being abused by local town councils to do things like find zoning violations (for example, pools with no CO). Misapplied laws can be much worse than letting some criminals go. This could easily become a nightmare, if we cede too much to AI. And that isn't even talking about totalitarian regimes, run by people of the same ilk as child sex traffickers (only wearing Gucci, and living in palaces). ”Any proposal must be viewed as follows. Do not pay overly much attention to the benefits that might be delivered were the law in question to be properly enforced, rather one needs to consider the harm done by the improper enforcement of this particular piece of legislation, whatever it might be.” -Lyndon B. Johnson [EDITED TO ADD]: And I 100% agree that, if we really want to help children, and victims of other crimes, then we need to start working on the root causes of the issues. Poverty is, arguably, the #1 human problem on Earth, today. It causes levels of desperation that ignore things like climate change, resource shortages, and pollution. People are so desperate to get out of the living hell that 90% of the world experiences, daily, that they will do anything (like sell children for sex), or are angry enough to cause great harm. If we really want to solve a significant number of world problems, we need to deal with poverty; and that is not simple at all. I have members of my family that have been working on that, for decades. I have heard all the "yeah...but" arguments that expose supposedly simple solutions as...not simple. Of course, the biggest issue, is that the folks in the 0.0001% need to loosen their hands on their stashes, and that ain't happening, anytime soon. I don't know if the demographic represented by the Tech scene is up for that, since the 0.0001% are our heroes.
• max_entropy 5y ago
  Imagine being a victim of CP, knowing that all over the world people hold in their hands devices storing artifacts of your terror.
• IshKebab 5y ago
  > Apple claims that there is a "one in one trillion chance per year of incorrectly flagging a given account". I'm calling bullshit on this. Why? You can get any false positive rate you want if you don't care about the false negative rate. It seems likely that that was a design criterion and they just tweaked the thresholds and number of hits required until they got it. The last analysis on HN about this made the exact same mistake, and it's a pretty obvious one so I'm skeptical about the rest of their analyses. It is nice to have some actual numbers from this article though about how much CP they report, the usefulness of MD5 hashes, etc. Edit: reading on, it seems like he just misread - it sounds like he thinks they're saying there's a 1 in a trillion chance of a false positive on a photo but Apple are talking about an account which requires multiple photo hits. The false positive rate per photo might be 1 in 1000 but if you need 10 hits then it's fine.
• jscipione 5y ago
  What assurances do we have that this system will not be used to flag "extremist content" i.e. memes and to send that information to law-enforcement in the future?
• mistermarcus 5y ago
  You believe CP is evil. You believe privacy is sacred. You are Tim Cook. What do you do? If this is a sincere effort, clearly Apple has failed to thread the needle. This announcement could also be a fumbled attempt to reframe what is already a common practice at the company, to get ahead of a leak. I heard from a friend who's related to an Apple employee that Apple scans the mountains of data on its servers, already, for "market research". The claims otherwise... marketing gambits.
• lizardmancan 5y ago
  in theory we could accomplish great things by spying on everyone but even a small amount of abuse of such tools is unacceptable. I dont think we can stop the mass spying indefinetly which makes me think the corrupt governments and dictators are increasingly viable low hanging fruits. there will be a point where we have no other choice but to dethrone them. if we fail there we will all be like the chinese captures in a panopticon of cameras driving large behavrioal datacenters.
• mistermarcus 5y ago
  You believe CP is evil. You believe privacy is sacred. You are Tim Cook. What do you do? If this is a sincere effort, clearly Apple has failed to thread the needle. This announcement could also be a fumbled attempt to reframe what is already a common practice at the company, to get ahead of a leak. I heard from a friend who's related to an Apple employee that Apple scans the mountains of data on its servers, already, for "market research". The claims otherwise... marketing gambits
• vpmpaul 5y ago
  Maybe you can answer this. What amount of the CP pictures are actual CP? Like are teens posting selfies of themselves in swimsuits or revealing clothes being submitted? People with their kids in the bathtub/pool? Are most of these pictures real give you nightmares CP? Its seems insane to me that anyone would knowingly upload CP to a forensics site on purpose. Much less several times a day.
• HALtheWise 5y ago
  Given that the weights for the NeuralHash algorithm are being shipped to every iOS device and neural network adversarial attacks are pretty well studied in literature, it should be trivial to make a website or Android camera app that tweaks a few pixels of an image to make it have a hash collision with apple's CSAM database. If anyone seriously wants to kill this initiative, widely distributing a few memes with hash collisions could go a long way.