9 ms·
Is macOS Look Up Destined for CSAM?
- buildbuildbuild 5y agoHow does someone truly test how this feature is being used without possessing illegal content? This is a nearly-impossible area to research. Frightening. (edit: I'm of course referring to possessing anti-Putin memes) (sarcasm)
- version_five 5y agoBy test it, do you mean see if the police show up at your door? If you know how it works, you just need a list of hashes and a way to find a collision which I believe exists. Otherwise, you're really just highlighting the problem with all closed source software, you don't really have a way to check what it does so you have to trust the vendor.
- rootusrootus 5y agoWe already know that a hash collision doesn't get far enough to involve police showing up at your door, so a full test would take something more substantial.
- chockchocschoir 5y agoYou don't have to test it against anti-Putin memes to see if it would work for anti-Putin memes. Algoritm would be something like: 1. Have image 2. Get hash of image 3. Get another hash from another similar image 4. Compare hashes The images themselves can be of whatever to see if it works as expected, they don't have to contain anti-Putin memes.
- willcipriano 5y agoCome up with something like: X51!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-CHILD-ABUSE-CONTENT-TEST-FILE!$H+H* https://en.m.wikipedia.org/wiki/EICAR_test_file https://en.m.wikipedia.org/wiki/EICAR_test_file
- buildbuildbuild 5y agoI thought about this, but you're still stuck trusting the implementation unless you test with actual illegal data, which is often criminal and immoral to obtain. Example: How does a researcher test whether algorithmically-classified illegal imagery stored on user devices is being scanned and reported home to Apple's servers, and what those bounds of AI-classified criminality are? (presumably with respect to what is illegal in the user's jurisdiction) Testing by using a test phrase, like in a spam context, is inadequate here as a scanning system can trivially be architected to pass those publicly-known tests, while still overreaching into people's personal files and likely miscategorizing content and intent. If a user connects via a VPN to Russia for whatever reasons, does their personal content start getting reported to that country's law enforcement by their notion of what is illegal? Parents often have all sorts of sensitive photos around which are not held with exploitative intent. "Computer says arrest."
- _fat_santa 5y ago> Example: How does a researcher test whether algorithmically-classified illegal imagery stored on user devices is being scanned and reported home to Apple's servers, and what those bounds of AI-classified criminality are? (presumably with respect to what is illegal in the user's jurisdiction) I'm not an expert in AI so this might be totally off base but I feel like you would be able to use an "intersection" of sorts for this type of detection. You detect children and pornography, the children portion trains it for age recognition and the porn portion trains it to see sexual acts. Slap those two together and you've got CSAM detection.
- Brian_K_White 5y agoI always imagine aliens hearing about something like this and being stunned. "How can data be illegal?" "There are bad things, but how can you decide what is bad and show how it's bad without examining and discussing it?" You can only go so far merely alluding to things. Somewhere the rubber has to meet the road and you have to have concrete data and examples of anything you need to study or make any sort of tools or policy about. It's like parents not talking to kids about sex. You can avoid it most of the time because decorum, but if you take that to it's extreme you have just made your child both helpless and dangerous through ignorance. Somewhere along the way, you have to explicitly wallow directly in the mess of stuff you seek to avoid most of the time. That "seek to avoid" can only ever be "most of the time". It's insane and counter-productive to try to see that "most of the time" as an incomplete job and improve that to 100%. I guess in this case there will eventually be some sort of approved certified group. A child porn researcher or investigator license. Cool. Cops with special powers never abuse them, and inhibiting study to a select few has always yielded the best results for any subject, and a dozen approved good guys can easily stay ahead of the world of bad guys.
- perihelions 5y agoObviously, definitionally, it's impossible to verify that server-side logic isn't doing something evil. (Local homeomorphic protocols count, when the secret logic is imported from remote servers). This is one reason FOSS is actually-important and actually-relevant. Isn't it valid to know exactly what your personal computer is doing, to be able to trust your own possessions? Richard Stallman was *never* crazy; his understanding of these issues is so cynical as to be shrill and off-putting, but that's well-calibrated to the severity of the issues at stake. You joke about anti-Putin memes. Here's a thought for well-calibrated cynics: Apple solemnly swears its hashes are attested by at least two independent countries. Russia and Belarus are two independent countries.
- azinman2 5y agoYou mean one of the countries sales of devices just stopped in? And the other already was announced to be a US org? And you need the intersection of both?
- perihelions 5y ago- "And the other already was announced to be a US org?" Then one rogue employee in a US org could be sufficient to get selective root to every Apple device everywhere? That's easy for a nation-state adversary. Here's demonstrated examples: MBS had US-based moles in Twitter corporate spying on Khashoggi [0], and Xi had Chinese-based Zoom employees spying on dissidents in America [1]. [0] https://www.npr.org/2019/11/06/777098293/2-former-twitter-employees-charged-with-spying-for-saudi-arabia https://www.npr.org/2019/11/06/777098293/2-former-twitter-em... [1] https://www.justice.gov/opa/pr/china-based-executive-us-telecommunications-company-charged-disrupting-video-meetings https://www.justice.gov/opa/pr/china-based-executive-us-tele... That second example is topical: the Chinese state used their Zoom assets to attempt to frame Americans for CSAM possession. - "As detailed in the complaint, Jin’s co-conspirators created fake email accounts and Company-1 accounts in the names of others, including PRC political dissidents, to fabricate evidence that the hosts of and participants in the meetings to commemorate the Tiananmen Square massacre were supporting terrorist organizations, inciting violence or distributing child pornography. The fabricated evidence falsely asserted that the meetings included discussions of child abuse or exploitation, terrorism, racism or incitements to violence, and sometimes included screenshots of the purported participants’ user profiles featuring, for example, a masked person holding a flag resembling that of the Islamic State terrorist group."
- mike_d 5y agoResearch has to be done in partnership with the NCMEC, which in turn partners with the Department of Justice to run the database of known CSAM material.
- deleted 5y ago[deleted]
- nyanpasu64 5y agoDoes macOS 12.3 and beyond phone home with details of images and documents you open in Preview?
- dev_tty01 5y agoNo. Why would you think that? It goes against everything they have stated and the designs of the software. They are heavily focused on keeping all of that on device.
- nkozyra 5y agoWell other than the proposal specifically outlined in this post, that is. (Pardon if I missed the sarcasm)
- nyanpasu64 5y agoI tested OCR with Wi-Fi disabled and it still functions. Is Visual Look Up (and Live Text) purely offline, phoning home with CSAM reports, or an offline preview of future technology which phones home with CSAM reports?
- nojito 5y agoEverything is done on device. Which is different than others who choose to do scanning on their servers.
- tylersmith 5y agoAnd if the filter thinks the image is positive for CSAM it sends it to Apple, correct? Otherwise there's literally no point.
- JimDabell 5y ago> And if the filter thinks the image is positive for CSAM it sends it to Apple, correct? No. The logic was never “if there’s a match the image is uploaded”. The device never knows if there’s a match. Under the process Apple described, an extra packet of data is attached to every iCloud upload. If there are enough matches, Apple can decode those packets to get a low-res thumbnail, which they then check against a second perceptual hash. The process doesn’t work on arbitrary images on your device, it’s specifically designed for iCloud uploads.
- symlinkk 5y agoRambling post, hard to follow or understand
- User23 5y agoWhat's the right number of lives to destroy over false positives from an algorithm? Is it some number other than zero? Why or why not?
- cmckn 5y agoWhat’s the right number of lives to destroy from CSAM? There’s a middle ground between doing nothing and totalitarianism. “Destroyed lives” from false positives are at this point hypothetical. Child abuse is not. It’s fair to be concerned about false positives and ensure the system handles such failures appropriately. It’s also fair to directly intervene in the widespread circulation of CSAM.
- logicchains 5y ago>What’s the right number of lives to destroy from CSAM? Is there any evidence such things actually reduce the production of CSAM? Or is it like the war on drugs where drug production is as high as it's ever been.
- vetinari 5y agoThere's no really a middle ground; the right number is 0. It is better that ten guilty persons escape than that one innocent suffer. -- Blackstone's ratio
- account-5 5y agoOr in this situation more like: It is better that ten children get sexually abused than one innocent person comes under suspicion /s
- omginternets 5y agoThis isn't combatting abuse because producing CSAM != consuming CSAM. It's far better that 10 people beat off to children than have one innocent person come under suspicion.
- KarlKemp 5y agoNo, this vaguely related technology has nothing to do with whatever you are associating with it. If Apple wants to surreptitiously spy on your porn collection, they will do so, and won’t need cover.
- mdoms 5y agoThis article could help by defining what Visual Look Up is... I have never heard of it.
- Angostura 5y agoIndeed - here is a nice little article that described one aspect of it that was posted the other day: https://eclecticlight.co/2022/03/16/how-good-is-montereys-visual-look-up/ https://eclecticlight.co/2022/03/16/how-good-is-montereys-vi...
- noasaservice 5y agoI'm fine with child (prepubescent) rapists whom are adults to be sentenced to death. If you were an accomplice to that, as a videographer or similar, I'm also OK with death. (There's a really weird social area from 13-18, with the weirdness and illegality going away up at 18. Stuff in this realm, especially around 2 similar ages, gets very stupid. This is where you can get 2 16yo's sexting and being charged with CSAM of their own body. I'm avoiding this in this post.) But what does this CSAM scanner do? It only catches already-produced pictures of CSAM. In other words, it's evidence of said crime. In no other area of criminal law is there a law against said evidence itself. And yes, given the statutory nature of these images (possession is criminal, even if you didnt' put them there), I'm not at all comfortable in charging people for simple possession. Even if they have urges of liking age-inappropriate pornography and "CSAM", as long as they're not doing any physical actions of harming humans, I'd much rather them do so in their own bedroom alone. Nor do I buy into the gateway theory that CSAM leads to production of CSAM by raping children. This smacks to me of the DARE drug propaganda and gateway theory (which is complete bullshit). And, we also already have harder situations that have been deemed legal: SCOTUS stated that Japanese Manga Hentai featuring schoolgirls (obviously under 18, sometimes by quite a lot), are completely and 100% legal. Again, SCOTUS foscused on 1fa and the fact that no children were harmed in its production. And that leads to what just happened a few days ago. With the Zelenskyy (badly done) deepfake, when can we expect 18yr women with very petite bodies, being deepfaked into 10-13 year olds? In those cases, we could attest that everyone in the production is of legal age and provided ongoing consent. Will this fall under the same as Hentai? Tl;Dr: I'm for the criminal legalization of CSAM. I'm for death penalty for child rapists/child sexual assault. But this, I can see going very very bad, in easily overscoping CSAM to the "cause of the day".
- CharlesW 5y ago> But what does this CSAM scanner do? It only catches already-produced pictures of CSAM. You say this as if it's bad to identify people who are distributing or collecting known child pornography. Are you recommending that companies implement technologies which go beyond this by not depending on a corpus of existing materials?
- selimthegrim 5y ago
- kayodelycaon 5y agoOkay… iOS has had on device image recognition since 2016.
- lilyball 5y agoRecognizing "this is a cat" and "this is a specific painting of a cat" are different challenges though.
- deleted 5y ago[deleted]
- WinterMount223 5y agoWhat’s with the obsession with CP? I agree that it’s morally wrong and should be penalized but why is it perceived as the Ultimate Crime? Why is this tool supposedly for detecting CP only and not stolen bikes for sale, bullying through SMS, etc which are also criminal offenses?
- notRobot 5y ago> why is it perceived as the Ultimate Crime? Because you can apparently justify any move, no matter how authoritarian, by saying "think of the kids"! It's politicians and governments exploitating psychology to get away with problematic crap. It's not the ultimate crime, it's the ultimate justification.
- userbinator 5y agoJust like how "security" is often used in the same manner, but I agree that CP is a much more persuasive and emotional argument.
- oh_sigh 5y agoCP is fairly easy to recognize if you see it I'd imagine. I'm sure there are some instances where an adult just looks very young, but there is probably a lot of CP out there with no potential for that. How exactly does one recognize a bike as stolen from a photograph?
- nbzso 5y agoI am using Little Snitch since version 2. Practically all my connections form MacOS to the mothership are stopped by default. I turn off the filter only for updates. I avoid in general Apple software (Preview, Photos, etc.). Macs are used only for work. My personal information or browsing is done only on Linux with Open Snitch and minimalistic install. In the new information landscape blind trust can be harmful.
- user3939382 5y agoI couldn’t find a way to black hole CIDR blocks on Little Snitch which is necessary to completely silence macOS. However if you do this at the router you break stuff like iMessage on phones running on the wifi and so forth.
- LinuxBender 5y agoOne small suggestion, consider using tcpdump on your router to make sure LS is really stopping everything. Apple and Microsoft wised up to this some time ago and some things hook in the network stack after the application firewalls. If you spot something please report it to the Little Snitch developers. The amount of data captured by tcpdump can be minimized by only capturing syn/fin/rst packets assuming proto 6 tcp. 'tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0' And if capturing to a file one can also limit the total number of packets captured with -c 1000 This may be useful if you are also capturing proto 17 udp.
- nbzso 5y agoThanks.