4 ms·

How does someone truly test how this feature is being used without possessing illegal content? This is a nearly-impossible area to research. Frightening. (edit

How does someone truly test how this feature is being used without possessing illegal content? This is a nearly-impossible area to research. Frightening. (edit: I'm of course referring to possessing anti-Putin memes) (sarcasm)

version_five 5y ago
By test it, do you mean see if the police show up at your door? If you know how it works, you just need a list of hashes and a way to find a collision which I believe exists. Otherwise, you're really just highlighting the problem with all closed source software, you don't really have a way to check what it does so you have to trust the vendor.
- rootusrootus 5y ago
  We already know that a hash collision doesn't get far enough to involve police showing up at your door, so a full test would take something more substantial.
chockchocschoir 5y ago
You don't have to test it against anti-Putin memes to see if it would work for anti-Putin memes. Algoritm would be something like: 1. Have image 2. Get hash of image 3. Get another hash from another similar image 4. Compare hashes The images themselves can be of whatever to see if it works as expected, they don't have to contain anti-Putin memes.
willcipriano 5y ago
Come up with something like: X51!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-CHILD-ABUSE-CONTENT-TEST-FILE!$H+H* https://en.m.wikipedia.org/wiki/EICAR_test_file https://en.m.wikipedia.org/wiki/EICAR_test_file
- buildbuildbuild 5y ago
  I thought about this, but you're still stuck trusting the implementation unless you test with actual illegal data, which is often criminal and immoral to obtain. Example: How does a researcher test whether algorithmically-classified illegal imagery stored on user devices is being scanned and reported home to Apple's servers, and what those bounds of AI-classified criminality are? (presumably with respect to what is illegal in the user's jurisdiction) Testing by using a test phrase, like in a spam context, is inadequate here as a scanning system can trivially be architected to pass those publicly-known tests, while still overreaching into people's personal files and likely miscategorizing content and intent. If a user connects via a VPN to Russia for whatever reasons, does their personal content start getting reported to that country's law enforcement by their notion of what is illegal? Parents often have all sorts of sensitive photos around which are not held with exploitative intent. "Computer says arrest."
  - _fat_santa 5y ago
    > Example: How does a researcher test whether algorithmically-classified illegal imagery stored on user devices is being scanned and reported home to Apple's servers, and what those bounds of AI-classified criminality are? (presumably with respect to what is illegal in the user's jurisdiction) I'm not an expert in AI so this might be totally off base but I feel like you would be able to use an "intersection" of sorts for this type of detection. You detect children and pornography, the children portion trains it for age recognition and the porn portion trains it to see sexual acts. Slap those two together and you've got CSAM detection.
  - Brian_K_White 5y ago
    I always imagine aliens hearing about something like this and being stunned. "How can data be illegal?" "There are bad things, but how can you decide what is bad and show how it's bad without examining and discussing it?" You can only go so far merely alluding to things. Somewhere the rubber has to meet the road and you have to have concrete data and examples of anything you need to study or make any sort of tools or policy about. It's like parents not talking to kids about sex. You can avoid it most of the time because decorum, but if you take that to it's extreme you have just made your child both helpless and dangerous through ignorance. Somewhere along the way, you have to explicitly wallow directly in the mess of stuff you seek to avoid most of the time. That "seek to avoid" can only ever be "most of the time". It's insane and counter-productive to try to see that "most of the time" as an incomplete job and improve that to 100%. I guess in this case there will eventually be some sort of approved certified group. A child porn researcher or investigator license. Cool. Cops with special powers never abuse them, and inhibiting study to a select few has always yielded the best results for any subject, and a dozen approved good guys can easily stay ahead of the world of bad guys.
perihelions 5y ago
Obviously, definitionally, it's impossible to verify that server-side logic isn't doing something evil. (Local homeomorphic protocols count, when the secret logic is imported from remote servers). This is one reason FOSS is actually-important and actually-relevant. Isn't it valid to know exactly what your personal computer is doing, to be able to trust your own possessions? Richard Stallman was *never* crazy; his understanding of these issues is so cynical as to be shrill and off-putting, but that's well-calibrated to the severity of the issues at stake. You joke about anti-Putin memes. Here's a thought for well-calibrated cynics: Apple solemnly swears its hashes are attested by at least two independent countries. Russia and Belarus are two independent countries.
- azinman2 5y ago
  You mean one of the countries sales of devices just stopped in? And the other already was announced to be a US org? And you need the intersection of both?
  - perihelions 5y ago
    - "And the other already was announced to be a US org?" Then one rogue employee in a US org could be sufficient to get selective root to every Apple device everywhere? That's easy for a nation-state adversary. Here's demonstrated examples: MBS had US-based moles in Twitter corporate spying on Khashoggi [0], and Xi had Chinese-based Zoom employees spying on dissidents in America [1]. [0] https://www.npr.org/2019/11/06/777098293/2-former-twitter-employees-charged-with-spying-for-saudi-arabia https://www.npr.org/2019/11/06/777098293/2-former-twitter-em... [1] https://www.justice.gov/opa/pr/china-based-executive-us-telecommunications-company-charged-disrupting-video-meetings https://www.justice.gov/opa/pr/china-based-executive-us-tele... That second example is topical: the Chinese state used their Zoom assets to attempt to frame Americans for CSAM possession. - "As detailed in the complaint, Jin’s co-conspirators created fake email accounts and Company-1 accounts in the names of others, including PRC political dissidents, to fabricate evidence that the hosts of and participants in the meetings to commemorate the Tiananmen Square massacre were supporting terrorist organizations, inciting violence or distributing child pornography. The fabricated evidence falsely asserted that the meetings included discussions of child abuse or exploitation, terrorism, racism or incitements to violence, and sometimes included screenshots of the purported participants’ user profiles featuring, for example, a masked person holding a flag resembling that of the Islamic State terrorist group."
mike_d 5y ago
Research has to be done in partnership with the NCMEC, which in turn partners with the Department of Justice to run the database of known CSAM material.
deleted 5y ago
[deleted]

version_five 5y ago

By test it, do you mean see if the police show up at your door? If you know how it works, you just need a list of hashes and a way to find a collision which I believe exists. Otherwise, you're really just highlighting the problem with all closed source software, you don't really have a way to check what it does so you have to trust the vendor.

rootusrootus 5y ago
We already know that a hash collision doesn't get far enough to involve police showing up at your door, so a full test would take something more substantial.

chockchocschoir 5y ago

You don't have to test it against anti-Putin memes to see if it would work for anti-Putin memes. Algoritm would be something like: 1. Have image 2. Get hash of image 3. Get another hash from another similar image 4. Compare hashes The images themselves can be of whatever to see if it works as expected, they don't have to contain anti-Putin memes.

willcipriano 5y ago

Come up with something like: X51!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-CHILD-ABUSE-CONTENT-TEST-FILE!$H+H* https://en.m.wikipedia.org/wiki/EICAR_test_file https://en.m.wikipedia.org/wiki/EICAR_test_file

buildbuildbuild 5y ago
I thought about this, but you're still stuck trusting the implementation unless you test with actual illegal data, which is often criminal and immoral to obtain. Example: How does a researcher test whether algorithmically-classified illegal imagery stored on user devices is being scanned and reported home to Apple's servers, and what those bounds of AI-classified criminality are? (presumably with respect to what is illegal in the user's jurisdiction) Testing by using a test phrase, like in a spam context, is inadequate here as a scanning system can trivially be architected to pass those publicly-known tests, while still overreaching into people's personal files and likely miscategorizing content and intent. If a user connects via a VPN to Russia for whatever reasons, does their personal content start getting reported to that country's law enforcement by their notion of what is illegal? Parents often have all sorts of sensitive photos around which are not held with exploitative intent. "Computer says arrest."
- _fat_santa 5y ago
  > Example: How does a researcher test whether algorithmically-classified illegal imagery stored on user devices is being scanned and reported home to Apple's servers, and what those bounds of AI-classified criminality are? (presumably with respect to what is illegal in the user's jurisdiction) I'm not an expert in AI so this might be totally off base but I feel like you would be able to use an "intersection" of sorts for this type of detection. You detect children and pornography, the children portion trains it for age recognition and the porn portion trains it to see sexual acts. Slap those two together and you've got CSAM detection.
- Brian_K_White 5y ago
  I always imagine aliens hearing about something like this and being stunned. "How can data be illegal?" "There are bad things, but how can you decide what is bad and show how it's bad without examining and discussing it?" You can only go so far merely alluding to things. Somewhere the rubber has to meet the road and you have to have concrete data and examples of anything you need to study or make any sort of tools or policy about. It's like parents not talking to kids about sex. You can avoid it most of the time because decorum, but if you take that to it's extreme you have just made your child both helpless and dangerous through ignorance. Somewhere along the way, you have to explicitly wallow directly in the mess of stuff you seek to avoid most of the time. That "seek to avoid" can only ever be "most of the time". It's insane and counter-productive to try to see that "most of the time" as an incomplete job and improve that to 100%. I guess in this case there will eventually be some sort of approved certified group. A child porn researcher or investigator license. Cool. Cops with special powers never abuse them, and inhibiting study to a select few has always yielded the best results for any subject, and a dozen approved good guys can easily stay ahead of the world of bad guys.

perihelions 5y ago

Obviously, definitionally, it's impossible to verify that server-side logic isn't doing something evil. (Local homeomorphic protocols count, when the secret logic is imported from remote servers). This is one reason FOSS is actually-important and actually-relevant. Isn't it valid to know exactly what your personal computer is doing, to be able to trust your own possessions? Richard Stallman was *never* crazy; his understanding of these issues is so cynical as to be shrill and off-putting, but that's well-calibrated to the severity of the issues at stake. You joke about anti-Putin memes. Here's a thought for well-calibrated cynics: Apple solemnly swears its hashes are attested by at least two independent countries. Russia and Belarus are two independent countries.

azinman2 5y ago
You mean one of the countries sales of devices just stopped in? And the other already was announced to be a US org? And you need the intersection of both?
- perihelions 5y ago
  - "And the other already was announced to be a US org?" Then one rogue employee in a US org could be sufficient to get selective root to every Apple device everywhere? That's easy for a nation-state adversary. Here's demonstrated examples: MBS had US-based moles in Twitter corporate spying on Khashoggi [0], and Xi had Chinese-based Zoom employees spying on dissidents in America [1]. [0] https://www.npr.org/2019/11/06/777098293/2-former-twitter-employees-charged-with-spying-for-saudi-arabia https://www.npr.org/2019/11/06/777098293/2-former-twitter-em... [1] https://www.justice.gov/opa/pr/china-based-executive-us-telecommunications-company-charged-disrupting-video-meetings https://www.justice.gov/opa/pr/china-based-executive-us-tele... That second example is topical: the Chinese state used their Zoom assets to attempt to frame Americans for CSAM possession. - "As detailed in the complaint, Jin’s co-conspirators created fake email accounts and Company-1 accounts in the names of others, including PRC political dissidents, to fabricate evidence that the hosts of and participants in the meetings to commemorate the Tiananmen Square massacre were supporting terrorist organizations, inciting violence or distributing child pornography. The fabricated evidence falsely asserted that the meetings included discussions of child abuse or exploitation, terrorism, racism or incitements to violence, and sometimes included screenshots of the purported participants’ user profiles featuring, for example, a masked person holding a flag resembling that of the Islamic State terrorist group."

mike_d 5y ago

Research has to be done in partnership with the NCMEC, which in turn partners with the Department of Justice to run the database of known CSAM material.

deleted 5y ago

[deleted]