4 ms·

I get people's opinions of law enforcement are low, but do you really think that no one would question why a huge number of people have a single CSAM image on t

by slg 3y ago

I get people's opinions of law enforcement are low, but do you really think that no one would question why a huge number of people have a single CSAM image on their device especially when the hash for that image was just added to the database? Do you think that no one would question that maybe there is something wrong with that hash? Do you think that no one would look at the flagged image on any of those devices? I understand people's concerns with this tech, but this seems like a rather silly hypothetical to me.

Hizonner 3y ago
It's really about preventing images from circulating. Yes, the database maintainers would notice, but it could take them a while to get around to it. And they're not going to be eager to remove a collision, because that would effectively "legalize" the child porn member of the image pair. There are lots of images that could be useful to suppress temporarily. And if you actually succeed in suppressing the false-positive image, it may not get passed around very fast, and it won't get vastly more hits than real target images, so it will take longer for anybody to notice or care. There's also a possible end game where the child porn traders start perturbing their images to collide with really common images like flags, corporate logos, iconic movie stills, and whatever else. So now you either have to ban the US flag, or let this or that actual child porn image go. I don't actually think that the false hits would ruin very many lives in most places. But it's worth noticing that the original article was talking about authoritarian regimes repurposing the system without the consent of the database maintainers. In the Orbán example, it's possible that the system might flag you for child porn, but you might actually get arrested for sedition. And that continues to happen to people until the database maintainers pull the hash.
- slg 3y ago
  >Yes, the database maintainers would notice, but it could take them a while to get around to it. And they're not going to be eager to remove a collision, because that would effectively "legalize" the child porn member of the image pair. There are lots of images that could be useful to suppress temporarily. They may not know it immediately, but the actual CSAM image wouldn't actually be shared in any real numbers which removes much of the concern with "legalizing" the image. You can argue this makes this system ineffectual, but that also means this isn't repeatable since weakening the impact of this system would quickly result in Hungary losing the power to add hashes to the database. >And if you actually succeed in suppressing the false-positive image, it may not get passed around very fast, and it won't get vastly more hits than real target images, so it will take longer for anybody to notice or care. Who is sharing the real target image? Is Hungary now an active creator and distributor of actual CSAM in addition to manipulating the database? >There's also a possible end game where the child porn traders start perturbing their images to collide with really common images like flags, corporate logos, iconic movie stills, and whatever else. So now you either have to ban the US flag, or let this or that actual child porn image go. Once again, this behavior would get Hungary booted pretty quickly. Also it is important to remember these are hashes. Not all images of the US flag would trigger the system only that specific image that has a hash collision. >I don't actually think that the false hits would ruin very many lives in most places. But it's worth noticing that the original article was talking about authoritarian regimes repurposing the system without the consent of the database maintainers. In the Orbán example, it's possible that the system might flag you for child porn, but you might actually get arrested for sedition. And that continues to happen to people until the database maintainers pull the hash. But this isn't a closed system within that authoritarian regime. It leaves bread crumbs of this behavior out for everyone to see. If Hungary is going to arrest people on made up charges, they can do that anyway. I just think this in a complicated and ineffectual bullet that can only be fired once because it has a fingerprint of the person who fired it. That adds up to make this type of abuse less of a worry.
  - Hizonner 3y ago
    >> There's also a possible end game where the child porn traders [...] > Once again, this behavior would get Hungary booted pretty quickly. I'm sorry; I should have made myself clearer. In that paragraph, I've taken an aside and moved from Hungary (or any other government) as the adversary to child-porn-sharers-in-general as the adversary, and also changed the adversary goal. No matter what you do, you can't "boot" the child porn sharers, because they're the ones who actually define what images you're legitimately trying to block. > Also it is important to remember these are hashes. Not all images of the US flag would trigger the system only that specific image that has a hash collision. They're approximate perceptual hashes, designed to come up with close values on close pictures. The US flag has an officially defined appearance. You'll get the same hash for any two close-cropped, straight-on images of the flag, the kind you might embed in your Web site. They'll be at least as close as two reprocessed versions of the same child porn image. I was oversimplifying, though. You're not going to be able to tweak just any child porn image to make it hash like just any flag, unless you're willing to distort it into unrecognizability. And flags and logos might be bad candidates in general, because they're going to give DCT output that's wildly different than what you'll get from most photos. But if you had a relatively large library of child porn and a relatively large library of heavily-used effectively unbannable images of whatever kind, you should be able to find a lot of the child porn images that you can tweak to hash like one or another of the heavily used ones. ... and if you're generating fake child porn from scratch using ML, you can probably hack your ML model to bake the hash of one or another unbannable image into everything it creates. You could probably make those matches pretty damned close. So, once they got the whole thing down, they should be able to force the system to greatly tighten its match thresholds and/or deal with a really high false positive rate. In fact, they could probably make things bad enough that they could end up with a pretty large collection of child porn that the operators would be forced to completely exclude from detection. Now back on the original authoritarian threat model: > I just think this in a complicated and ineffectual bullet that can only be fired once because it has a fingerprint of the person who fired it. For the "authoritarian" purpose, you may be right... although I wouldn't be surprised if it stayed under the radar for longer than you think. If the image you want to suppress only circulates among your own people, and if you're the authority who receives and verifies the reports on your own people, then all you have to do is to keep the overall volume down enough that it doesn't make anybody suspicious enough to demand that you show them the reports you're getting. If you're a relatively small country, the number of people who share some local meme you care about may be quite a bit smaller than the number of people who share some new real child porn image.

by slg 3y ago

Hizonner 3y ago

It's really about preventing images from circulating. Yes, the database maintainers would notice, but it could take them a while to get around to it. And they're not going to be eager to remove a collision, because that would effectively "legalize" the child porn member of the image pair. There are lots of images that could be useful to suppress temporarily. And if you actually succeed in suppressing the false-positive image, it may not get passed around very fast, and it won't get vastly more hits than real target images, so it will take longer for anybody to notice or care. There's also a possible end game where the child porn traders start perturbing their images to collide with really common images like flags, corporate logos, iconic movie stills, and whatever else. So now you either have to ban the US flag, or let this or that actual child porn image go. I don't actually think that the false hits would ruin very many lives in most places. But it's worth noticing that the original article was talking about authoritarian regimes repurposing the system without the consent of the database maintainers. In the Orbán example, it's possible that the system might flag you for child porn, but you might actually get arrested for sedition. And that continues to happen to people until the database maintainers pull the hash.

slg 3y ago
>Yes, the database maintainers would notice, but it could take them a while to get around to it. And they're not going to be eager to remove a collision, because that would effectively "legalize" the child porn member of the image pair. There are lots of images that could be useful to suppress temporarily. They may not know it immediately, but the actual CSAM image wouldn't actually be shared in any real numbers which removes much of the concern with "legalizing" the image. You can argue this makes this system ineffectual, but that also means this isn't repeatable since weakening the impact of this system would quickly result in Hungary losing the power to add hashes to the database. >And if you actually succeed in suppressing the false-positive image, it may not get passed around very fast, and it won't get vastly more hits than real target images, so it will take longer for anybody to notice or care. Who is sharing the real target image? Is Hungary now an active creator and distributor of actual CSAM in addition to manipulating the database? >There's also a possible end game where the child porn traders start perturbing their images to collide with really common images like flags, corporate logos, iconic movie stills, and whatever else. So now you either have to ban the US flag, or let this or that actual child porn image go. Once again, this behavior would get Hungary booted pretty quickly. Also it is important to remember these are hashes. Not all images of the US flag would trigger the system only that specific image that has a hash collision. >I don't actually think that the false hits would ruin very many lives in most places. But it's worth noticing that the original article was talking about authoritarian regimes repurposing the system without the consent of the database maintainers. In the Orbán example, it's possible that the system might flag you for child porn, but you might actually get arrested for sedition. And that continues to happen to people until the database maintainers pull the hash. But this isn't a closed system within that authoritarian regime. It leaves bread crumbs of this behavior out for everyone to see. If Hungary is going to arrest people on made up charges, they can do that anyway. I just think this in a complicated and ineffectual bullet that can only be fired once because it has a fingerprint of the person who fired it. That adds up to make this type of abuse less of a worry.
- Hizonner 3y ago
  >> There's also a possible end game where the child porn traders [...] > Once again, this behavior would get Hungary booted pretty quickly. I'm sorry; I should have made myself clearer. In that paragraph, I've taken an aside and moved from Hungary (or any other government) as the adversary to child-porn-sharers-in-general as the adversary, and also changed the adversary goal. No matter what you do, you can't "boot" the child porn sharers, because they're the ones who actually define what images you're legitimately trying to block. > Also it is important to remember these are hashes. Not all images of the US flag would trigger the system only that specific image that has a hash collision. They're approximate perceptual hashes, designed to come up with close values on close pictures. The US flag has an officially defined appearance. You'll get the same hash for any two close-cropped, straight-on images of the flag, the kind you might embed in your Web site. They'll be at least as close as two reprocessed versions of the same child porn image. I was oversimplifying, though. You're not going to be able to tweak just any child porn image to make it hash like just any flag, unless you're willing to distort it into unrecognizability. And flags and logos might be bad candidates in general, because they're going to give DCT output that's wildly different than what you'll get from most photos. But if you had a relatively large library of child porn and a relatively large library of heavily-used effectively unbannable images of whatever kind, you should be able to find a lot of the child porn images that you can tweak to hash like one or another of the heavily used ones. ... and if you're generating fake child porn from scratch using ML, you can probably hack your ML model to bake the hash of one or another unbannable image into everything it creates. You could probably make those matches pretty damned close. So, once they got the whole thing down, they should be able to force the system to greatly tighten its match thresholds and/or deal with a really high false positive rate. In fact, they could probably make things bad enough that they could end up with a pretty large collection of child porn that the operators would be forced to completely exclude from detection. Now back on the original authoritarian threat model: > I just think this in a complicated and ineffectual bullet that can only be fired once because it has a fingerprint of the person who fired it. For the "authoritarian" purpose, you may be right... although I wouldn't be surprised if it stayed under the radar for longer than you think. If the image you want to suppress only circulates among your own people, and if you're the authority who receives and verifies the reports on your own people, then all you have to do is to keep the overall volume down enough that it doesn't make anybody suspicious enough to demand that you show them the reports you're getting. If you're a relatively small country, the number of people who share some local meme you care about may be quite a bit smaller than the number of people who share some new real child porn image.