3 ms·

I get the concern, but is there any evidence to suggest technology based solutions by themselves lead to any of those obviously bad false positive outcomes? Ha

I get the concern, but is there any evidence to suggest technology based solutions by themselves lead to any of those obviously bad false positive outcomes? Has anybody ever been send to prison, placed on a sex offender list, or even been charged based purely on the output of an automated evaluation that you posses CSAM or some other illegal material? One can certainly imagine the dystopian horror of such a scenario, but whatever the specific technology in question, it's difficult to imagine it playing out in the current legal system of any of country that isn't already based around just semi-randomly throwing people in jail.

omginternets 4y ago
>I get the concern, but is there any evidence to suggest technology based solutions by themselves lead to any of those obviously bad false positive outcomes? Yes. In fact, it's mathematically provable that these false positives must happen, unless you never label anything as CSAM. Welcome to signal detection theory; it's a really cool, and useful field. I recommend this book: https://www.taylorfrancis.com/books/mono/10.4324/9781410611949/primer-signal-detection-theory-mcnicol https://www.taylorfrancis.com/books/mono/10.4324/97814106119...
- JimDabell 4y ago
  > it's mathematically provable that these false positives must happen This is not true. It’s mathematically provable that false positives are possible, not that they must happen. In fact, the chances of collisions happening are astronomically low and basically nil when you use them in the way Apple was proposing (two separate hashes; several matches required).
  - imchillyb 4y ago
    > This is not true. It’s mathematically provable that false positives are possible, not that they must happen. In fact, the chances of collisions happening are astronomically low and basically nil when you use them in the way Apple was proposing @JimDabell WTF are you on about?! That IS NOT how math works. If millions of images are uploaded per day, there are GUARANTEED false positives. It's not a /maybe/ but an absolute fucking certainty. This guy's a shill...
    - omginternets 4y ago
      I agree his overconfident tone isn't helping, but let's not assume malevolence. I think his argument is that the FP rate can be set to a vanishingly small value, and so we can in theory proceed as if it were effectively zero, as we do with hash collisions. This reasoning is false for two reasons: one theoretical, and one practical. The theoretical reason is that a nonzero FP rate means that as the number of samples approaches infinity, a false-positive is guaranteed to occur (as you rightly point out). The practical reason is that complex image classifiers do not exhibit what any sane person would consider a "vanishingly small" FP rate, especially given the volume of samples being processed in this case.
  - omginternets 4y ago
    I'm sorry, you are mistaken. You can set the FP rate arbitrarily low, but they will eventually happen. "Basically nil" is not the same as "actually nil". I also suspect you are grossly exaggerating the d', and grossly understating the FP rate of Apple's solution, as I am unaware of any image processing system whose FP rate is "basically nil". Your assertion warrants hard numbers.
    - JimDabell 4y ago
      > You can set the FP rate arbitrarily low, but they will eventually happen. "Basically nil" is not the same as "actually nil". I know “basically nil” is not the same as “actually nil”, that’s why I said “basically nil” and not just “nil”. It is “nil for all practical purposes”. This is a good enough standard for software the whole world uses all day every day to depend upon – otherwise hashes would be pretty useless. “Eventually” needs clarification here. Of course, if you spend from now until the heat death of the universe iterating through the search space you could get there. But that doesn’t mean that in practical use, such a collision is guaranteed as you claim. This system could have run its entire lifetime without a collision. You are mistaking “if you cover the entire search space” for what happens in the real world. Do you understand that there is not one, but two hashes, and that they both need to collide simultaneously for each image? And do you understand that this has to happen not just a single time, but for several images on your system? That’s why I specifically said “basically nil when you use them in the way Apple was proposing” and described how it worked. Everybody with even a basic grasp of what hashes are understand that collisions are certain when considering the entire search space. But that doesn’t correspond with what happens in practical terms, which is why hashing is actually useful in general, and I think people are getting too fixated on “hashes can have collisions” to notice the other properties of the system. Apple can tune the false positive rate by varying the number of matches necessary to flag an account. They say they chose a threshold that would result in a false positive rate of one in a trillion. Are you saying they got their maths wrong or were lying, and that’s actually impossible? Because there’s only eight billion people on the planet and most of them don’t have Apple accounts, so if one in a trillion is accurate, then it seems entirely possible for this system to have run indefinitely without a single false positive. Remember – I said “basically nil when you use them in the way Apple was proposing (two separate hashes; several matches required)”, and not that a single hash function wouldn’t ever produce a single collision for a single image. Do you really disagree with that?
TeMPOraL 4y ago
> is there any evidence to suggest technology based solutions by themselves lead to any of those obviously bad false positive outcomes? Specific to this type of technology? YouTube and their Content ID comes to mind - stories abound about people getting random videos flagged or demonetized because of a match to something in a background, or false positive match to their own original rendition, or to their own original rendition of their own work they own copyright to, or even against noise. Some of those stories end up harming those YouTubers financially. It's happening often enough that pretty much every channel I've watched has a video commenting or complaining about this at this point. Then, related, random bans of Google accounts, or people having their apps kicked off Apple's App store or Google's Play Store for seemingly no reason (following some of those stories over the years, I estimate it's 3:1 false positive to the person actually violating ToS). Google account bans, in particular, can easily make your life very difficult for some time, or even kill your company, and they have a nasty feature of being transitive (if you have multiple Google Accounts connected in an obvious way, e.g. by recovery number, having one banned tends to be followed by having the rest banned too). Point being, those big cloud companies don't have particularly good reputation when it comes to algorithmic moderation. So when one of those companies wants to deploy another set of automated scans, with a twist that a false positive now has a chance of quickly and irreversibly derailing your whole life, it's hopefully understandable why people are apprehensive. > whatever the specific technology in question, it's difficult to imagine it playing out in the current legal system of any of country that isn't already based around just semi-randomly throwing people in jail. With this particular crime, you don't have to go into jail to have your life destroyed. You don't even have to go to court. All it takes is that a rumor leaks out that you're accused of consuming or producing CSAM - it'll do irreparable damage to your relationship with others, as they're always be wondering whether there wasn't something to those rumors. In a way, this technology being criticized so loudly and broadly is a form of mitigating the damage it could cause: the more people are aware it's liable to spurious false positives, the more chance the victims of such false positives have that others will believe them.

omginternets 4y ago

>I get the concern, but is there any evidence to suggest technology based solutions by themselves lead to any of those obviously bad false positive outcomes? Yes. In fact, it's mathematically provable that these false positives must happen, unless you never label anything as CSAM. Welcome to signal detection theory; it's a really cool, and useful field. I recommend this book: https://www.taylorfrancis.com/books/mono/10.4324/9781410611949/primer-signal-detection-theory-mcnicol https://www.taylorfrancis.com/books/mono/10.4324/97814106119...

JimDabell 4y ago
> it's mathematically provable that these false positives must happen This is not true. It’s mathematically provable that false positives are possible, not that they must happen. In fact, the chances of collisions happening are astronomically low and basically nil when you use them in the way Apple was proposing (two separate hashes; several matches required).
- imchillyb 4y ago
  > This is not true. It’s mathematically provable that false positives are possible, not that they must happen. In fact, the chances of collisions happening are astronomically low and basically nil when you use them in the way Apple was proposing @JimDabell WTF are you on about?! That IS NOT how math works. If millions of images are uploaded per day, there are GUARANTEED false positives. It's not a /maybe/ but an absolute fucking certainty. This guy's a shill...
  - omginternets 4y ago
    I agree his overconfident tone isn't helping, but let's not assume malevolence. I think his argument is that the FP rate can be set to a vanishingly small value, and so we can in theory proceed as if it were effectively zero, as we do with hash collisions. This reasoning is false for two reasons: one theoretical, and one practical. The theoretical reason is that a nonzero FP rate means that as the number of samples approaches infinity, a false-positive is guaranteed to occur (as you rightly point out). The practical reason is that complex image classifiers do not exhibit what any sane person would consider a "vanishingly small" FP rate, especially given the volume of samples being processed in this case.
- omginternets 4y ago
  I'm sorry, you are mistaken. You can set the FP rate arbitrarily low, but they will eventually happen. "Basically nil" is not the same as "actually nil". I also suspect you are grossly exaggerating the d', and grossly understating the FP rate of Apple's solution, as I am unaware of any image processing system whose FP rate is "basically nil". Your assertion warrants hard numbers.
  - JimDabell 4y ago
    > You can set the FP rate arbitrarily low, but they will eventually happen. "Basically nil" is not the same as "actually nil". I know “basically nil” is not the same as “actually nil”, that’s why I said “basically nil” and not just “nil”. It is “nil for all practical purposes”. This is a good enough standard for software the whole world uses all day every day to depend upon – otherwise hashes would be pretty useless. “Eventually” needs clarification here. Of course, if you spend from now until the heat death of the universe iterating through the search space you could get there. But that doesn’t mean that in practical use, such a collision is guaranteed as you claim. This system could have run its entire lifetime without a collision. You are mistaking “if you cover the entire search space” for what happens in the real world. Do you understand that there is not one, but two hashes, and that they both need to collide simultaneously for each image? And do you understand that this has to happen not just a single time, but for several images on your system? That’s why I specifically said “basically nil when you use them in the way Apple was proposing” and described how it worked. Everybody with even a basic grasp of what hashes are understand that collisions are certain when considering the entire search space. But that doesn’t correspond with what happens in practical terms, which is why hashing is actually useful in general, and I think people are getting too fixated on “hashes can have collisions” to notice the other properties of the system. Apple can tune the false positive rate by varying the number of matches necessary to flag an account. They say they chose a threshold that would result in a false positive rate of one in a trillion. Are you saying they got their maths wrong or were lying, and that’s actually impossible? Because there’s only eight billion people on the planet and most of them don’t have Apple accounts, so if one in a trillion is accurate, then it seems entirely possible for this system to have run indefinitely without a single false positive. Remember – I said “basically nil when you use them in the way Apple was proposing (two separate hashes; several matches required)”, and not that a single hash function wouldn’t ever produce a single collision for a single image. Do you really disagree with that?

TeMPOraL 4y ago

> is there any evidence to suggest technology based solutions by themselves lead to any of those obviously bad false positive outcomes? Specific to this type of technology? YouTube and their Content ID comes to mind - stories abound about people getting random videos flagged or demonetized because of a match to something in a background, or false positive match to their own original rendition, or to their own original rendition of their own work they own copyright to, or even against noise. Some of those stories end up harming those YouTubers financially. It's happening often enough that pretty much every channel I've watched has a video commenting or complaining about this at this point. Then, related, random bans of Google accounts, or people having their apps kicked off Apple's App store or Google's Play Store for seemingly no reason (following some of those stories over the years, I estimate it's 3:1 false positive to the person actually violating ToS). Google account bans, in particular, can easily make your life very difficult for some time, or even kill your company, and they have a nasty feature of being transitive (if you have multiple Google Accounts connected in an obvious way, e.g. by recovery number, having one banned tends to be followed by having the rest banned too). Point being, those big cloud companies don't have particularly good reputation when it comes to algorithmic moderation. So when one of those companies wants to deploy another set of automated scans, with a twist that a false positive now has a chance of quickly and irreversibly derailing your whole life, it's hopefully understandable why people are apprehensive. > whatever the specific technology in question, it's difficult to imagine it playing out in the current legal system of any of country that isn't already based around just semi-randomly throwing people in jail. With this particular crime, you don't have to go into jail to have your life destroyed. You don't even have to go to court. All it takes is that a rumor leaks out that you're accused of consuming or producing CSAM - it'll do irreparable damage to your relationship with others, as they're always be wondering whether there wasn't something to those rumors. In a way, this technology being criticized so loudly and broadly is a form of mitigating the damage it could cause: the more people are aware it's liable to spurious false positives, the more chance the victims of such false positives have that others will believe them.