4 ms·

> This is an extremely good point. If the whole system, end to end, after all safeguards (e.g. human reviewers which can also make mistakes) has a one-in-a-bill

> This is an extremely good point. If the whole system, end to end, after all safeguards (e.g. human reviewers which can also make mistakes) has a one-in-a-billion chance to ruin a user's life, then statistically, we can expect 1-2 users to have their lives ruined. No one will have their lives ruined. If there's a false collision, someone at Apple has to look at the images as a final safeguard. If it's not actually CSAM then it won't ever make it to law enforcement.

tgsovlerkhgsel 5y ago
What probability do you ascribe to that reviewer clicking the wrong button, be it out of habit/zoning out (because the system usually shows them true positives), cheating (always clicking "yes" because it's usually correct and allows them to get paid without having to look at horrible images all day), mistake, wrong instructions (e.g. thinking that all images of children, or all porn including adult porn, should be flagged), confusion, or malice? 1 in 10? 1 in 100? 1 in 1000? Pick a number, you now have an estimate of how much room for error there is in the whole system. If you consider 0.15 lives ruined on average per year acceptable, and the reviewers have a 1-in-1000 error rate, then the rest of the system has to make less than 1-in-10-million mistakes per year. And I'm pretty sure 1-in-1000 for the reviewers is very, very optimistic, even if you do quality checks, control for fatigue, etc. On the other hand, hopefully there are some safeguards after the reviewers (e.g. human prosecutors who don't blindly rubber stamp). But my point is: The room for error in anything done at scale that has severe consequences is extremely small.
- robertoandred 5y ago
  Even if Apple’s manual review fails, there’s still the NCMEC’s review. There are several layers before anything goes to law enforcement.
  - dhosek 5y ago
    And presumably, at some point even if it makes it to a trial, the accused would be able to point to the flagged images to show that they aren't actually child porn. And a country in which this isn't possible, is likely one that is going to ruin people's lives just fine without Apple.

tgsovlerkhgsel 5y ago

What probability do you ascribe to that reviewer clicking the wrong button, be it out of habit/zoning out (because the system usually shows them true positives), cheating (always clicking "yes" because it's usually correct and allows them to get paid without having to look at horrible images all day), mistake, wrong instructions (e.g. thinking that all images of children, or all porn including adult porn, should be flagged), confusion, or malice? 1 in 10? 1 in 100? 1 in 1000? Pick a number, you now have an estimate of how much room for error there is in the whole system. If you consider 0.15 lives ruined on average per year acceptable, and the reviewers have a 1-in-1000 error rate, then the rest of the system has to make less than 1-in-10-million mistakes per year. And I'm pretty sure 1-in-1000 for the reviewers is very, very optimistic, even if you do quality checks, control for fatigue, etc. On the other hand, hopefully there are some safeguards after the reviewers (e.g. human prosecutors who don't blindly rubber stamp). But my point is: The room for error in anything done at scale that has severe consequences is extremely small.

robertoandred 5y ago
Even if Apple’s manual review fails, there’s still the NCMEC’s review. There are several layers before anything goes to law enforcement.
- dhosek 5y ago
  And presumably, at some point even if it makes it to a trial, the accused would be able to point to the flagged images to show that they aren't actually child porn. And a country in which this isn't possible, is likely one that is going to ruin people's lives just fine without Apple.