3 ms·

Regardless of how you feel about content moderation, 48 hours is a ridiculously long time given what AI can do today. That “bad” image could have been propagat

Regardless of how you feel about content moderation, 48 hours is a ridiculously long time given what AI can do today. That “bad” image could have been propagated around the world to millions of people in that time. It can and should be removed in minutes because AI can evaluate the “bad” image quickly and a human moderator isn’t required anymore. However, the compute costs would eat into profits… Again, I’m not judging about content moderation, but this is an extremely weak initiative.

Manuel_D 8mo ago
The issue is that if you need to achieve 0% false negatives, you're going to get a lot of false positives.
- timr 8mo ago
  Another, related issue is that the takedown mechanism becomes a de facto censorship mechanism, as anyone who has dealt with DMCA takedowns and automated detectors can tell you. Someone reports something for Special Pleading X, and you (the operator) have to ~instantly take down the thing, by law. There is never an equally efficient mechanism to push back against abuses -- there can't be, because it exposes the operator to legal risk in doing so. So you effectively have a one-sided mechanism for removal of unwanted content. Maybe this is fine for "revenge porn", but even ignoring the slippery slope argument (which is real -- we already have these kinds of rules for copyrighted content!) it's not so easy to cleanly define "revenge porn".
  - bluGill 8mo ago
    DMCA isn't directly that bad. DMCA is under penalty of perjury, so false take downs are rare. The problem is most take downs are not actually DMCA, they are some other non-legal process that isn't under any legal penalty. Though if it ever happens to you I suspect you have a good case against whoever did this - but the lawyer costs will far exceed your total gain. (as in spend $30 million or more to collect $100). Either we need enough people affected by a false non-DMCA take down that a class action can work (you get $0.50 but at least they pay something), or we need legal reform so that all take downs against a third party are ???
    - timr 8mo ago
      > DMCA is under penalty of perjury, so false take downs are rare. Maybe true with the platonic ideal "DMCA takedown letter" (though these are rarely litigated, so who really knows), but as you note, they're incredibly common with things like the automated systems that scan for music in videos (and which actually are related to DMCA takedowns), "bad words" and the like. > The problem is most take downs are not actually DMCA, they are some other non-legal process that isn't under any legal penalty. It's true that most takedowns in the US aren't under DMCA, but even that once-limited process has metastasized into large, fully automated content scanning systems that proactively take down huge amounts of content without much recourse. Companies do this to avoid liability as part of safe harbor laws, or just to curry favor with powerful interests. We're talking about US laws here, but in general, these kinds of instant-takedown laws become huge loopholes in whatever free speech provisions a country might have. The asymmetric exercise of rights essentially guarantees abuse.
    - worthless-trash 8mo ago
      I believe google issues legitimate dmca takedowns for copyright strikes, even when there is no infringement. They put the work to defend the strike on the apparent commiter, often with little to no detail. While the false takedown may be rare. Using dmca as a mechanism to inflict pain where no copyright infringement has taken place is indeed common enough that it happens to small time youtubers like myself and others I have talked to.
Ray20 8mo ago
Regardless of how you feel about content moderation, we are talking about a situation where the government is DEMANDING corporations to implement automated, totalitarian surveillance tools. This is the key factor here. The next step would be for the government to demand direct access to these tools. Then the government would be able to carry out holocausts against any ethnic group, only 10 times more effectively and inevitably than Hitler did.
- oneeyedpigeon 8mo ago
  Are you conflating this specific principle with the much wider Online Safety Act? Because, while the latter has certain privacy-undermining elements to it, I'm not sure how asking social media companies to take down content has anything to do with 'surveillance'.
  - Ray20 8mo ago
    No, I'm not conflating. The point here is that for taking content down in the way they want it, the implementation of totalitarian surveillance tools is necessary.
- Manuel_D 8mo ago
  As bad as I think this law is, this isn't demanding any degree of surveillance in the sense that real human beings have their information or activity tracked. This is mandating taking down content, not surveiling anyone.
  - Ray20 8mo ago
    > This is mandating taking down content, not surveiling anyone. As far as I understand, it precisely mandating to monitor EVERYONE. They are not talking about removing a specific image from the platform based on its hash or something. They are talking about actions that involve automated analysis of all content on the platform for patterns arbitrarily specified by the government. The technologies discussed differ from totalitarian surveillance by simply toggling a single flag on the platform, and are indistinguishable from such surveillance for the user.
Ukv 8mo ago
> It can and should be removed in minutes because AI can evaluate the “bad” image quickly and a human moderator isn’t required anymore. CSAM can be detected through hashes or a machine-learning image classifier (with some false positives), whereas whether an image was shared nonconsensually seems like it'd often require context that is not in the image itself, possibly contacting the parties involved.
- pjc50 8mo ago
  Indeed. It seems that the process being described is some kind of one-stop portal, operated by or for OFCOM or the police, where someone can attest "this is a nonconsensual intimate image of me" (hopefully in some legally binding way!), triggering a cross-system takedown. Not all that dissimilar to DMCA.
- thaumasiotes 8mo ago
  > CSAM can be detected through hashes or a machine-learning image classifier (with some false positives), whereas Everything can be detected "with some false positives". If you're happy with "with some false positives", why would you need any context?
  - deleted 8mo ago
    [deleted]
- voidUpdate 8mo ago
  I would not want to be the supervisor that has to review any CSAM positives to check for false ones

Manuel_D 8mo ago

The issue is that if you need to achieve 0% false negatives, you're going to get a lot of false positives.

timr 8mo ago
Another, related issue is that the takedown mechanism becomes a de facto censorship mechanism, as anyone who has dealt with DMCA takedowns and automated detectors can tell you. Someone reports something for Special Pleading X, and you (the operator) have to ~instantly take down the thing, by law. There is never an equally efficient mechanism to push back against abuses -- there can't be, because it exposes the operator to legal risk in doing so. So you effectively have a one-sided mechanism for removal of unwanted content. Maybe this is fine for "revenge porn", but even ignoring the slippery slope argument (which is real -- we already have these kinds of rules for copyrighted content!) it's not so easy to cleanly define "revenge porn".
- bluGill 8mo ago
  DMCA isn't directly that bad. DMCA is under penalty of perjury, so false take downs are rare. The problem is most take downs are not actually DMCA, they are some other non-legal process that isn't under any legal penalty. Though if it ever happens to you I suspect you have a good case against whoever did this - but the lawyer costs will far exceed your total gain. (as in spend $30 million or more to collect $100). Either we need enough people affected by a false non-DMCA take down that a class action can work (you get $0.50 but at least they pay something), or we need legal reform so that all take downs against a third party are ???
  - timr 8mo ago
    > DMCA is under penalty of perjury, so false take downs are rare. Maybe true with the platonic ideal "DMCA takedown letter" (though these are rarely litigated, so who really knows), but as you note, they're incredibly common with things like the automated systems that scan for music in videos (and which actually are related to DMCA takedowns), "bad words" and the like. > The problem is most take downs are not actually DMCA, they are some other non-legal process that isn't under any legal penalty. It's true that most takedowns in the US aren't under DMCA, but even that once-limited process has metastasized into large, fully automated content scanning systems that proactively take down huge amounts of content without much recourse. Companies do this to avoid liability as part of safe harbor laws, or just to curry favor with powerful interests. We're talking about US laws here, but in general, these kinds of instant-takedown laws become huge loopholes in whatever free speech provisions a country might have. The asymmetric exercise of rights essentially guarantees abuse.
  - worthless-trash 8mo ago
    I believe google issues legitimate dmca takedowns for copyright strikes, even when there is no infringement. They put the work to defend the strike on the apparent commiter, often with little to no detail. While the false takedown may be rare. Using dmca as a mechanism to inflict pain where no copyright infringement has taken place is indeed common enough that it happens to small time youtubers like myself and others I have talked to.

Ray20 8mo ago

Regardless of how you feel about content moderation, we are talking about a situation where the government is DEMANDING corporations to implement automated, totalitarian surveillance tools. This is the key factor here. The next step would be for the government to demand direct access to these tools. Then the government would be able to carry out holocausts against any ethnic group, only 10 times more effectively and inevitably than Hitler did.

oneeyedpigeon 8mo ago
Are you conflating this specific principle with the much wider Online Safety Act? Because, while the latter has certain privacy-undermining elements to it, I'm not sure how asking social media companies to take down content has anything to do with 'surveillance'.
- Ray20 8mo ago
  No, I'm not conflating. The point here is that for taking content down in the way they want it, the implementation of totalitarian surveillance tools is necessary.
Manuel_D 8mo ago
As bad as I think this law is, this isn't demanding any degree of surveillance in the sense that real human beings have their information or activity tracked. This is mandating taking down content, not surveiling anyone.
- Ray20 8mo ago
  > This is mandating taking down content, not surveiling anyone. As far as I understand, it precisely mandating to monitor EVERYONE. They are not talking about removing a specific image from the platform based on its hash or something. They are talking about actions that involve automated analysis of all content on the platform for patterns arbitrarily specified by the government. The technologies discussed differ from totalitarian surveillance by simply toggling a single flag on the platform, and are indistinguishable from such surveillance for the user.

Ukv 8mo ago

> It can and should be removed in minutes because AI can evaluate the “bad” image quickly and a human moderator isn’t required anymore. CSAM can be detected through hashes or a machine-learning image classifier (with some false positives), whereas whether an image was shared nonconsensually seems like it'd often require context that is not in the image itself, possibly contacting the parties involved.

pjc50 8mo ago
Indeed. It seems that the process being described is some kind of one-stop portal, operated by or for OFCOM or the police, where someone can attest "this is a nonconsensual intimate image of me" (hopefully in some legally binding way!), triggering a cross-system takedown. Not all that dissimilar to DMCA.
thaumasiotes 8mo ago
> CSAM can be detected through hashes or a machine-learning image classifier (with some false positives), whereas Everything can be detected "with some false positives". If you're happy with "with some false positives", why would you need any context?
- deleted 8mo ago
  [deleted]
voidUpdate 8mo ago
I would not want to be the supervisor that has to review any CSAM positives to check for false ones