8 ms·

It seems like the problem with many of these tech companies (and many modern companies in general) is that they've scaled faster than they have the ability to o

It seems like the problem with many of these tech companies (and many modern companies in general) is that they've scaled faster than they have the ability to operate properly. "How can we possibly moderate properly without AI?" is really just another way of saying "We don't want to pay the money it's going to cost to moderate effectively and we're going to cross our fingers that we can solve problems caused by technology with more technology." That failure to operate is causing harm but the kind of harm it causes is hard for the public or politicians to understand, so it goes on unaddressed.

musingsole 4y ago
I don't think it's even the money anymore. I think technologists are convinced they are ALMOST there with automoderation systems...just a few more data points and another thousand hyperparameters and it'll be perfect! I think they've yet learn to recognize the good because they're too busy dreaming of the perfect.
vincvinc 4y ago
I think this is a fair assessment. Big social media companies are not incentivized to care about psychological and social side effects of their product. A bit like cigarette companies offloading externalities onto society, but much harder to specifically identify and address.
ISL 4y ago
Do you think that Twitter could possibly survive as a company without large amounts of automated moderation?
- gambiting 4y ago
  It couldn't, but what all of these companies could absolutely do is pair automated moderation with human review. When AI flags something as innapropriate, a human has to sign it off before it's removed or the account suspended. The only exception for it is when content matches a known hash for illegal content - then it should be removed automatically.
  - ISL 4y ago
    How many automated-review actions do you think Twitter performs in a day that would require such a review? How many seconds might each human-review take? What false-positive rate is acceptable on that human-review?
    - gambiting 4y ago
      Is that an argument that they shouldn't do it then? Because right now, if this article is to be believed, they don't do manual review even on appeals. That's obviously not ok.
      - Kalium 4y ago
        As other commenters point out, that characterization is not not accurate. There were most definitely humans involved, including on appeal. Does that make things OK? I suppose that depends on whether you object more to the decision or the process.
- lapcat 4y ago
  No. And that's why Twitter should not exist as a company. If they were legally required to have human moderation, they would go out of business. And IMO the world would be better off.
  - edanm 4y ago
    I think if the phone companies can't monitor every conversation to make sure no one is planning a crime, they're clearly not actually able to scale in a correct manner and therefore we shouldn't have phones. (Obviously a different situation, but by what principle is it different?)
    - lapcat 4y ago
      > Obviously a different situation, but by what principle is it different? The principle is that one is a public conversation, and the other is private. Private conversations between two people should not be moderated. Twitter should not auto-moderate Direct Messages, and they don't AFAIK.
- mantas 4y ago
  Do we need Twitter that much? Is it worth ruining the society in the name of having twitter or a similar social media? 90s and 00s with all sorts of independent discussion boards were great. Somehow moderation kept up without crappy AI. I can’t see a single benefit of Twitter/FB/etc over fragmented forums as a user.
  - bandyaboot 4y ago
    I’m guessing the “somehow” has something to do with the order of magnitude fewer users on the internet.
    - mantas 4y ago
      More like forums were fragmented. Topics, geographically etc. And politics was either sophisticated forums or just a byproduct of those fragmented topics. More users, many of them lurkers, would just result in more fragmentation and niches in that old world. The old forums was safe from feeling of censorship too. If you didn’t like mods too much, you could not easily go to another forum or start your own.
      - hinkley 4y ago
        With fragmentation, several other things also happened. One, getting banned from one community was not the end of the world. You could join another. Two, because the forums were small, if you got banned from one, you couldn’t seek sympathy by complaining to the entire internet about the unfairness, because nobody knew about that community so getting them to care was a challenge. And three, you couldn’t complain to the entire internet because there was no soapbox large enough to do it. Only your friends and the new community had to listen to you complain. On the flip side, an angry user with some technical abilities could poison the well by trying to take your community offline. Whether they succeed or not depends on who has more patience, the custodian or the manchild throwing an extended tantrum. That is at least a one place where centralization helps. Getting resilience set up is a full time job. Though I suppose you could say that’s an area where technology might actually help.
mikkergp 4y ago
When you say moderate without AI are you suggesting human moderation? This is not really a good solution to this problem either. Paying people to traumatize themselves by watching video after video or reading tweet after tweet of the worst content society has to offer is dehumanizing.
- mordae 4y ago
  With good moderation it won't spiral out of control that much.
  - carbadtraingood 4y ago
    What? Even with the best moderation, humans will periodically brigade a site with awful content because they think it's funny to do so.
- troymc 4y ago
  AFAIK, most of the moderation on Reddit is done by unpaid humans, not AI. Reddit isn't perfect, of course, but their approach does scale.
  - kkklloufffr 4y ago
    Moderators on Reddit are extremely petty and vindictive and I have hypothesized for a long time that this is because they are not paid with money.
    - vintermann 4y ago
      Yes 1. If they didn't get something out of it, they wouldn't be doing it. 2. They don't get money out of it. 3. So...
  - galleywest200 4y ago
    As someone who helps moderate a very large subreddit I can assure you we write our own bots to help. A lot of stuff is manually reviewed "in queue" and we review any complains in modmail manually. But there are too many comment for us unpaid internet janitors to not use our own robots. Also since we are not paid our caring levels are lower, thus heavy use of regex and automation.
mod 4y ago
Per the article, there was a human involved in an appeal for this matter.
- vintermann 4y ago
  Per the article, per Twitter. It sounds unlikely. Though it could be that there's a guy paid to look over stuff the AI flags. If that's the same guy who does the appeals, they might reject the appeal in order to hide that they didn't actually look at it before banning (and hope the complainant just goes away quietly. They probably often do.)
TheRealPomax 4y ago
It's not "want", Twitter (or youtube, or facebook, etc. etc.) can't in a million years afford content moderation because of the sheer firehose of content. They'd literally have to employ everyone on the planet to get enough moderators in every language for every tweet, post, video, etc. on the platform. And then only one company would have enough moderators. Moderators are a finite resource, there aren't even enough humans alive to moderate all content on all platforms. But, the law goes "well figure something out, you have to moderate", and now we've by law guaranteed that whatever solution that gets implemented to follow the letter of the law is going to be shit, because it's by definition going to be automated content moderation, and given the volumes involved, it's modern neural-net based because that's the only technology that's even remotely shown it has an over 50% success rate. Well done, us.
- ShamelessC 4y ago
  My frustration isn't really rooted in legality and has more to do with my personal beliefs about tech and unregulated capitalism. I've worked in tech a bit - I know how this goes. Some company wants to get a massive valuation by hitting DAU's in the millions, but you can't do maintenance/support with that many users in a single org even if you wanted to. As an engineer, I can totally relate to the feeling of "fuck support - that sounds awful", and I struggle to blame _them_ because, well - like I said, even if they tried they couldn't possible cover that much work due to an inability to scale their organization. Psychologically? It's pretty easy to get pissed off at these companies when this happens. These companies are growth-obsessed and the ostensible philosophy is always "making the world a better place". STFU and do your jobs or don't build a product that can't scale and try to pretend it's scaling just fine.
- lifeformed 4y ago
  You don't need human eyes on every single tweet, just need enough people to review the flagged content. You could still have algorithms flag things and rank them by confidence levels. You could also hide the content first and queue it for human review instead of auto-banning the user. It's a lot of work, but I feel like there's a lot more that could be done.
deleted 4y ago
[deleted]
pyuser583 4y ago
One complaint is that moderation gets worse when dealing with less common languages/cultures. Not long ago there as an HN post about someone being banned for being Syrian (sanctions) when they were speaking in Syriac. For a non-native English speaker, Syrian (Arabic) and Syriac (an ancient language not spoken for over a thousand years) is really hard to know. There are going to similar mistakes in every language. Getting a native speaker for some languages is borderline impossible. Some languages don’t even have native speakers (Syriac, for example).

musingsole 4y ago

I don't think it's even the money anymore. I think technologists are convinced they are ALMOST there with automoderation systems...just a few more data points and another thousand hyperparameters and it'll be perfect! I think they've yet learn to recognize the good because they're too busy dreaming of the perfect.

vincvinc 4y ago

I think this is a fair assessment. Big social media companies are not incentivized to care about psychological and social side effects of their product. A bit like cigarette companies offloading externalities onto society, but much harder to specifically identify and address.

ISL 4y ago

Do you think that Twitter could possibly survive as a company without large amounts of automated moderation?

gambiting 4y ago
It couldn't, but what all of these companies could absolutely do is pair automated moderation with human review. When AI flags something as innapropriate, a human has to sign it off before it's removed or the account suspended. The only exception for it is when content matches a known hash for illegal content - then it should be removed automatically.
- ISL 4y ago
  How many automated-review actions do you think Twitter performs in a day that would require such a review? How many seconds might each human-review take? What false-positive rate is acceptable on that human-review?
  - gambiting 4y ago
    Is that an argument that they shouldn't do it then? Because right now, if this article is to be believed, they don't do manual review even on appeals. That's obviously not ok.
    - Kalium 4y ago
      As other commenters point out, that characterization is not not accurate. There were most definitely humans involved, including on appeal. Does that make things OK? I suppose that depends on whether you object more to the decision or the process.
lapcat 4y ago
No. And that's why Twitter should not exist as a company. If they were legally required to have human moderation, they would go out of business. And IMO the world would be better off.
- edanm 4y ago
  I think if the phone companies can't monitor every conversation to make sure no one is planning a crime, they're clearly not actually able to scale in a correct manner and therefore we shouldn't have phones. (Obviously a different situation, but by what principle is it different?)
  - lapcat 4y ago
    > Obviously a different situation, but by what principle is it different? The principle is that one is a public conversation, and the other is private. Private conversations between two people should not be moderated. Twitter should not auto-moderate Direct Messages, and they don't AFAIK.
mantas 4y ago
Do we need Twitter that much? Is it worth ruining the society in the name of having twitter or a similar social media? 90s and 00s with all sorts of independent discussion boards were great. Somehow moderation kept up without crappy AI. I can’t see a single benefit of Twitter/FB/etc over fragmented forums as a user.
- bandyaboot 4y ago
  I’m guessing the “somehow” has something to do with the order of magnitude fewer users on the internet.
  - mantas 4y ago
    More like forums were fragmented. Topics, geographically etc. And politics was either sophisticated forums or just a byproduct of those fragmented topics. More users, many of them lurkers, would just result in more fragmentation and niches in that old world. The old forums was safe from feeling of censorship too. If you didn’t like mods too much, you could not easily go to another forum or start your own.
    - hinkley 4y ago
      With fragmentation, several other things also happened. One, getting banned from one community was not the end of the world. You could join another. Two, because the forums were small, if you got banned from one, you couldn’t seek sympathy by complaining to the entire internet about the unfairness, because nobody knew about that community so getting them to care was a challenge. And three, you couldn’t complain to the entire internet because there was no soapbox large enough to do it. Only your friends and the new community had to listen to you complain. On the flip side, an angry user with some technical abilities could poison the well by trying to take your community offline. Whether they succeed or not depends on who has more patience, the custodian or the manchild throwing an extended tantrum. That is at least a one place where centralization helps. Getting resilience set up is a full time job. Though I suppose you could say that’s an area where technology might actually help.

mikkergp 4y ago

When you say moderate without AI are you suggesting human moderation? This is not really a good solution to this problem either. Paying people to traumatize themselves by watching video after video or reading tweet after tweet of the worst content society has to offer is dehumanizing.

mordae 4y ago
With good moderation it won't spiral out of control that much.
- carbadtraingood 4y ago
  What? Even with the best moderation, humans will periodically brigade a site with awful content because they think it's funny to do so.
troymc 4y ago
AFAIK, most of the moderation on Reddit is done by unpaid humans, not AI. Reddit isn't perfect, of course, but their approach does scale.
- kkklloufffr 4y ago
  Moderators on Reddit are extremely petty and vindictive and I have hypothesized for a long time that this is because they are not paid with money.
  - vintermann 4y ago
    Yes 1. If they didn't get something out of it, they wouldn't be doing it. 2. They don't get money out of it. 3. So...
- galleywest200 4y ago
  As someone who helps moderate a very large subreddit I can assure you we write our own bots to help. A lot of stuff is manually reviewed "in queue" and we review any complains in modmail manually. But there are too many comment for us unpaid internet janitors to not use our own robots. Also since we are not paid our caring levels are lower, thus heavy use of regex and automation.

mod 4y ago

Per the article, there was a human involved in an appeal for this matter.

vintermann 4y ago
Per the article, per Twitter. It sounds unlikely. Though it could be that there's a guy paid to look over stuff the AI flags. If that's the same guy who does the appeals, they might reject the appeal in order to hide that they didn't actually look at it before banning (and hope the complainant just goes away quietly. They probably often do.)

TheRealPomax 4y ago

It's not "want", Twitter (or youtube, or facebook, etc. etc.) can't in a million years afford content moderation because of the sheer firehose of content. They'd literally have to employ everyone on the planet to get enough moderators in every language for every tweet, post, video, etc. on the platform. And then only one company would have enough moderators. Moderators are a finite resource, there aren't even enough humans alive to moderate all content on all platforms. But, the law goes "well figure something out, you have to moderate", and now we've by law guaranteed that whatever solution that gets implemented to follow the letter of the law is going to be shit, because it's by definition going to be automated content moderation, and given the volumes involved, it's modern neural-net based because that's the only technology that's even remotely shown it has an over 50% success rate. Well done, us.

ShamelessC 4y ago
My frustration isn't really rooted in legality and has more to do with my personal beliefs about tech and unregulated capitalism. I've worked in tech a bit - I know how this goes. Some company wants to get a massive valuation by hitting DAU's in the millions, but you can't do maintenance/support with that many users in a single org even if you wanted to. As an engineer, I can totally relate to the feeling of "fuck support - that sounds awful", and I struggle to blame _them_ because, well - like I said, even if they tried they couldn't possible cover that much work due to an inability to scale their organization. Psychologically? It's pretty easy to get pissed off at these companies when this happens. These companies are growth-obsessed and the ostensible philosophy is always "making the world a better place". STFU and do your jobs or don't build a product that can't scale and try to pretend it's scaling just fine.
lifeformed 4y ago
You don't need human eyes on every single tweet, just need enough people to review the flagged content. You could still have algorithms flag things and rank them by confidence levels. You could also hide the content first and queue it for human review instead of auto-banning the user. It's a lot of work, but I feel like there's a lot more that could be done.

deleted 4y ago

[deleted]

pyuser583 4y ago

One complaint is that moderation gets worse when dealing with less common languages/cultures. Not long ago there as an HN post about someone being banned for being Syrian (sanctions) when they were speaking in Syriac. For a non-native English speaker, Syrian (Arabic) and Syriac (an ancient language not spoken for over a thousand years) is really hard to know. There are going to similar mistakes in every language. Getting a native speaker for some languages is borderline impossible. Some languages don’t even have native speakers (Syriac, for example).