5 ms·

It looks like moderation is one of the biggest problem we're facing in tech, in different ways: - fake reviews which make the ratings useless - scam apps that

It looks like moderation is one of the biggest problem we're facing in tech, in different ways: - fake reviews which make the ratings useless - scam apps that riddle the various app stores - abusive / illegal content on platforms that can't afford better moderation - moderators who struggle with mental health issues, as per this tweet - footballers receiving racist comments on their socials Is there space for a startup to solve this problem with AI? And should it be a B2B or B2C scenario, as in working with platforms to moderate in the backend or being a user app / extension that does it in the frontend? For example, Fakespot works great with Amazon but it's an extension you have to install, so it's not available on Safari / on Mobile. So it's still quite unknown and I'm not sure how much it can grow

AbrahamParangi 5y ago
The essential problem is that excellent moderation can be counter to the platform incentives. Fake reviews can be bad for the customer, but potentially neutral or good for the platform. Same with fake engagement. Doing a really good job with moderation will materially and negatively impact KPIs in the short term.
Juliate 5y ago
Why would AI solve this problem in the first place? We barely understand how biased it is, and even if/why it's necessarily biased, not by design, but by nature. As in, biased, because the reference you use is subjective to the time and society and culture it is built in. Which is different from the whole context 30 years ago, and will be different in 30 years. Why, also, would tech solve a problem tech did not create, but amplified?
motohagiography 5y ago
Consider the economics a bit. The point of moderation is to sustain the business model of the platforms that need it most. It's necessarily going to be a cost centre that will only get optimized down to where it's a kind of impoverished hellscape to work in. The reason it's a cost centre is because it's trivial to stand up a new platform with a higher bar to entry that precludes almost all evil content, with the limiting factor that it won't have exposure to the total size of the legacy platforms that have no bar to entry. So for FB, better moderation is not a differentiator with an ROI, it's a risk management cost. This puts a ceiling on what they'll pay for it. The ideal solution for an FB-like platform that is already committed to open doors user policy means that the cost model/value prop is that the solution must be able to show it's doing something above all. If you sold them an AI solution that "just worked" for 98% of material and had a tiny relative cost, FB's real risk from the 2% that still got through doesn't change, because when FB gets asked by regulators and others what they are doing about it, your incredibly cost-effective AI solution just shows FB isn't committing resources to the problem. When FB can show rooms full of workers at screens on 24/7 shifts responding to the problem (at some tiny foreign hourly wage that their employer with the moderation contract is probably making a 300%+ margin on) that is the solution. These peoples' job is to essentially be psychological human sacrifices to regulators. This also describes most of the economics of the managed security services SOC business model. If you want to get into horror, find out if there is a correlation between the production of evil content and the regional presence of a moderation subcontractor. It's like anti-virus and security companies releasing viruses and exploits, but next level worse. If there is $100m/year market for moderation services, that literally creates the perverse incentive to ensure a flow of evil content. Could there be a CrowdStrike of content moderation? Maybe. Especially in markets like China, Singapore, and increasingly Britain, where policing content could be creating the same regulatory environment that hackers created for security.
- intended 5y ago
  > These peoples' job is to essentially be psychological human sacrifices to regulators. Darkly true. I think this shows that the incentives are not aligned. Outsourcing makes this a “solved” problem. Content moderation which ends up reducing time on site but saving people will not see the light of day. So the job is to flag and send content down the moderation pipeline. For now the frontier is dealing with the politics, not improving the lot of content moderators. Policy people will get hired because of how well they know/navigate the politics of their region, not because of their ability to design good moderation policy.

AbrahamParangi 5y ago

The essential problem is that excellent moderation can be counter to the platform incentives. Fake reviews can be bad for the customer, but potentially neutral or good for the platform. Same with fake engagement. Doing a really good job with moderation will materially and negatively impact KPIs in the short term.

Juliate 5y ago

Why would AI solve this problem in the first place? We barely understand how biased it is, and even if/why it's necessarily biased, not by design, but by nature. As in, biased, because the reference you use is subjective to the time and society and culture it is built in. Which is different from the whole context 30 years ago, and will be different in 30 years. Why, also, would tech solve a problem tech did not create, but amplified?

motohagiography 5y ago

Consider the economics a bit. The point of moderation is to sustain the business model of the platforms that need it most. It's necessarily going to be a cost centre that will only get optimized down to where it's a kind of impoverished hellscape to work in. The reason it's a cost centre is because it's trivial to stand up a new platform with a higher bar to entry that precludes almost all evil content, with the limiting factor that it won't have exposure to the total size of the legacy platforms that have no bar to entry. So for FB, better moderation is not a differentiator with an ROI, it's a risk management cost. This puts a ceiling on what they'll pay for it. The ideal solution for an FB-like platform that is already committed to open doors user policy means that the cost model/value prop is that the solution must be able to show it's doing something above all. If you sold them an AI solution that "just worked" for 98% of material and had a tiny relative cost, FB's real risk from the 2% that still got through doesn't change, because when FB gets asked by regulators and others what they are doing about it, your incredibly cost-effective AI solution just shows FB isn't committing resources to the problem. When FB can show rooms full of workers at screens on 24/7 shifts responding to the problem (at some tiny foreign hourly wage that their employer with the moderation contract is probably making a 300%+ margin on) that is the solution. These peoples' job is to essentially be psychological human sacrifices to regulators. This also describes most of the economics of the managed security services SOC business model. If you want to get into horror, find out if there is a correlation between the production of evil content and the regional presence of a moderation subcontractor. It's like anti-virus and security companies releasing viruses and exploits, but next level worse. If there is $100m/year market for moderation services, that literally creates the perverse incentive to ensure a flow of evil content. Could there be a CrowdStrike of content moderation? Maybe. Especially in markets like China, Singapore, and increasingly Britain, where policing content could be creating the same regulatory environment that hackers created for security.

intended 5y ago
> These peoples' job is to essentially be psychological human sacrifices to regulators. Darkly true. I think this shows that the incentives are not aligned. Outsourcing makes this a “solved” problem. Content moderation which ends up reducing time on site but saving people will not see the light of day. So the job is to flag and send content down the moderation pipeline. For now the frontier is dealing with the politics, not improving the lot of content moderators. Policy people will get hired because of how well they know/navigate the politics of their region, not because of their ability to design good moderation policy.