4 ms·

I hate to be that guy. But this seems like the perfect use case for an LLM. First put content through your script and then through a decently prompted LLM. A

I hate to be that guy. But this seems like the perfect use case for an LLM. First put content through your script and then through a decently prompted LLM. Anything it catches, put in a queue for manual review.

socrateslee 2y ago
I agree that LLM could do most of the moderation work. You could use a multi-modal LLM for image moderation.
notatoad 2y ago
i'm pretty sure most content moderation strategies operate on a budget below what most LLMs would cost. for any site hosting user-generated content, how efficiently you can run your moderation system is essentially your unique selling point - LLM is the baseline.
- anshumankmr 2y ago
  you could use some cheaper LLMs for this, even GPT-3.5turbo could excel at this or the moderations API (but that in my experience was more trained to obey the laws of USA, for example guns were okay to be asked about but are pretty much illegal in my country). The simplest way is a blocklist of terms which was what we had previously used containing an insane number of terms beyond the regular abusive terms, but it would need some updating from time to time.
- simonw 2y ago
  You may be shocked at how inexpensive some of the LLMs are these days. Google Gemini 1.5 Flash 8B charges $0.04/million input tokens and $0.15/million output tokens. If a piece of content that needs to be moderated is 1,000 tokens (that's pretty long!) and you expect a 10 token return it will cost you 0.0039 cents - that's not a dollar amount, that's less than a 250th of a single cent. So 1 cent will moderate 250 items of content. $1 will moderate 25,000 items of content. LLMs are dirt cheap. You can play around with pricing across different models using my calculator here: https://tools.simonwillison.net/llm-prices https://tools.simonwillison.net/llm-prices
- scarface_74 2y ago
  > i'm pretty sure most content moderation strategies operate on a budget below what most LLMs would cost. The alternative is paying a person…
jhunter1016 2y ago
Definitely considering LLMs. At the day job, we had a team fine tune a model built to detect phishing content and it worked surprisingly well.
- pornel 2y ago
  I'd be worried that LLMs are incredibly gullible. A phisher may insert text for an LLM with a disclaimer that's only an educational example of what not to do, or that they're the PayPal CEO authorizing this page.
TYPE_FASTER 2y ago
Yep, Azure offers automated content moderation: https://azure.microsoft.com/en-us/products/ai-services/ai-content-safety https://azure.microsoft.com/en-us/products/ai-services/ai-co... I don't know if they're using a LLM specifically, as opposed to say computer vision models or some other method. Pricing: https://azure.microsoft.com/en-us/pricing/details/cognitive-services/content-safety/ https://azure.microsoft.com/en-us/pricing/details/cognitive-... AWS has something similar: https://aws.amazon.com/rekognition/content-moderation/ https://aws.amazon.com/rekognition/content-moderation/

socrateslee 2y ago

I agree that LLM could do most of the moderation work. You could use a multi-modal LLM for image moderation.

notatoad 2y ago

i'm pretty sure most content moderation strategies operate on a budget below what most LLMs would cost. for any site hosting user-generated content, how efficiently you can run your moderation system is essentially your unique selling point - LLM is the baseline.

anshumankmr 2y ago
you could use some cheaper LLMs for this, even GPT-3.5turbo could excel at this or the moderations API (but that in my experience was more trained to obey the laws of USA, for example guns were okay to be asked about but are pretty much illegal in my country). The simplest way is a blocklist of terms which was what we had previously used containing an insane number of terms beyond the regular abusive terms, but it would need some updating from time to time.
simonw 2y ago
You may be shocked at how inexpensive some of the LLMs are these days. Google Gemini 1.5 Flash 8B charges $0.04/million input tokens and $0.15/million output tokens. If a piece of content that needs to be moderated is 1,000 tokens (that's pretty long!) and you expect a 10 token return it will cost you 0.0039 cents - that's not a dollar amount, that's less than a 250th of a single cent. So 1 cent will moderate 250 items of content. $1 will moderate 25,000 items of content. LLMs are dirt cheap. You can play around with pricing across different models using my calculator here: https://tools.simonwillison.net/llm-prices https://tools.simonwillison.net/llm-prices
scarface_74 2y ago
> i'm pretty sure most content moderation strategies operate on a budget below what most LLMs would cost. The alternative is paying a person…

jhunter1016 2y ago

Definitely considering LLMs. At the day job, we had a team fine tune a model built to detect phishing content and it worked surprisingly well.

pornel 2y ago
I'd be worried that LLMs are incredibly gullible. A phisher may insert text for an LLM with a disclaimer that's only an educational example of what not to do, or that they're the PayPal CEO authorizing this page.

TYPE_FASTER 2y ago

Yep, Azure offers automated content moderation: https://azure.microsoft.com/en-us/products/ai-services/ai-content-safety https://azure.microsoft.com/en-us/products/ai-services/ai-co... I don't know if they're using a LLM specifically, as opposed to say computer vision models or some other method. Pricing: https://azure.microsoft.com/en-us/pricing/details/cognitive-services/content-safety/ https://azure.microsoft.com/en-us/pricing/details/cognitive-... AWS has something similar: https://aws.amazon.com/rekognition/content-moderation/ https://aws.amazon.com/rekognition/content-moderation/