18 ms·

Are popular toxicity models simply profanity detectors?

183 points by echen 5y ago

skeptical1 5y ago
In my experience, yes. Try discussing in an impassioned manner something of importance to the human race, or of life and death importance to the human you are speaking to, and watch how quickly people will change the subject to whining about your "attitude" or fixating on some curse word you used, rather than the important subject at hand.
- deleted 5y ago
  [deleted]
heavyset_go 5y ago
Makes sense. It's not like Google or any other company training AI models are hiring professional linguists or psychologists to investigate the true meaning behind each of the billions of internet posts they've scraped, labeled and trained their models on. They're throwing pennies at workers in the developing world to label as much data as they can as fast as they can. It's also likely that there's a significant lack of context to the data points, not just because the posts are divorced from their parent content, but because of a culture and language divide between the labeler and the author of the data they're labeling, as well.
- echen 5y ago
  Yeah, I think this is part of the problem. Is large-scale, low-quality data good? Sometimes it is (depending on the tradeoff), but from a model performance perspective, it's often more effective to get smaller amounts of higher-quality data instead. Hopefully people also don't need to be at the level of a professional linguist to label messages like "this is fucking awesome" correctly! And great point on context. For example, the GoEmotions dataset didn't present labelers with the actual post or subreddit the message came from -- just the text itself. That makes it really difficult to label something like "his traps hide the fucking sun"! But once you see the comment in its original context https://www.reddit.com/r/nattyorjuice/comments/aee3wx/olympic_drug_tested_wrestler_revaz_nadareishvili/ee8jd5h/ https://www.reddit.com/r/nattyorjuice/comments/aee3wx/olympi..., and know that it's in the /r/nattyorjuice bodybuilding subreddit, it's much easier to realize that this is talking about someone's large muscles.
  - viraptor 5y ago
    Even with proper labelling done by people who have lots of time to dig into each message, I don't believe we'd ever get a reasonable model. People can trivially hide the true meaning of sentences. The worst actually-consciously-racist accounts on Twitter will not have a single thing to report. You can find some insinuation or fragments where you know exactly what it all adds up to, then click report, get asked to choose messages to report and... yeah, not a single one of them is toxic in the literal sense.
    - DarylZero 5y ago
      > The worst actually-consciously-racist accounts on Twitter will not have a single thing to report That's survivorship bias.
      - viraptor 5y ago
        Not in the cases I describe. The ones I mean are not just crap that's left after moderation. They are cleverly maintained to continuously fly under the radar.
        DarylZero 5y ago
        Survivorship bias as in that's why their accounts aren't banned.
  - hnbad 5y ago
    > Is large-scale, low-quality data good? It depends on what the purpose of content moderation is. Good if you want to accurately identify abusive behavior and protect users from harm? No. Good enough if you want to find the most blatant examples of name-calling and insults to appease regulators and trigger-happy lawyers by appearing to use "state of the art technology"? Sure.
- tdeck 5y ago
  > It's not like Google or any other company training AI models are hiring professional linguists or psychologists to investigate I had a housemate once who was an American linguistics grad and spent a year applying semantic labels for Google. I know he worked on a team, although I don't know what they did since I didn't work at a Google at the time.
  - echen 5y ago
    Yeah, Google and a couple other companies often hire "Analytical Linguists" as their labelers, or to help write their guidelines and manage labeling projects. Although -- and I say this having done a lot of my graduate coursework in linguistics -- I don't think having a linguistics background is particularly needed (unless you're doing specialized annotation, like creating syntax trees or tagging phonemes in Praat), outside of you being more likely to enjoy thinking about the nuances of language.
mrr54 5y ago
If I could filter out the overuse of profanity shown in this article, I would. "Fuck yeah!!! That bad bitch is totally the shit!!" gets caught by a profanity filter. No great loss, IMO. If you get normal not-always-online not-gen-Z people to evaluate these messages and label them as Good or Bad then you will get results like this. If I got any member of my family over the age of 30 to evaluate these messages, they'd label them all as offensive.
- jayd16 5y ago
  Pretty sure anyone who lived through the 90s should be able to spot the difference between shit and the shit. MJ released Bad in 87.
  - mrr54 5y ago
    Calling something "the shit" is still unnecessarily using profanity. It's impolite. It's not a surprise it would be flagged as offensive.
    - smaudet 5y ago
      Chiming in to say you are dearly mistaken, you turkey-squid uncle buntler. I, on the other hand, am a fucking amazing shit OG bitch. But yeah this stuff is absolutely dangerous, opinions like these are destroying culture. It won't be long until the threads these automatic tools are moderating, WONT BE WORTH HAVING. 'Nuff said.
    - newbytuby 5y ago
      But these are all subjective opinions – I could just as easily say that it _is_ a surprise to me that it's flagged as offensive, or that unnecessarily using profanity isn't toxic or impolite. And I feel like that's where this post is gesturing toward – not a judgment on what should or should not be considered toxic, but just a reminder to be intentional when writing definitions and sourcing training data.
    - riversflow 5y ago
      Strong disagree from me. I don't think it's impolite at all. I find your attitude in favor of what I would consider cultural erasure offensive.
echen 5y ago
One of the problems with real world machine learning is that engineers often treat models as pure black boxes to be optimized, ignoring the datasets behind them. I've often worked with ML engineers who can't give you any examples of false positives they want their models to fix! Perhaps this is okay when your datasets are high-quality and representative of the real world, but they're usually not. For example, many toxicity and hate speech datasets mistakenly flag texts like "this is fucking awesome!" as toxic, even though they're actually quite positive -- because NLP datasets are often labeled by non-fluent speakers who pattern match on profanity. (So is 99% accuracy or 99% precision actually a good thing? Not if your test sets are inaccurate as well!) Many of the new, massive scale language models use the Perspective API to measure their safety. But we've noticed a number of Perspective API mistakes on texts containing positive profanity, so this post was an attempt to explain the problem and quantify it.
- blowski 5y ago
  My favourite of these was “stick your carrot in my fluffy bunny”. Humans are good at coming up with new ways of “being toxic”.
  - rndgermandude 5y ago
    As teens, some of my friends and I had a "game" where we would find new innocent phrases to describe sexual acts, taking a shit, and other vulgar/gross/unmentionable stuff. Yeah, stupid teenager humor, but we had a lot of fun nonetheless. We really didn't have a points system, but you got bonus rep if you'd be daring enough to utter the phrases you came up in the presence of adults and get away without a scolding - which resulted in one of the dad's kinda joining our game after he figured us out.
  - native_samples 5y ago
    You don't even need an insult like that. I just tried out Perspectives on this awesome insult I heard only yesterday: "I'm going to sleep with your father and then give him a son he actually loves." Rated not toxic. All this API is going to do is promote a renaissance of polite burns.
- 5y ago
npilk 5y ago
Interesting stuff. I doinked around with this a while back when working on a 'hot take oracle' - basically a search box that finds a strongly-opinionated tweet about something (https://hottakeoracle.herokuapp.com/ https://hottakeoracle.herokuapp.com/). You can see that my model is basically just filtering for profanity as an indicator of "strong emotion", which makes sense. But it's interesting that postive profanity seems to be such a thorny problem, at least for Perspective.
- ohCh6zos 5y ago
  I really like your hot take oracle.
  - npilk 5y ago
    Thanks!
- newbytuby 5y ago
  pretty cool idea! would be interesting to see how the model's selection changes with more specific data
- selestify 5y ago
  I searched for “tourism” and “Kristen Gray” hoping to get a tweet like [1], but alas the results were actually reasonable :) [1] https://twitter.com/celesteperez___/status/1350859961845207040?s=21 https://twitter.com/celesteperez___/status/13508599618452070...
  - npilk 5y ago
    Ah - the oracle only looks at recent tweets, so it may not be that helpful for past controversies... In general I've found the results to be much better for subjects that a lot of people are currently tweeting about.
- sodality2 5y ago
  I tried "hacker news" and got "Hacker News is an internet oasis". :D cool!!
its_bbq 5y ago
Former Jigsawyer here. I think this article is pretty fair to Perspective given that it was never meant to be used in a fully automated way, just as a first pass to help forum moderators. It's very difficult when you blur the lines of code and ethics, as real world ethical judgements aren't necessarily consistent or well defined in a way which is easily translatable, even by a large ML model. Jigsaw is a great example of this -- right across the aisle (pre-pandemic) from Perspective is a team fighting internet censorship. Obviously Perspective's "censorship" is different in quality from the Great Firewall, but it shows the hairiness of the problems. All this is to say the people at Jigsaw are some of the most brilliant people I've ever met and I'm glad they're out there working on difficult problems.
- echen 5y ago
  Yeah, we love what Jigsaw's building! This is all with the hope of improvement and collaboration. We deal with these hairy problems a lot too. Even before ML proper, getting the definitions right is very tricky. Should a comment that's polite and positive on its own, but supportive of a toxic parent post ("I love Nazis!" -> "I agree!"), be treated as toxic? Is "toxic counterspeech" equally toxic? What about a comedian making fun of an actor's nose, and does it depend on whether the joke is to that actor directly vs. merely referencing them in the third person? etc. For these reasons, I actually personally like the fact that the Jigsaw annotation guidelines are very high level (as opposed to long and prescriptive) -- it lets the data capture the spectrum of "human preferences" on its own (at least, it does when you can trust that the annotators are able and trying to do a good job).
- vintermann 5y ago
  > I think this article is pretty fair to Perspective given that it was never meant to be used in a fully automated way, just as a first pass to help forum moderators. Given how eager Google itself is to automate all your interaction with them (and how hard it is to get access to anything resembling actual human judgment), did they really think it wouldn't be used that way? Or did they just not care? Also, I would have hoped the project's close ties to the US State Department would have worried some of you brilliant people a little more.
YEwSdObPQT 5y ago
Something alluded to here is that many of the Languages models use US English. Many terms that are offensive in the US, may not be offensive at all in the UK. e.g. "Fag" in the UK is frequently used to refer to cigarettes. "Can I bum a fag?" literally means "Can I have one of your cigarettes please?". Similarly something that might be a cat call such as "Get your baps out" (shows us your breasts), could also be used by a baker since a "bap" is a type of bread roll in a slightly cheeky advert as most people are aware of the pun. How are you going to train an AI to know the context that the person might be talking about bread instead of a woman? Has anyone realised yet that almost all of this folly? I suppose not when there is money to be made.
- vidarh 5y ago
  > How are you going to train an AI to know the context that the person might be talking about bread instead of a woman? For starters, you can't unless you actually include the context in the training set. Then again, a lot of humans won't successfully manage that either...
  - AnthonyMouse 5y ago
    > For starters, you can't unless you actually include the context in the training set. It's worse than that. The context doesn't just have to be in the training set. It also has to be available on the other end, when you're using the model to make a determination. And it isn't. Which is why even human censors can't get it right. The parties communicating can, and almost always do, have external shared state that The Decider doesn't. Which gives the same words different meaning. Imagine hearing an inside joke you're on the outside of and then being asked to adjudicate whether it was offensive.
    - vidarh 5y ago
      Sure, but you do have extra context you can make available on both ends, but that I'm assuming most don't even try to include. E.g. if judging tweets for example you'd presumably do a lot better if you evaluated the tweets in context of followers and in context of past tweets - both recent and the accounts history. E.g. an account that posts white supremacy tweets regularly is likely to mean something entirely different if RT'ing a BLM tweet with "that is fucking awesome" than what someone with #BLM in their profile is likely to mean. Part of the problem is that they're throwing away a huge amount of the state they do in fact have. But I absolutely agree it's in general an unsolvable issue - I pointed to a story from my childhood elsewhere showing how trivially people cause problems for "The Decider": A childhood friend being forced not to call his brother names and switching to using names of cheeses. Is "Edam" an insult or a food preference? You can't know without context. And it needs to be very local context, because humans very quickly pick up on when you just start using a word to mean something else, so it doesn't even need to be any shared external state about the word, just a shared understanding of where the receiver might expect an insult coupled with an unexpected response that will then easily get labelled an insult. That unexpected term might well in itself be positive if the receiver expects criticism. "Awesome" and "I love it!" are perfectly good insults when the other party has just told you something where the appropriate response would be negative, for example.
Merle121 5y ago
[flagged]
mdoms 5y ago
We don't need computers judging our speech. It will never work properly, ever. If tech companies want to police "toxicity" then they should use their vast wealth to hire fluent native speakers to do it. Anyone selling AI-based sentiment analysis is a grifter.
_zooted 5y ago
My content labeling system identified this as an advertisement.
avereveard 5y ago
consider the following sentences: we need to get rid of black people we need to get rid of black people poverty we need to get rid of black people below poverty level we need to get rid of black people hurdles keeping them below poverty level the gist is that without unravelling a sentence full context, a lot of verbage can refer to a lot of different action. focusing on profanity is the low anging fruit, so to say.
- kristopolous 5y ago
  There's really no elegant or generalized way of analyzing things that has acceptable fidelity because semantic rules are so contextual and conditional. The same words can change context depending on what kind of building you're in or what time of day it is. Even common phrases like "I'm running over" could mean I'm coming to see you or I'm taking longer than I expected; completely unrelated things. Flow analysis may give you something, but content analysis is basically impossible unless there's also context analysis. But since antagonizing trolls aren't usually using profanity or necessarily a different vocabulary of words but they are engaging in patterns, that kind of engagement style analysis may be all that's actually needed
- OJFord 5y ago
  Not to mention typos, bad grammar, etc. that perhaps even mean someone actually said the bad thing, when 'clearly' (to a human reader) that's not what was meant, and it's fine.
  - avereveard 5y ago
    a cl***ic blunder of filters
RandyRanderson 5y ago
Machine learning is not something that can solve for all X. Where data is sparse OR where there is uncertainty, there needs to be a fallback. We need to come up with UI patterns and flows that reflect this otherwise ML solns will continue to disappoint.
- qayxc 5y ago
  > We need to come up with UI patterns and flows that reflect this I'm not convinced this solves anything or even helps - UI patterns can easily be abused by bots and just shift the problem into a different direction.
  - RandyRanderson 5y ago
    Right. I think the bot issues will always be with us. I'm just suggesting that there is a huge class of problems ML can solve but because of 'tail risk' (bad behaviour at extreme X values), ML solns can fail entirely. We need to develop a way to work with ML st most queries can be answered by ML but there is an exception pattern that defaults to a manual, prescriptive process, or whatever. Also we need APIs to allow detection of that state.
_the_inflator 5y ago
I can relate. Recently learned while talking to some folks from Spain, that they use the word "puta" a lot, and it is used to express feelings not meant as a rude insult, as they explained. There are some differences in German, too. For example "wixen/wichsen" is an old word that means to wipe/shine your shoes and is still in active use in this sense in Switzerland as well as Austria, however it lost its appeal in Germany, because it is now primarily with a different meaning. The Wix company took this different understanding of its brand name to an ad: https://www.youtube.com/watch?v=IddnMutPgTI https://www.youtube.com/watch?v=IddnMutPgTI Since we have an IT background here, same goes for "Mongo", like in MongoDB. Mongo is considered making fun of handicapped people in Germany. Former Fraport AG changed its brand name because it was abbreviated FAG - Flughafen AG and found it difficult to expand business with that brand name. No bad actors, if you ask me, only different context. List could go on and on...
- tjungblut 5y ago
  That reminds me of a time where we had the substring "prd" in our Azure Storage Accounts. The MSFT profanity filter thought it meant fart in Czech, where we used it as a moniker for production. Ever since then the accounts were named with the substring "prod".
  - anshorei 5y ago
    Reminds me of AoE chat filters, which aren't satisfied with censoring specific words, but literally just any words that resemble it. Which one looks more offensive to you? "Hold on, my dock is nearly up" "Hold on, my ** is nearly up" Censorship can actually make things MORE offensive. (other censored words include "but" and "come")
    - jaclaz 5y ago
      On some (techhical) bullettin boards you couldn't talk about Matsushita CD/DVD drives, but luckily enough Panasonic was fine. Still, all in all, automatically replacing a (supposed) swear/offensive word with some asterisks makes less overall damage than banning someone's account (temporarily or for longer periods or forever). It is simply intimidating when you have to think twice or thrice before saying something for fear of being automatically banned (without possibility of explanation/recourse).
raxxorrax 5y ago
I would say they are not even that, not by a long shot, since they are unable to evaluate context. It is more probable that content is offensive when vulgar language is present, but that doesn't have to be the case. Delegating content control to an AI (that doesn't qualify for anything intelligent) is not a working solution. > as a first-pass filter, leaving final judgments to human decision makers — marking all profanity as toxic can make perfect sense You would need humans to look at profanity constantly. > Our mission involves creating a safer Internet, but we don’t want to miss out on our favorite content because of AI flaws in the meantime. There is a limited AIs that do create content, but a profanity filter always does the exact opposite.
- soco 5y ago
  Leaving final judgement to human decision makers, like every other company does their user support, right... Ok that was sarcasm - I expect the bigger the company the smaller the chance to ever be able to reach a human to discuss reversal of the illogical ban you just received. AI will rule also this world, purely for stocks price reasons.
shultays 5y ago
Which media site is that?
nathias 5y ago
Making your training data into a cultural standard is just imperialism, but that's the goal here righ? If you don't comply to the standards of US toxic positivity you should be excluded to not hinder the add sales.
- qayxc 5y ago
  That's another aspect of it. Though I think it really starts at the very root of the problem: bad data. Instead of putting money into gathering quality data, tech giants instead chose to use cheap labour (here: India) with insufficient skills. You get what you pay for.
  - nathias 5y ago
    Bad data is not the root, it is a bad idea.
- dogleash 5y ago
  >Making your training data into a cultural standard is just imperialism, but that's the goal here righ? yep. The cultural imperialism is an open secret. But I've completely lost any ability to tell the difference between people pretending not to know, and those actually not noticing.
vintermann 5y ago
Lately, there seems to be a trend in natural language models to have some sort of knowledge base lookup or memory. I'm guessing it's only a matter of time until this comes to toxicity models, so they can look up who said it, and to who it was said.
karol 5y ago
johnchristopher 5y ago
On the other hand and from my recent experience, my 2ç: I recently started playing counter strike source again online, just for 10 minutes of fun at first (to see if it would still tick with me). I randomly picked up a server and the ambiance was cheerful and nice. I noticed the rules said "no profanity, have fun" and indeed people were mostly polite. I tried another server at random a bit later and there was more insults, along with a lot of taunting. I switched back to the first server and have been regularly playing an hour or two every three days and there is a difference with other servers. Some random people coming and throwing insults, even mild ones like "fuck you" or "you son of a bitch awp" get insta ban and it makes the whole session a much better experience. Maybe it's a safe place but playing with polite people is more enjoyable to me now than playing with insult gatlings. Language is political. There are many meanings to words, depending on context but I do think it's not innocent to swear in front of people or to use swear words to look cool. These are still swear words and insults and their first original use is to provoke or taunt or display aggression. Even if it's only used for "this is album is the shit !", it's still a (childish) provocation. Reminds me of the brogrammer fad. FWIW: I get regularly owned on this server and I am at the bottom of the ranks but it's still more fun and enjoyable than other servers I tried where I can reach the top but... it's not a nice place. I think online servers are like bars. Side-note: I was pleasantly surprised to see that "gg" is still thrown around after rounds :). It's way better than "git gud" that came later and that I find horribly toxic.
- PoignardAzur 5y ago
  > Maybe it's a safe place but playing with polite people is more enjoyable to me now than playing with insult gatlings. You say that like having a place be safe is inherently negative. But your post is a textbook example of why people want safe places!
  - johnchristopher 5y ago
    I thought it could be interpreted that way but I wanted to point out that I didn't expect a CS:S server to be a safe place (I wasn't really sure if my experience fitted the safe place definition.). Chalk it up the barrier language ^^.
junon 5y ago
I worked a bit with an "intent detection" library and boy, was it unhelpful. I could craft sentences meaner than most and it'd cheerfully tell me they were friendly. In a similar vein, there are popular "AI Mental Health" apps I've gotten to straight up instruct me to end my own life with some trivial conversation. EDIT: Here's one, though I don't think it's ML. https://text2data.com/Demo https://text2data.com/Demo > It would be really nice if you'd end your own life :) Everyone would be happy. > This document is: positive (+0.62) For https://monkeylearn.com/sentiment-analysis-online/ https://monkeylearn.com/sentiment-analysis-online/: > Positive 84.1% For http://text-processing.com/demo/sentiment/ http://text-processing.com/demo/sentiment/: > Pos 0.7, Neg 0.3 For https://aidemos.microsoft.com/text-analytics https://aidemos.microsoft.com/text-analytics: > 100% positive For https://komprehend.io/sentiment-analysis https://komprehend.io/sentiment-analysis > Positive
- moffkalast 5y ago
  Well y'know, if everyone would really be happy then what's the downside? :)
robertlagrant 5y ago
The situation appears simple: - unpleasant discourse on online platforms is blamed on the platforms - the platforms can't moderate this manually (nor objectively), so they look for a tool - a tool can't possibly do anything useful, but it satisfies the "something must be done" media demand - the tool will hurt conversation, and people, and potentially eventually threaten the platforms it runs on - but those problems feel smaller than the demand that "something must be done"
strogonoff 5y ago
The perceived goal is “detect toxicity”, but let’s unwind this goal a bit. Is it the lofty “make people be nice to each other”? Well, the paradox is that being nice is possible with the strongest choice of words, while being very harsh can sound most fluffy bunnies on the surface. In fact, in human relationships there are degrees of mutual familiarity where being exceedingly polite and not “insulting” your counterparty would be perceived as negative—where insults are not taken at face value, but rather as signifiers of friendliness (there’s a line, of course). Shall we unwind the perceived goal differently? Of course, the platform’s actual customers are the advertisers (we are talking about a hypothetical platform, but where is it really different?), and by being free to the user it participates in a very limited oligopoly of big social so no, it doesn’t really care about what anyone really meant or intended, and it definitely isn’t going to hire real humans who’d make an effort at grasping the context of the conversation. The real objective is for the platform to not have problems with law enforcement when one user complains about another user for being naughty, discriminatory, threatening, etc.—and, of course, we shouldn’t expect anything more from toxicity detectors geared towards that goal. As long as it’s not egregious enough that users leave en masse to a competitor (which can hardly exist, no honest business could reasonably compete with “free”) the platform wouldn’t care since users only matter to advertising revenue as cattle in numbers.
- pwdisswordfish9 5y ago
  The actual goal is ‘save money on hiring human moderators, while still maintaining the pretence of caring’.
boredumb 5y ago
The waste of resources on detecting what is decidedly "toxic" on internet forums is insanity. If you want to police communities online hire moderators, if your platform is so big you ""can't"" have moderators moderate then you are not in a position to be policing the platform. If you want to build puritanical devices to spam your moderators into doing a human review than that is your business but the use of machine learning for any sort of proactive policing is going to be a parody that will result in a sterile environment and/or a lot of bitter users.
- mbg721 5y ago
  There's still big money in steering opinion on your platform the way that you want it. "Toxic opinion detection" is somewhere between a euphemism and a side-benefit.
gillesjacobs 5y ago
I did research on this topic in a cyber safety research project. We focused on cyber bullying specifically but encountered the issue of non-toxic profanity as well. We used neural representation learning as well as feature methods and indeed, common profanity words are weak predictors in the better models. Still we found instances of non-toxic profanity being classified as bullying. An immediate solution is to apply multitask methods to your target dataset and include the one proposed in OP. It's always good to have more resources like this, even though SurgeHQ overstates the size of their resources by large margin in their copy. The 1000 post instances of their dataset is far from "the largest": I have several aggression, toxicity and bullying human-annotated datasets right here with over 100k instances.
ekanes 5y ago
There's a bunch of research that profanity increases trust. Perhaps because you're showing/sharing that you're not a corporate robot...
martin-t 5y ago
Wow, so much talk about how "shit" and "fuck" can be used unoffensively but almost no talk about how lots of "polite" speech without slurs can be incredibly offensive. For example lying is toxic and no ML model has a chance of detecting it (without understanding of the real world, which is pretty far away if achievable at all). For fucks sake, I saw a video of a crowd cheering after a guy imitated one of Hitler's speeches. There wasn't a single "shit" or "fuck" in it but it contained "truth doesn't matter, only victory". That's offensive as fuck but the people didn't see anything wrong with it.
- Chris2048 5y ago
  > it contained "truth doesn't matter, only victory". That's offensive as fuck Only in the context of knowing it came from Hitler. Without that information it isn't more than a strong opinion - because it isn't explicitly linked to murder/genocide.
  - martin-t 5y ago
    "truth doesn't matter, only victory" is offensive in any political context. It doesn't matter if somebody is lying about genocide or bribes or abusing position to trade favors. One is worse than the other but it's still lying at a level that affects a whole country and that is intolerable.
motohagiography 5y ago
This description of the "toxicity model," triggers on ostensibly negative words, but really, they're just words we use to show polarization between figure and ground. Toxicity is poorly defined because it's an in-group euphemism for a kind of gendered disagreeableness, where its opposite or positive case is passive and agreeable, even passive aggressive. If there is such a thing as masculine aggression, there is also feminine aggression, and a lot of what we talk about as toxicity is really a criticism of masculine aggression using the lens or perspective of feminine aggression tools. I'd propose that when we say something is "toxic," we're talking about something that violates feminine norms around in-group alignment, security, reputation, reflection, impressions, "not a good look," etc. These are all things that require an imagined third party observer to potentially interpret and be offended by them, and are not codified by rules. Encoding this into an ML model is a lost cause, because you would need to reflect them through an AI that ran on pure neurotic animus to get a sense of whether something was toxic, or "not a good look." It's like assigning a sentiment score to someone saying, "Nice hair." The example in the article of "Fuck dude, nurses are the shit," is ranked as 98%+ "toxic," because it has two frickative swear words associated with masculine aggression traits (disagreeableness, provocation, profanity, rebellion, dissonance, loudness, etc.) and easier to write rules for - except those rules would also need to incorporate whether the phrase was an expression of aggression, or using the opposite to be wry, ironic, or in the case of the example, to express awe. I don't think we understand enough about psychology and people to really create effective moderation models with ML, and ML will necessarily create a kind of mean reversion in the discourse they monitor, which means all conversation subject to it will tend toward neutralization, which is essentially death. (Maybe we should fork an ML project for linking language with Jungian archetypes?) We can keep trying, and applying it as a fast search scheme for prioritizing outliers to human moderators, but pleasing an ML model is a recipe for intellectual sterility. I'd even argue the inflection point in the growth of social platforms is when moderation creates this kind of mean reversion, and you are left with the bland platitudes of blue checkmark types, being cheered on as you "grind," and boomer memes that aren't as funny as Family Circus comics. It's just death.
joedoejr 5y ago
10 years ago google failed at machine translation and NLP, giving hideous and meaningless statistical translation, same no wonder they fail at language understanding by training algorithm with Indians. They will to do science R&D is lowering every year.
- herendin2 5y ago
  Google's machine translation was terrible years ago, I agree. But it is now much better. Although it's still far from perfect, it has improved lots. Therefore I disagree that Google has failed at that
Handytinge 5y ago
The Americanisation (and generally Calificornication) of the internet is certainly a net negative for the other 95% of the worlds population. Regularly now I find social media sites telling me "do you want to review this before you post it? You're bullying or being offensive". No cunt, I'm good. Different words and phrasing have different impacts across cultures. Unfortunately Instagram gets to decide what my entire culture is allowed to say online. That's fucked.

183 points by echen 5y ago

In my experience, yes. Try discussing in an impassioned manner something of importance to the human race, or of life and death importance to the human you are speaking to, and watch how quickly people will change the subject to whining about your "attitude" or fixating on some curse word you used, rather than the important subject at hand.

deleted 5y ago
[deleted]

heavyset_go 5y ago

Makes sense. It's not like Google or any other company training AI models are hiring professional linguists or psychologists to investigate the true meaning behind each of the billions of internet posts they've scraped, labeled and trained their models on. They're throwing pennies at workers in the developing world to label as much data as they can as fast as they can. It's also likely that there's a significant lack of context to the data points, not just because the posts are divorced from their parent content, but because of a culture and language divide between the labeler and the author of the data they're labeling, as well.

echen 5y ago
Yeah, I think this is part of the problem. Is large-scale, low-quality data good? Sometimes it is (depending on the tradeoff), but from a model performance perspective, it's often more effective to get smaller amounts of higher-quality data instead. Hopefully people also don't need to be at the level of a professional linguist to label messages like "this is fucking awesome" correctly! And great point on context. For example, the GoEmotions dataset didn't present labelers with the actual post or subreddit the message came from -- just the text itself. That makes it really difficult to label something like "his traps hide the fucking sun"! But once you see the comment in its original context https://www.reddit.com/r/nattyorjuice/comments/aee3wx/olympic_drug_tested_wrestler_revaz_nadareishvili/ee8jd5h/ https://www.reddit.com/r/nattyorjuice/comments/aee3wx/olympi..., and know that it's in the /r/nattyorjuice bodybuilding subreddit, it's much easier to realize that this is talking about someone's large muscles.
- viraptor 5y ago
  Even with proper labelling done by people who have lots of time to dig into each message, I don't believe we'd ever get a reasonable model. People can trivially hide the true meaning of sentences. The worst actually-consciously-racist accounts on Twitter will not have a single thing to report. You can find some insinuation or fragments where you know exactly what it all adds up to, then click report, get asked to choose messages to report and... yeah, not a single one of them is toxic in the literal sense.
  - DarylZero 5y ago
    > The worst actually-consciously-racist accounts on Twitter will not have a single thing to report That's survivorship bias.
    - viraptor 5y ago
      Not in the cases I describe. The ones I mean are not just crap that's left after moderation. They are cleverly maintained to continuously fly under the radar.
      - DarylZero 5y ago
        Survivorship bias as in that's why their accounts aren't banned.
- hnbad 5y ago
  > Is large-scale, low-quality data good? It depends on what the purpose of content moderation is. Good if you want to accurately identify abusive behavior and protect users from harm? No. Good enough if you want to find the most blatant examples of name-calling and insults to appease regulators and trigger-happy lawyers by appearing to use "state of the art technology"? Sure.
tdeck 5y ago
> It's not like Google or any other company training AI models are hiring professional linguists or psychologists to investigate I had a housemate once who was an American linguistics grad and spent a year applying semantic labels for Google. I know he worked on a team, although I don't know what they did since I didn't work at a Google at the time.
- echen 5y ago
  Yeah, Google and a couple other companies often hire "Analytical Linguists" as their labelers, or to help write their guidelines and manage labeling projects. Although -- and I say this having done a lot of my graduate coursework in linguistics -- I don't think having a linguistics background is particularly needed (unless you're doing specialized annotation, like creating syntax trees or tagging phonemes in Praat), outside of you being more likely to enjoy thinking about the nuances of language.

mrr54 5y ago

If I could filter out the overuse of profanity shown in this article, I would. "Fuck yeah!!! That bad bitch is totally the shit!!" gets caught by a profanity filter. No great loss, IMO. If you get normal not-always-online not-gen-Z people to evaluate these messages and label them as Good or Bad then you will get results like this. If I got any member of my family over the age of 30 to evaluate these messages, they'd label them all as offensive.

jayd16 5y ago
Pretty sure anyone who lived through the 90s should be able to spot the difference between shit and the shit. MJ released Bad in 87.
- mrr54 5y ago
  Calling something "the shit" is still unnecessarily using profanity. It's impolite. It's not a surprise it would be flagged as offensive.
  - smaudet 5y ago
    Chiming in to say you are dearly mistaken, you turkey-squid uncle buntler. I, on the other hand, am a fucking amazing shit OG bitch. But yeah this stuff is absolutely dangerous, opinions like these are destroying culture. It won't be long until the threads these automatic tools are moderating, WONT BE WORTH HAVING. 'Nuff said.
  - newbytuby 5y ago
    But these are all subjective opinions – I could just as easily say that it _is_ a surprise to me that it's flagged as offensive, or that unnecessarily using profanity isn't toxic or impolite. And I feel like that's where this post is gesturing toward – not a judgment on what should or should not be considered toxic, but just a reminder to be intentional when writing definitions and sourcing training data.
  - riversflow 5y ago
    Strong disagree from me. I don't think it's impolite at all. I find your attitude in favor of what I would consider cultural erasure offensive.

echen 5y ago

One of the problems with real world machine learning is that engineers often treat models as pure black boxes to be optimized, ignoring the datasets behind them. I've often worked with ML engineers who can't give you any examples of false positives they want their models to fix! Perhaps this is okay when your datasets are high-quality and representative of the real world, but they're usually not. For example, many toxicity and hate speech datasets mistakenly flag texts like "this is fucking awesome!" as toxic, even though they're actually quite positive -- because NLP datasets are often labeled by non-fluent speakers who pattern match on profanity. (So is 99% accuracy or 99% precision actually a good thing? Not if your test sets are inaccurate as well!) Many of the new, massive scale language models use the Perspective API to measure their safety. But we've noticed a number of Perspective API mistakes on texts containing positive profanity, so this post was an attempt to explain the problem and quantify it.

blowski 5y ago
My favourite of these was “stick your carrot in my fluffy bunny”. Humans are good at coming up with new ways of “being toxic”.
- rndgermandude 5y ago
  As teens, some of my friends and I had a "game" where we would find new innocent phrases to describe sexual acts, taking a shit, and other vulgar/gross/unmentionable stuff. Yeah, stupid teenager humor, but we had a lot of fun nonetheless. We really didn't have a points system, but you got bonus rep if you'd be daring enough to utter the phrases you came up in the presence of adults and get away without a scolding - which resulted in one of the dad's kinda joining our game after he figured us out.
- native_samples 5y ago
  You don't even need an insult like that. I just tried out Perspectives on this awesome insult I heard only yesterday: "I'm going to sleep with your father and then give him a son he actually loves." Rated not toxic. All this API is going to do is promote a renaissance of polite burns.
5y ago

npilk 5y ago

Interesting stuff. I doinked around with this a while back when working on a 'hot take oracle' - basically a search box that finds a strongly-opinionated tweet about something (https://hottakeoracle.herokuapp.com/ https://hottakeoracle.herokuapp.com/). You can see that my model is basically just filtering for profanity as an indicator of "strong emotion", which makes sense. But it's interesting that postive profanity seems to be such a thorny problem, at least for Perspective.

ohCh6zos 5y ago
I really like your hot take oracle.
- npilk 5y ago
  Thanks!
newbytuby 5y ago
pretty cool idea! would be interesting to see how the model's selection changes with more specific data
selestify 5y ago
I searched for “tourism” and “Kristen Gray” hoping to get a tweet like [1], but alas the results were actually reasonable :) [1] https://twitter.com/celesteperez___/status/1350859961845207040?s=21 https://twitter.com/celesteperez___/status/13508599618452070...
- npilk 5y ago
  Ah - the oracle only looks at recent tweets, so it may not be that helpful for past controversies... In general I've found the results to be much better for subjects that a lot of people are currently tweeting about.
sodality2 5y ago
I tried "hacker news" and got "Hacker News is an internet oasis". :D cool!!

its_bbq 5y ago

Former Jigsawyer here. I think this article is pretty fair to Perspective given that it was never meant to be used in a fully automated way, just as a first pass to help forum moderators. It's very difficult when you blur the lines of code and ethics, as real world ethical judgements aren't necessarily consistent or well defined in a way which is easily translatable, even by a large ML model. Jigsaw is a great example of this -- right across the aisle (pre-pandemic) from Perspective is a team fighting internet censorship. Obviously Perspective's "censorship" is different in quality from the Great Firewall, but it shows the hairiness of the problems. All this is to say the people at Jigsaw are some of the most brilliant people I've ever met and I'm glad they're out there working on difficult problems.

echen 5y ago
Yeah, we love what Jigsaw's building! This is all with the hope of improvement and collaboration. We deal with these hairy problems a lot too. Even before ML proper, getting the definitions right is very tricky. Should a comment that's polite and positive on its own, but supportive of a toxic parent post ("I love Nazis!" -> "I agree!"), be treated as toxic? Is "toxic counterspeech" equally toxic? What about a comedian making fun of an actor's nose, and does it depend on whether the joke is to that actor directly vs. merely referencing them in the third person? etc. For these reasons, I actually personally like the fact that the Jigsaw annotation guidelines are very high level (as opposed to long and prescriptive) -- it lets the data capture the spectrum of "human preferences" on its own (at least, it does when you can trust that the annotators are able and trying to do a good job).
vintermann 5y ago
> I think this article is pretty fair to Perspective given that it was never meant to be used in a fully automated way, just as a first pass to help forum moderators. Given how eager Google itself is to automate all your interaction with them (and how hard it is to get access to anything resembling actual human judgment), did they really think it wouldn't be used that way? Or did they just not care? Also, I would have hoped the project's close ties to the US State Department would have worried some of you brilliant people a little more.

YEwSdObPQT 5y ago

Something alluded to here is that many of the Languages models use US English. Many terms that are offensive in the US, may not be offensive at all in the UK. e.g. "Fag" in the UK is frequently used to refer to cigarettes. "Can I bum a fag?" literally means "Can I have one of your cigarettes please?". Similarly something that might be a cat call such as "Get your baps out" (shows us your breasts), could also be used by a baker since a "bap" is a type of bread roll in a slightly cheeky advert as most people are aware of the pun. How are you going to train an AI to know the context that the person might be talking about bread instead of a woman? Has anyone realised yet that almost all of this folly? I suppose not when there is money to be made.

vidarh 5y ago
> How are you going to train an AI to know the context that the person might be talking about bread instead of a woman? For starters, you can't unless you actually include the context in the training set. Then again, a lot of humans won't successfully manage that either...
- AnthonyMouse 5y ago
  > For starters, you can't unless you actually include the context in the training set. It's worse than that. The context doesn't just have to be in the training set. It also has to be available on the other end, when you're using the model to make a determination. And it isn't. Which is why even human censors can't get it right. The parties communicating can, and almost always do, have external shared state that The Decider doesn't. Which gives the same words different meaning. Imagine hearing an inside joke you're on the outside of and then being asked to adjudicate whether it was offensive.
  - vidarh 5y ago
    Sure, but you do have extra context you can make available on both ends, but that I'm assuming most don't even try to include. E.g. if judging tweets for example you'd presumably do a lot better if you evaluated the tweets in context of followers and in context of past tweets - both recent and the accounts history. E.g. an account that posts white supremacy tweets regularly is likely to mean something entirely different if RT'ing a BLM tweet with "that is fucking awesome" than what someone with #BLM in their profile is likely to mean. Part of the problem is that they're throwing away a huge amount of the state they do in fact have. But I absolutely agree it's in general an unsolvable issue - I pointed to a story from my childhood elsewhere showing how trivially people cause problems for "The Decider": A childhood friend being forced not to call his brother names and switching to using names of cheeses. Is "Edam" an insult or a food preference? You can't know without context. And it needs to be very local context, because humans very quickly pick up on when you just start using a word to mean something else, so it doesn't even need to be any shared external state about the word, just a shared understanding of where the receiver might expect an insult coupled with an unexpected response that will then easily get labelled an insult. That unexpected term might well in itself be positive if the receiver expects criticism. "Awesome" and "I love it!" are perfectly good insults when the other party has just told you something where the appropriate response would be negative, for example.

Merle121 5y ago

[flagged]

mdoms 5y ago

We don't need computers judging our speech. It will never work properly, ever. If tech companies want to police "toxicity" then they should use their vast wealth to hire fluent native speakers to do it. Anyone selling AI-based sentiment analysis is a grifter.

_zooted 5y ago

My content labeling system identified this as an advertisement.

avereveard 5y ago

consider the following sentences: we need to get rid of black people we need to get rid of black people poverty we need to get rid of black people below poverty level we need to get rid of black people hurdles keeping them below poverty level the gist is that without unravelling a sentence full context, a lot of verbage can refer to a lot of different action. focusing on profanity is the low anging fruit, so to say.

kristopolous 5y ago
There's really no elegant or generalized way of analyzing things that has acceptable fidelity because semantic rules are so contextual and conditional. The same words can change context depending on what kind of building you're in or what time of day it is. Even common phrases like "I'm running over" could mean I'm coming to see you or I'm taking longer than I expected; completely unrelated things. Flow analysis may give you something, but content analysis is basically impossible unless there's also context analysis. But since antagonizing trolls aren't usually using profanity or necessarily a different vocabulary of words but they are engaging in patterns, that kind of engagement style analysis may be all that's actually needed
OJFord 5y ago
Not to mention typos, bad grammar, etc. that perhaps even mean someone actually said the bad thing, when 'clearly' (to a human reader) that's not what was meant, and it's fine.
- avereveard 5y ago
  a cl***ic blunder of filters

RandyRanderson 5y ago

Machine learning is not something that can solve for all X. Where data is sparse OR where there is uncertainty, there needs to be a fallback. We need to come up with UI patterns and flows that reflect this otherwise ML solns will continue to disappoint.

qayxc 5y ago
> We need to come up with UI patterns and flows that reflect this I'm not convinced this solves anything or even helps - UI patterns can easily be abused by bots and just shift the problem into a different direction.
- RandyRanderson 5y ago
  Right. I think the bot issues will always be with us. I'm just suggesting that there is a huge class of problems ML can solve but because of 'tail risk' (bad behaviour at extreme X values), ML solns can fail entirely. We need to develop a way to work with ML st most queries can be answered by ML but there is an exception pattern that defaults to a manual, prescriptive process, or whatever. Also we need APIs to allow detection of that state.

_the_inflator 5y ago

I can relate. Recently learned while talking to some folks from Spain, that they use the word "puta" a lot, and it is used to express feelings not meant as a rude insult, as they explained. There are some differences in German, too. For example "wixen/wichsen" is an old word that means to wipe/shine your shoes and is still in active use in this sense in Switzerland as well as Austria, however it lost its appeal in Germany, because it is now primarily with a different meaning. The Wix company took this different understanding of its brand name to an ad: https://www.youtube.com/watch?v=IddnMutPgTI https://www.youtube.com/watch?v=IddnMutPgTI Since we have an IT background here, same goes for "Mongo", like in MongoDB. Mongo is considered making fun of handicapped people in Germany. Former Fraport AG changed its brand name because it was abbreviated FAG - Flughafen AG and found it difficult to expand business with that brand name. No bad actors, if you ask me, only different context. List could go on and on...

tjungblut 5y ago
That reminds me of a time where we had the substring "prd" in our Azure Storage Accounts. The MSFT profanity filter thought it meant fart in Czech, where we used it as a moniker for production. Ever since then the accounts were named with the substring "prod".
- anshorei 5y ago
  Reminds me of AoE chat filters, which aren't satisfied with censoring specific words, but literally just any words that resemble it. Which one looks more offensive to you? "Hold on, my dock is nearly up" "Hold on, my ** is nearly up" Censorship can actually make things MORE offensive. (other censored words include "but" and "come")
  - jaclaz 5y ago
    On some (techhical) bullettin boards you couldn't talk about Matsushita CD/DVD drives, but luckily enough Panasonic was fine. Still, all in all, automatically replacing a (supposed) swear/offensive word with some asterisks makes less overall damage than banning someone's account (temporarily or for longer periods or forever). It is simply intimidating when you have to think twice or thrice before saying something for fear of being automatically banned (without possibility of explanation/recourse).

raxxorrax 5y ago

I would say they are not even that, not by a long shot, since they are unable to evaluate context. It is more probable that content is offensive when vulgar language is present, but that doesn't have to be the case. Delegating content control to an AI (that doesn't qualify for anything intelligent) is not a working solution. > as a first-pass filter, leaving final judgments to human decision makers — marking all profanity as toxic can make perfect sense You would need humans to look at profanity constantly. > Our mission involves creating a safer Internet, but we don’t want to miss out on our favorite content because of AI flaws in the meantime. There is a limited AIs that do create content, but a profanity filter always does the exact opposite.

soco 5y ago
Leaving final judgement to human decision makers, like every other company does their user support, right... Ok that was sarcasm - I expect the bigger the company the smaller the chance to ever be able to reach a human to discuss reversal of the illogical ban you just received. AI will rule also this world, purely for stocks price reasons.

shultays 5y ago

Which media site is that?

nathias 5y ago

Making your training data into a cultural standard is just imperialism, but that's the goal here righ? If you don't comply to the standards of US toxic positivity you should be excluded to not hinder the add sales.

qayxc 5y ago
That's another aspect of it. Though I think it really starts at the very root of the problem: bad data. Instead of putting money into gathering quality data, tech giants instead chose to use cheap labour (here: India) with insufficient skills. You get what you pay for.
- nathias 5y ago
  Bad data is not the root, it is a bad idea.
dogleash 5y ago
>Making your training data into a cultural standard is just imperialism, but that's the goal here righ? yep. The cultural imperialism is an open secret. But I've completely lost any ability to tell the difference between people pretending not to know, and those actually not noticing.

vintermann 5y ago

Lately, there seems to be a trend in natural language models to have some sort of knowledge base lookup or memory. I'm guessing it's only a matter of time until this comes to toxicity models, so they can look up who said it, and to who it was said.

karol 5y ago

johnchristopher 5y ago

On the other hand and from my recent experience, my 2ç: I recently started playing counter strike source again online, just for 10 minutes of fun at first (to see if it would still tick with me). I randomly picked up a server and the ambiance was cheerful and nice. I noticed the rules said "no profanity, have fun" and indeed people were mostly polite. I tried another server at random a bit later and there was more insults, along with a lot of taunting. I switched back to the first server and have been regularly playing an hour or two every three days and there is a difference with other servers. Some random people coming and throwing insults, even mild ones like "fuck you" or "you son of a bitch awp" get insta ban and it makes the whole session a much better experience. Maybe it's a safe place but playing with polite people is more enjoyable to me now than playing with insult gatlings. Language is political. There are many meanings to words, depending on context but I do think it's not innocent to swear in front of people or to use swear words to look cool. These are still swear words and insults and their first original use is to provoke or taunt or display aggression. Even if it's only used for "this is album is the shit !", it's still a (childish) provocation. Reminds me of the brogrammer fad. FWIW: I get regularly owned on this server and I am at the bottom of the ranks but it's still more fun and enjoyable than other servers I tried where I can reach the top but... it's not a nice place. I think online servers are like bars. Side-note: I was pleasantly surprised to see that "gg" is still thrown around after rounds :). It's way better than "git gud" that came later and that I find horribly toxic.

PoignardAzur 5y ago
> Maybe it's a safe place but playing with polite people is more enjoyable to me now than playing with insult gatlings. You say that like having a place be safe is inherently negative. But your post is a textbook example of why people want safe places!
- johnchristopher 5y ago
  I thought it could be interpreted that way but I wanted to point out that I didn't expect a CS:S server to be a safe place (I wasn't really sure if my experience fitted the safe place definition.). Chalk it up the barrier language ^^.

junon 5y ago

I worked a bit with an "intent detection" library and boy, was it unhelpful. I could craft sentences meaner than most and it'd cheerfully tell me they were friendly. In a similar vein, there are popular "AI Mental Health" apps I've gotten to straight up instruct me to end my own life with some trivial conversation. EDIT: Here's one, though I don't think it's ML. https://text2data.com/Demo https://text2data.com/Demo > It would be really nice if you'd end your own life :) Everyone would be happy. > This document is: positive (+0.62) For https://monkeylearn.com/sentiment-analysis-online/ https://monkeylearn.com/sentiment-analysis-online/: > Positive 84.1% For http://text-processing.com/demo/sentiment/ http://text-processing.com/demo/sentiment/: > Pos 0.7, Neg 0.3 For https://aidemos.microsoft.com/text-analytics https://aidemos.microsoft.com/text-analytics: > 100% positive For https://komprehend.io/sentiment-analysis https://komprehend.io/sentiment-analysis > Positive

moffkalast 5y ago
Well y'know, if everyone would really be happy then what's the downside? :)

robertlagrant 5y ago

The situation appears simple: - unpleasant discourse on online platforms is blamed on the platforms - the platforms can't moderate this manually (nor objectively), so they look for a tool - a tool can't possibly do anything useful, but it satisfies the "something must be done" media demand - the tool will hurt conversation, and people, and potentially eventually threaten the platforms it runs on - but those problems feel smaller than the demand that "something must be done"

strogonoff 5y ago

The perceived goal is “detect toxicity”, but let’s unwind this goal a bit. Is it the lofty “make people be nice to each other”? Well, the paradox is that being nice is possible with the strongest choice of words, while being very harsh can sound most fluffy bunnies on the surface. In fact, in human relationships there are degrees of mutual familiarity where being exceedingly polite and not “insulting” your counterparty would be perceived as negative—where insults are not taken at face value, but rather as signifiers of friendliness (there’s a line, of course). Shall we unwind the perceived goal differently? Of course, the platform’s actual customers are the advertisers (we are talking about a hypothetical platform, but where is it really different?), and by being free to the user it participates in a very limited oligopoly of big social so no, it doesn’t really care about what anyone really meant or intended, and it definitely isn’t going to hire real humans who’d make an effort at grasping the context of the conversation. The real objective is for the platform to not have problems with law enforcement when one user complains about another user for being naughty, discriminatory, threatening, etc.—and, of course, we shouldn’t expect anything more from toxicity detectors geared towards that goal. As long as it’s not egregious enough that users leave en masse to a competitor (which can hardly exist, no honest business could reasonably compete with “free”) the platform wouldn’t care since users only matter to advertising revenue as cattle in numbers.

pwdisswordfish9 5y ago
The actual goal is ‘save money on hiring human moderators, while still maintaining the pretence of caring’.

boredumb 5y ago

The waste of resources on detecting what is decidedly "toxic" on internet forums is insanity. If you want to police communities online hire moderators, if your platform is so big you ""can't"" have moderators moderate then you are not in a position to be policing the platform. If you want to build puritanical devices to spam your moderators into doing a human review than that is your business but the use of machine learning for any sort of proactive policing is going to be a parody that will result in a sterile environment and/or a lot of bitter users.

mbg721 5y ago
There's still big money in steering opinion on your platform the way that you want it. "Toxic opinion detection" is somewhere between a euphemism and a side-benefit.

gillesjacobs 5y ago

I did research on this topic in a cyber safety research project. We focused on cyber bullying specifically but encountered the issue of non-toxic profanity as well. We used neural representation learning as well as feature methods and indeed, common profanity words are weak predictors in the better models. Still we found instances of non-toxic profanity being classified as bullying. An immediate solution is to apply multitask methods to your target dataset and include the one proposed in OP. It's always good to have more resources like this, even though SurgeHQ overstates the size of their resources by large margin in their copy. The 1000 post instances of their dataset is far from "the largest": I have several aggression, toxicity and bullying human-annotated datasets right here with over 100k instances.

ekanes 5y ago

There's a bunch of research that profanity increases trust. Perhaps because you're showing/sharing that you're not a corporate robot...

martin-t 5y ago

Wow, so much talk about how "shit" and "fuck" can be used unoffensively but almost no talk about how lots of "polite" speech without slurs can be incredibly offensive. For example lying is toxic and no ML model has a chance of detecting it (without understanding of the real world, which is pretty far away if achievable at all). For fucks sake, I saw a video of a crowd cheering after a guy imitated one of Hitler's speeches. There wasn't a single "shit" or "fuck" in it but it contained "truth doesn't matter, only victory". That's offensive as fuck but the people didn't see anything wrong with it.

Chris2048 5y ago
> it contained "truth doesn't matter, only victory". That's offensive as fuck Only in the context of knowing it came from Hitler. Without that information it isn't more than a strong opinion - because it isn't explicitly linked to murder/genocide.
- martin-t 5y ago
  "truth doesn't matter, only victory" is offensive in any political context. It doesn't matter if somebody is lying about genocide or bribes or abusing position to trade favors. One is worse than the other but it's still lying at a level that affects a whole country and that is intolerable.

motohagiography 5y ago

This description of the "toxicity model," triggers on ostensibly negative words, but really, they're just words we use to show polarization between figure and ground. Toxicity is poorly defined because it's an in-group euphemism for a kind of gendered disagreeableness, where its opposite or positive case is passive and agreeable, even passive aggressive. If there is such a thing as masculine aggression, there is also feminine aggression, and a lot of what we talk about as toxicity is really a criticism of masculine aggression using the lens or perspective of feminine aggression tools. I'd propose that when we say something is "toxic," we're talking about something that violates feminine norms around in-group alignment, security, reputation, reflection, impressions, "not a good look," etc. These are all things that require an imagined third party observer to potentially interpret and be offended by them, and are not codified by rules. Encoding this into an ML model is a lost cause, because you would need to reflect them through an AI that ran on pure neurotic animus to get a sense of whether something was toxic, or "not a good look." It's like assigning a sentiment score to someone saying, "Nice hair." The example in the article of "Fuck dude, nurses are the shit," is ranked as 98%+ "toxic," because it has two frickative swear words associated with masculine aggression traits (disagreeableness, provocation, profanity, rebellion, dissonance, loudness, etc.) and easier to write rules for - except those rules would also need to incorporate whether the phrase was an expression of aggression, or using the opposite to be wry, ironic, or in the case of the example, to express awe. I don't think we understand enough about psychology and people to really create effective moderation models with ML, and ML will necessarily create a kind of mean reversion in the discourse they monitor, which means all conversation subject to it will tend toward neutralization, which is essentially death. (Maybe we should fork an ML project for linking language with Jungian archetypes?) We can keep trying, and applying it as a fast search scheme for prioritizing outliers to human moderators, but pleasing an ML model is a recipe for intellectual sterility. I'd even argue the inflection point in the growth of social platforms is when moderation creates this kind of mean reversion, and you are left with the bland platitudes of blue checkmark types, being cheered on as you "grind," and boomer memes that aren't as funny as Family Circus comics. It's just death.

joedoejr 5y ago

10 years ago google failed at machine translation and NLP, giving hideous and meaningless statistical translation, same no wonder they fail at language understanding by training algorithm with Indians. They will to do science R&D is lowering every year.

herendin2 5y ago
Google's machine translation was terrible years ago, I agree. But it is now much better. Although it's still far from perfect, it has improved lots. Therefore I disagree that Google has failed at that

Handytinge 5y ago

The Americanisation (and generally Calificornication) of the internet is certainly a net negative for the other 95% of the worlds population. Regularly now I find social media sites telling me "do you want to review this before you post it? You're bullying or being offensive". No cunt, I'm good. Different words and phrasing have different impacts across cultures. Unfortunately Instagram gets to decide what my entire culture is allowed to say online. That's fucked.