8 ms·

AI service gives Wikipedians ‘X-ray specs’ to see through bad edits

garblegarble 11y ago
I can't help but wonder if this will have the side-effect that vandalism will become a more subtle subversion of the pages (or maybe there's already a number of people who do just that and this will thin out the field to make them more obvious)
- data_scientist 11y ago
  Relevant XKCD https://xkcd.com/810/ https://xkcd.com/810/
jacquesm 11y ago
Is there anything about the false positive rate of this system? Any binary 'good or bad' classifier should have at least a couple of stats attached to it to give an indication of how reliable the classifier is. To complete the circle, it would be nice to know how it performs according to this: https://en.wikipedia.org/wiki/Precision_and_recall https://en.wikipedia.org/wiki/Precision_and_recall
- kenrick95 11y ago
  Not really, but one can report a false positives and false negatives here: https://meta.wikimedia.org/wiki/Research_talk:Revision_scoring_as_a_service#Misclassifications https://meta.wikimedia.org/wiki/Research_talk:Revision_scori... Since it is activated on Indonesian Wikipedia (I helped them to extend the tool to Indonesian language), I noticed that this tool can hardly capture an obvious vandalism yet. Some other edits are hard to verify whether it is vandalism or not, even by human. I believe this is a work in progress and hope that it will improve over time. But hey, it is the first time a vandalism detection tool is deployed in Indonesian Wikipedia. :)
  - jacquesm 11y ago
    > Not really That's worrisome then. You'd expect a bit more rigor before putting a bot like that into production.
    - tptacek 11y ago
      It's not actually doing reversions, right? It's just scoring edits. Wikipedia editors keep hot-lists of articles they watch edits for, or patrol lists of edits or page creations; ostensibly, all this needs to do is sort those lists of edits.
      - kenrick95 11y ago
        Yup, it's just scoring the revisions. Currently there is no straightforward way to feedback to the tool for learning false positives yet.
      - jacquesm 11y ago
        It's augmentation, not making any decisions on its own (and that's logical given the false positive rate). So it's just an aid at this point but it can probably help quite a bit with triage.
    - halfak_wmf 11y ago
      Actually, we openly report a wide variety of statistics about our models. E.g. if you go to https://ores.wmflabs.org/scores/enwiki https://ores.wmflabs.org/scores/enwiki, the service lists out the models that are available for English Wikipedia along with test statistics. You can also browse our test statistics in our documentation. See https://meta.wikimedia.org/wiki/ORES/damaging#English_Wikipedia_.28enwiki.29 https://meta.wikimedia.org/wiki/ORES/damaging#English_Wikipe... Currently, we're optimizing for AUC since it (1) is hard to cheat -- unlike accuracy or precision/recall individually and (2) it represents the ranking problem well. See https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve https://en.wikipedia.org/wiki/Receiver_operating_characteris... "[AUC] is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one"
exo762 11y ago
This is very peculiar. New tool is introduced, which joins a set of existing tools (bots) which address the same problem. Some comparison of quality and robustness is expected. Instead author gives us some insight into his/hers source of inspiration: > A feminist inspiration > “Please exercise extreme caution to avoid encoding racism or other biases into an AI scheme.” > Wnt (from The Signpost) And few lines below we have this gem: > While artificial intelligence may prove essential for solving problems at Wikipedia’s scale, algorithms that replicate subjective judgements can also dehumanize and subjugate people and obfuscate inherent biases. Essence of not being sexist or racist is not to evaluate people basing on sex or race! Author instead accents that we should totally pay attention to subjective factors (such as sex or race) instead of objective factors (such as quality of edit). And let me guess whos edits will be targeted with extra scrutiny.. Edits of people who don't buy feminist agenda! (Disclosure, I'm egalitarian and MRA; While I fully support women rights, I just believe that modern feminism does not have anything to do with equal rights for anyone. This post nicely fall into this pattern.)
- bglazer 11y ago
  I agree the "feminist inspiration" part is a confusing non-sequitur. I think you're missing what they're trying to say, though. I interpreted that section as "we're making our algorithms and data open so that everyone can be sure they aren't implicitly racist or sexist". Suppose, for example, that a user repeatedly rejected edits for anyone with a noticeably feminine username. Wikimedia wants to be sure that their algorithms don't take this person's rejects to be valid data. Again, they don't really say how they'll do this, which makes the whole section rather confusing. It's a valid concern, just presented awkwardly.
  - HarryHirsch 11y ago
    Attempting to have processes that aren't implicity racist or sexist is a feminist practice?! If that doesn't qualify as an implicitly sexist statement I wouldn't know what would.
    - Chris2048 11y ago
cowardlydragon 11y ago
As referenced in the article, what is "good" and "bad" is dependent on perspective. Wikipedia has several perspectives just within the colloquial / perceived mission. Is it for academic integrity? For common education? For the rich and powerful? For the everyman? To record as much as possible? To cull the best knowledge from the stream? Is it for progressive ideals, which may be opposed by the common majority? Or does it reflect a conservative viewpoint, adopting only what has proven to coalesce in society? For who's society? What about two sides of a war? Or two equally opposing economic interests (entrenched oil vs alternative energy)? "good" or "bad" isn't a number...
- tptacek 11y ago
  "Good" simply means "unlikely to be reverted". The objective is to save time for editors.
  - HarryHirsch 11y ago
    Maybe the better metric is usefulness to the reader. A suitable algorithm can recognize things like "N.N. smells", but so can any user. The current algorithms are very effective at dealing with "XY is gay" and "Buy herbal viagra"; you don't see much of that. The really insidious stuff is spin doctors for companies and nations. So far, Wikipedia does a very poor job with geopolitical conflicts, and competing financial interests are not taken seriously enough either.
    - iraphael 11y ago
      But the people who are supposed to define what's good for the reader are the editors. They show the system what they think is good by either reverting edits or not, so by saying "this edit is likely to be reverted", the AI is saying "based on input from all editors, this edit may not be good for the users". Effectively, those are the same thing, unless you want to create a hard definition of "good for the users". But that's going around the editors, and taking their opinion of "good" out of the equation, which IMO is a very bad idea.
  - 11y ago
tptacek 11y ago
I'm surprised it took this long. Wikipedia has what would seem to be an excellent and straightforwardly encoded set of training data: a constant stream of edits and reversions.
- logicallee 11y ago
  That's not training a model to predict the quality of edits - that's training a model to predict the reactions of admins/editors :) Perhaps a better title is "AI service models Wikipedia editors"?
  - tptacek 11y ago
    That's all the model needs to do to be valuable.
    - logicallee 11y ago
      Well, suppose something told me what I would decide if I gave any matter 15 seconds of consideration. It would be an invaluable tool I might use often, for example for sorting and classifying, throwing out trash and so forth. For example, it would be a fantastic spam classifier, since even after 15 seconds there is no doubt. I could use it to get through hundreds of spam messages easily, or delegate things to appropriate departments and so forth. But it would not give me "x-ray vision", since at the end the day it is the same thing I myself conclude if I give the matter 15 seconds of consideration. Something that I miss - a genuine mail written the way a spam might be ("hey! it was nice to meet you!" in the subject, etc), by a recent acquaintance I forgot I made - would be misclassified as spam. There is no x-ray vision involved here - it's not training on the data - it's training on me! I feel the distinction is an important one.
      - tptacek 11y ago
        It's 15 seconds of consideration you have to give to huge volumes of edits, so singling out the ones where that 15 seconds is likely to be profitable for an editor is extremely helpful. (I have no idea how well the scoring works in practice). Remember, the editorial tasks we're talking about here are enervating.
zellyn 11y ago
Welcome to the spammer vs machine learning arms race, Wikipedia. :-) (Former YouTube Abuse team member)
- MollyR 11y ago
  Can't spammers also use machine learning, or maybe I misunderstood something.
  - zellyn 11y ago
    What I meant is that I expect a huge initial win, as this catches 80-90% of spammy edits, until the spammers figure out how to get around it. Hopefully the constant stream of human-created training data (in the form of reverted or non-reverted edits) helps with this. But undoubtedly there will need to be some continual feature engineering to stay ahead.
    - halfak_wmf 11y ago
      Yeah. We're actively working with patrollers to chase down mistakes and retrain the models. This is a system that will require continual improvement. Luckily, there's a lot of researchers working on this problem so we're drawing from a lot of brilliance beyond our little team at the Wikimedia Foundation.
- kuschku 11y ago
  Well, Wikipedia at least has humans working on it, who can catch issues and improve it. It can’t be worse than YouTube/Google’s systems anyway.
  - titanomachy 11y ago
    I don't see that many bot-generated comments anymore on YouTube. The genuine user-generated comments are among the lowest-quality on any site that I frequent, but I suspect there isn't much that Google can do about that.
Estragon 11y ago
In case anyone else is interested in the source code, it's here: https://github.com/wiki-ai/ores https://github.com/wiki-ai/ores

I can't help but wonder if this will have the side-effect that vandalism will become a more subtle subversion of the pages (or maybe there's already a number of people who do just that and this will thin out the field to make them more obvious)

data_scientist 11y ago
Relevant XKCD https://xkcd.com/810/ https://xkcd.com/810/

jacquesm 11y ago

Is there anything about the false positive rate of this system? Any binary 'good or bad' classifier should have at least a couple of stats attached to it to give an indication of how reliable the classifier is. To complete the circle, it would be nice to know how it performs according to this: https://en.wikipedia.org/wiki/Precision_and_recall https://en.wikipedia.org/wiki/Precision_and_recall

kenrick95 11y ago
Not really, but one can report a false positives and false negatives here: https://meta.wikimedia.org/wiki/Research_talk:Revision_scoring_as_a_service#Misclassifications https://meta.wikimedia.org/wiki/Research_talk:Revision_scori... Since it is activated on Indonesian Wikipedia (I helped them to extend the tool to Indonesian language), I noticed that this tool can hardly capture an obvious vandalism yet. Some other edits are hard to verify whether it is vandalism or not, even by human. I believe this is a work in progress and hope that it will improve over time. But hey, it is the first time a vandalism detection tool is deployed in Indonesian Wikipedia. :)
- jacquesm 11y ago
  > Not really That's worrisome then. You'd expect a bit more rigor before putting a bot like that into production.
  - tptacek 11y ago
    It's not actually doing reversions, right? It's just scoring edits. Wikipedia editors keep hot-lists of articles they watch edits for, or patrol lists of edits or page creations; ostensibly, all this needs to do is sort those lists of edits.
    - kenrick95 11y ago
      Yup, it's just scoring the revisions. Currently there is no straightforward way to feedback to the tool for learning false positives yet.
    - jacquesm 11y ago
      It's augmentation, not making any decisions on its own (and that's logical given the false positive rate). So it's just an aid at this point but it can probably help quite a bit with triage.
  - halfak_wmf 11y ago
    Actually, we openly report a wide variety of statistics about our models. E.g. if you go to https://ores.wmflabs.org/scores/enwiki https://ores.wmflabs.org/scores/enwiki, the service lists out the models that are available for English Wikipedia along with test statistics. You can also browse our test statistics in our documentation. See https://meta.wikimedia.org/wiki/ORES/damaging#English_Wikipedia_.28enwiki.29 https://meta.wikimedia.org/wiki/ORES/damaging#English_Wikipe... Currently, we're optimizing for AUC since it (1) is hard to cheat -- unlike accuracy or precision/recall individually and (2) it represents the ranking problem well. See https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve https://en.wikipedia.org/wiki/Receiver_operating_characteris... "[AUC] is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one"

exo762 11y ago

This is very peculiar. New tool is introduced, which joins a set of existing tools (bots) which address the same problem. Some comparison of quality and robustness is expected. Instead author gives us some insight into his/hers source of inspiration: > A feminist inspiration > “Please exercise extreme caution to avoid encoding racism or other biases into an AI scheme.” > Wnt (from The Signpost) And few lines below we have this gem: > While artificial intelligence may prove essential for solving problems at Wikipedia’s scale, algorithms that replicate subjective judgements can also dehumanize and subjugate people and obfuscate inherent biases. Essence of not being sexist or racist is not to evaluate people basing on sex or race! Author instead accents that we should totally pay attention to subjective factors (such as sex or race) instead of objective factors (such as quality of edit). And let me guess whos edits will be targeted with extra scrutiny.. Edits of people who don't buy feminist agenda! (Disclosure, I'm egalitarian and MRA; While I fully support women rights, I just believe that modern feminism does not have anything to do with equal rights for anyone. This post nicely fall into this pattern.)

bglazer 11y ago
I agree the "feminist inspiration" part is a confusing non-sequitur. I think you're missing what they're trying to say, though. I interpreted that section as "we're making our algorithms and data open so that everyone can be sure they aren't implicitly racist or sexist". Suppose, for example, that a user repeatedly rejected edits for anyone with a noticeably feminine username. Wikimedia wants to be sure that their algorithms don't take this person's rejects to be valid data. Again, they don't really say how they'll do this, which makes the whole section rather confusing. It's a valid concern, just presented awkwardly.
- HarryHirsch 11y ago
  Attempting to have processes that aren't implicity racist or sexist is a feminist practice?! If that doesn't qualify as an implicitly sexist statement I wouldn't know what would.
  - Chris2048 11y ago

cowardlydragon 11y ago

As referenced in the article, what is "good" and "bad" is dependent on perspective. Wikipedia has several perspectives just within the colloquial / perceived mission. Is it for academic integrity? For common education? For the rich and powerful? For the everyman? To record as much as possible? To cull the best knowledge from the stream? Is it for progressive ideals, which may be opposed by the common majority? Or does it reflect a conservative viewpoint, adopting only what has proven to coalesce in society? For who's society? What about two sides of a war? Or two equally opposing economic interests (entrenched oil vs alternative energy)? "good" or "bad" isn't a number...

tptacek 11y ago
"Good" simply means "unlikely to be reverted". The objective is to save time for editors.
- HarryHirsch 11y ago
  Maybe the better metric is usefulness to the reader. A suitable algorithm can recognize things like "N.N. smells", but so can any user. The current algorithms are very effective at dealing with "XY is gay" and "Buy herbal viagra"; you don't see much of that. The really insidious stuff is spin doctors for companies and nations. So far, Wikipedia does a very poor job with geopolitical conflicts, and competing financial interests are not taken seriously enough either.
  - iraphael 11y ago
    But the people who are supposed to define what's good for the reader are the editors. They show the system what they think is good by either reverting edits or not, so by saying "this edit is likely to be reverted", the AI is saying "based on input from all editors, this edit may not be good for the users". Effectively, those are the same thing, unless you want to create a hard definition of "good for the users". But that's going around the editors, and taking their opinion of "good" out of the equation, which IMO is a very bad idea.
- 11y ago

tptacek 11y ago

I'm surprised it took this long. Wikipedia has what would seem to be an excellent and straightforwardly encoded set of training data: a constant stream of edits and reversions.

logicallee 11y ago
That's not training a model to predict the quality of edits - that's training a model to predict the reactions of admins/editors :) Perhaps a better title is "AI service models Wikipedia editors"?
- tptacek 11y ago
  That's all the model needs to do to be valuable.
  - logicallee 11y ago
    Well, suppose something told me what I would decide if I gave any matter 15 seconds of consideration. It would be an invaluable tool I might use often, for example for sorting and classifying, throwing out trash and so forth. For example, it would be a fantastic spam classifier, since even after 15 seconds there is no doubt. I could use it to get through hundreds of spam messages easily, or delegate things to appropriate departments and so forth. But it would not give me "x-ray vision", since at the end the day it is the same thing I myself conclude if I give the matter 15 seconds of consideration. Something that I miss - a genuine mail written the way a spam might be ("hey! it was nice to meet you!" in the subject, etc), by a recent acquaintance I forgot I made - would be misclassified as spam. There is no x-ray vision involved here - it's not training on the data - it's training on me! I feel the distinction is an important one.
    - tptacek 11y ago
      It's 15 seconds of consideration you have to give to huge volumes of edits, so singling out the ones where that 15 seconds is likely to be profitable for an editor is extremely helpful. (I have no idea how well the scoring works in practice). Remember, the editorial tasks we're talking about here are enervating.

zellyn 11y ago

Welcome to the spammer vs machine learning arms race, Wikipedia. :-) (Former YouTube Abuse team member)

MollyR 11y ago
Can't spammers also use machine learning, or maybe I misunderstood something.
- zellyn 11y ago
  What I meant is that I expect a huge initial win, as this catches 80-90% of spammy edits, until the spammers figure out how to get around it. Hopefully the constant stream of human-created training data (in the form of reverted or non-reverted edits) helps with this. But undoubtedly there will need to be some continual feature engineering to stay ahead.
  - halfak_wmf 11y ago
    Yeah. We're actively working with patrollers to chase down mistakes and retrain the models. This is a system that will require continual improvement. Luckily, there's a lot of researchers working on this problem so we're drawing from a lot of brilliance beyond our little team at the Wikimedia Foundation.
kuschku 11y ago
Well, Wikipedia at least has humans working on it, who can catch issues and improve it. It can’t be worse than YouTube/Google’s systems anyway.
- titanomachy 11y ago
  I don't see that many bot-generated comments anymore on YouTube. The genuine user-generated comments are among the lowest-quality on any site that I frequent, but I suspect there isn't much that Google can do about that.

Estragon 11y ago

In case anyone else is interested in the source code, it's here: https://github.com/wiki-ai/ores https://github.com/wiki-ai/ores