6 ms·

Question to people interested in designing those types of models : what does it matter if the output is "toxic" ? People are toxic ! I find the idea of sanitizi

by Bayart 5y ago

Question to people interested in designing those types of models : what does it matter if the output is "toxic" ? People are toxic ! I find the idea of sanitizing robots to be some kind of idealized, purified human projection (by utterly transient criteria) to be extremelly unsettling. The current sanitization of public discourse is already terrifying. Now you want that crap to be AI-powered ?!

tasogare 5y ago
Right, especially since what's considered "toxic" is a moving needle (that quote of the General de Gaulle that Zemmour like to use for instance was perfectly fine at the time it was said and still is for a lot of people) and is also not uniform across the world. In particular sexual jokes or double entendre are perfectly fine in French most of time while they're often considered harassment in the US culture.
twanvl 5y ago
It matters if people figure out how to make your model output inappropriate things and start sharing screenshots on twitter. What if you use a GPT model as part of a chat bot? Then you definitely don't want to risk any sexual or discriminatory output for example. If that was a person talking, they would be fired.
- hungryforcodes 5y ago
  There is no bot to fire here though, as it's not employed.
bckr 5y ago
We could think of language models as public figures and developers and users as supporters. Every public figure has heard nasty language: it's up to you to decide whether to support a public figure who goes around repeating said language.
jhgb 5y ago
Even worse, filtering the data in this way makes the model biased, i.e., not a useful model. Imagine for example that someone created a model of human behavior...but filtered out behavior considered "toxic" (whatever that means). Then someone else would naively create for example a simulation of the reaction of a large group of people to some expected events. Now there's a chance that if these events occurred, that mass of people would actually react very differently...because the model was biased! I.e., it did not approximate the average reaction since reactions in some directions were artificially clamped. So now any decisions based on that model would be worthless.
- skybrian 5y ago
  That’s making a lot of assumptions. This isn’t the sort of model that you could base such a simulation on. It’s glorified autocomplete. (Impressive for what it is, but unreliable.)
  - jhgb 5y ago
    I'm speaking in generic terms, of course. All models should be at the very least unbiased, though -- that's the least demand you can make of them (and generally an easy one to check, compared to other properties, such as having minimum variance of all possible models and such.)
    - skybrian 5y ago
      I sort of agree in principle but I’m not sure what unbiased means for a language model or how you would measure it, so this doesn’t sound all that easy to me.

by Bayart 5y ago

tasogare 5y ago

Right, especially since what's considered "toxic" is a moving needle (that quote of the General de Gaulle that Zemmour like to use for instance was perfectly fine at the time it was said and still is for a lot of people) and is also not uniform across the world. In particular sexual jokes or double entendre are perfectly fine in French most of time while they're often considered harassment in the US culture.

twanvl 5y ago

It matters if people figure out how to make your model output inappropriate things and start sharing screenshots on twitter. What if you use a GPT model as part of a chat bot? Then you definitely don't want to risk any sexual or discriminatory output for example. If that was a person talking, they would be fired.

hungryforcodes 5y ago
There is no bot to fire here though, as it's not employed.

bckr 5y ago

We could think of language models as public figures and developers and users as supporters. Every public figure has heard nasty language: it's up to you to decide whether to support a public figure who goes around repeating said language.

jhgb 5y ago

Even worse, filtering the data in this way makes the model biased, i.e., not a useful model. Imagine for example that someone created a model of human behavior...but filtered out behavior considered "toxic" (whatever that means). Then someone else would naively create for example a simulation of the reaction of a large group of people to some expected events. Now there's a chance that if these events occurred, that mass of people would actually react very differently...because the model was biased! I.e., it did not approximate the average reaction since reactions in some directions were artificially clamped. So now any decisions based on that model would be worthless.

skybrian 5y ago
That’s making a lot of assumptions. This isn’t the sort of model that you could base such a simulation on. It’s glorified autocomplete. (Impressive for what it is, but unreliable.)
- jhgb 5y ago
  I'm speaking in generic terms, of course. All models should be at the very least unbiased, though -- that's the least demand you can make of them (and generally an easy one to check, compared to other properties, such as having minimum variance of all possible models and such.)
  - skybrian 5y ago
    I sort of agree in principle but I’m not sure what unbiased means for a language model or how you would measure it, so this doesn’t sound all that easy to me.