7 ms·

Explanation from their Discord (more info/links below) “Earlier tonight, we started having an outage using OpenAI’s GPT-3 Davinci model, which caused the show

Explanation from their Discord (more info/links below) “Earlier tonight, we started having an outage using OpenAI’s GPT-3 Davinci model, which caused the show to exhibit errant behaviors (you may have seen empty rooms cycling through). OpenAI has a less sophisticated model, Curie, that was the predecessor to Davinci. When davinci started failing, we switched over to Curie to try to keep the show running without any downtime. The switch to Curie was what resulted in the inappropriate text being generated.” —— A bit more detail: https://www.vice.com/en/article/y3pymx/ai-generated-seinfeld-show-nothing-forever-banned-on-twitch-after-transphobic-standup-bit https://www.vice.com/en/article/y3pymx/ai-generated-seinfeld... And screenshots from their Discord with an explanation (this is quoted in the article above) https://twitter.com/ryanbrianjones/status/1622548545898250242?s=46&t=ezKYHgVdG_1QsYDiSHP69Q https://twitter.com/ryanbrianjones/status/162254854589825024...

cobertos 4y ago
A seemingly innocuous engineering change led to some drastic consequences. Imagine the other more critical production applications who have made the same reasonable engineering decision.
- draw_down 4y ago
  I dunno about that description. They essentially put a different brain in their guy’s body and he started acting different. One could consider that expected behavior.
  - bogwog 4y ago
    [flagged]
  - cobertos 4y ago
    This sounds logical, though my brain has always stored Curie as just like a Davinci divided by 2. They're right next two each other in the sandbox drop-down, it _feels_ swappable, in an innocuous way, or at least it did to me until this point.
- anononaut 4y ago
  I challenge your interpretation of a joke generator making a joke as "drastic consequences." Twitch has arcane rules about what can be streamed. As a streamer you have to tip-toe around topics (or jokes) regarding specific protected classes. This is a matter of arbitrary lines in the sand.
  - Macha 4y ago
    For a joke generator, making an offensive joke that gets the generator banned from the primary source it uses to reach users seems pretty high up their on the range of negative possible outcomes. That's the point of the comment, if you were applying AI in more serious scenarios, would this be an AI confidently advising a doctor to overdose a patient, or advising a judge to incarcerate someone based on racist inferences from the set of people currently incarcerated?
    - iso1631 4y ago
      > For a joke generator I'm confused, I thought it was supposed to be generating Seinfeld episodes, not jokes?
      - adamrezich 4y ago
        Seinfeld episodes usually open and close with Jerry doing standup in a club, a format that Nothing, Forever emulates.
    - neuronexmachina 4y ago
      Or mental health: https://gizmodo.com/mental-health-therapy-app-ai-koko-chatgpt-rob-morris-1849965534 https://gizmodo.com/mental-health-therapy-app-ai-koko-chatgp...
  - causi 4y ago
    Am I wrong in thinking that getting your feelings hurt by an AI making an offensive joke is the equivalent of thinking you've been doxxed when a random number generator spits out your phone number?
- version_five 4y ago
  Drastic is a bit strong. A cartoon character on an infinite streaming tv experiment made a bad joke (which appears to be more of a bad meta-joke). The fact that the stakes are so low is presumably why they changed the model. If it was flying an airplane or something that mattered, they probably would have taken more care. The change they made had a completely acceptable risk / consequence tradeoff
  - yifanl 4y ago
    Drastic is relative to the expected impact of the application. There isn't much worse that can possibly happen to a bot-generated stream than having the stream taken down by the powers-that-be.
    - timr 4y ago
      This is like saying that having something taken down by a mod on reddit is a "drastic consequence". I mean, sure...the content is removed, but in most cases, the reason for the action is subjective and somewhat arbitrary. To me, this is less a commentary on AI than on the absurd sensitivities of community moderation.
      - madeofpalk 4y ago
        "Website makes moderation decision some people disagree with" is a lot less novel or interesting.
      - yifanl 4y ago
        Again, the drasticness of a consequence is defined by the system you're trying to build for. If you build a bot to automate your streaming so you can have 100% stream uptime and you end up with 0% uptime, that's because your product failed drastically. (Keeping in mind, as arbitrary and subjective the rules of Twitch are, the vast majority of human streamers have never been banned for arbitrary moderation)
      - tinus_hn 4y ago
        For many people Twitch streams are their primary source of income, this kind of ban can have quite a lot of impact. And then besides the ban there is the chilling effect because no one wants to skirt the rules and risk a ban. And that is what allows social justice warriors to dictate what others can make jokes about.
- gardenhedge 4y ago
  > innocuous engineering change > reasonable engineering decision They swapped out the AI model. That's kind of a big deal.
  - cobertos 4y ago
    In my mind, if I look at a drop-down and "model D" is down but "model C" is up and I need the API running NOW, I'm going to change the option to the one that's working. That being said, every company I've been at with a production outage has gone into "fix it and ask questions later" mode. Maybe other companies have more process? But when you put it like you said in your comment, it does feel more weighty
    - gardenhedge 4y ago
      In contrast the (non-AI) companies I've worked with have a rigorous change release process which includes impact assessments, back out plans, getting multiple sign offs including QA testing and post-release testing, etc.
      - cobertos 4y ago
        I mean specifically for a production outage fix. For planned changes, most of the companies I've worked at have similar release processes
- hourago 4y ago
  > A seemingly innocuous engineering change led to some drastic consequences. If you think that this is "drastic" you should read "Case Study 4: The $440 Million Software Error at Knight Capital". https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-... tldr; At Knight, some new trading software contained a flaw that became apparent only after the software was activated when the New York Stock Exchange (NYSE) opened that day. The errant software sent Knight on a buying spree, snapping up 150 different stocks at a total cost of around $7 billion, all in the first hour of trading.
  - KMag 4y ago
    I was developing trading systems at Goldman then. I remember that day. I got a call that evening to come in early the next day to discuss additional preventions and mitigations to implement. Scarier was the day I put out a change request shortly before the end of the day in Asia, went out to dinner and drinks with some colleagues, stopped by my office on the way home, and merged the approved change request, during early morning trading in the US. The next day (in Asia) I woke up to discover Goldman had a roughly 28 million dollar trading loss. I spent a couple of minutes proving to myself that my change couldn't possibly have been the cause, and then realized I would have been woken in the middle of the night by a phone call if there was any possibility that my change had caused the trading loss. Since then, I don't merge code at the end of the day, and try to avoid Fridays. I was already in the habit of not coding if I had consumed any alcohol, but added the habit of not merging if I've consumed any alcohol. There's nothing like a 28 million dollar panic to get your practices in shape.
  - cobertos 4y ago
    I meant drastic for their use case. They put all this work into a piece of software specifically for streaming and it gets banned from the streaming platform. Definitely not life altering or a safety issue, but worst case for them, and I can imagine this going similarly wrong in a different piece of software where stakes actually matter
- bravetraveler 4y ago
  Is changing the source of data really that innocuous? I find it not that surprising that the output changed with the input Especially considering this is the description for the model they left: > higher quality, longer output and better instruction-following and this is what they moved to: > Very capable, but faster and lower cost than Davinci. If nothing else, the adage 'you get what you pay for' works. ref: https://platform.openai.com/docs/models https://platform.openai.com/docs/models
  - cobertos 4y ago
    I've only been through a couple production outages, but at those companies, usually we'd get things patched and ask questions later. I could definitely imagine switching to another model that is made to sound like a smaller version of Davinci, at least from how it's been stored in my mind up until now.
    - bravetraveler 4y ago
      I've been there quite a bit, and it's pretty true! It's best to worry about how the ship started sinking after you've saved it :p In that situation I still try to exercise my options a bit... Seeing that this new model is implicitly worse at following directions, knowing that 'production' (this is a Twitch channel) depends on obeying a certain set of rules... switching to it may be ill-advised. I don't want to be seen as overly judgmental - the Twitch channel comment works both ways; derogatory/supportive. I get why one wouldn't really foresee this or even care to. Then I wonder, why not leave it down for a bit? The hype train is fickle but it's not that precious, either.
- skim_milk 4y ago
  They've been developing this on-the-fly since it gained popularity (and profitability). Basically, they ran fast and broke things, like any good startup. I'm nearly 100% certain that the scripts are generated through prompt engineering, with a random prompt (e.g. tell a joke prompt, talk about a new restaurant prompt) being selected for the scene. From what I can gather they first used the older, cheaper GPT-3 models, only upgrading to davinci-003 when it was profitable. The older GPT-3 models proved fine and didn't generate edgy content for the several months they were up and running. But I think the change that broke the camels back was they added a "2006 Laugh Factory incident with edgy content" prompt and only tested it on the davinci-003 model - the new models having been wiped clean of antisocial training data, while the older smaller models still having contentious content encoded in the model. So, davinci-003 did fine producing "politically aligned" text with the "edgy" prompt because it's "cleaned", but when the openai API for davinci went down the fallback was curie. The older "unclean" curie model combined with an edgy prompt inevitably caused what we saw here.
  - ethbr0 4y ago
    My takeaway is that the more we turn to larger and larger models (developed by third parties) that obscure their underlying nuances, the more risky swapping models in a particular domain becomes. Cue MRE/MLOps job listings.
sillysaurusx 4y ago
Is there any info on how long the ban is for? I assume a week, but forsen’s was a month.
- nemothekid 4y ago
  14 days according to the screenshots posted by the GP
- hot_gril 4y ago
  0 days if you make a new account
Cthulhu_ 4y ago
This is why, like with every live show, you should have a delay and moderation so you can intervene before unwanted things are broadcasted and get you into trouble.
guywithahat 4y ago
This makes sense because most of the political ChatGPT headlines I see are it refusing to say nice things about republicans or crashing because it's unable to say anything discriminatory. Using a different model here makes sense
- deleted 4y ago
  [deleted]
- pleadthefifth 4y ago
  [flagged]
paganel 4y ago
What’s “errant” about it? Lots of transphobic people around, if anything I’d say the AI was closer to “real life” than lots of other AI systems.
- GenerocUsername 4y ago
  Would it have been transphobic if the sentence had been prefixed with 'some people say...' At a certain level AI trained on inputs is just a big 'some people say' box.
  - paganel 4y ago
    People saying transphobic stuff are indeed transphobic.
    - GenerocUsername 4y ago
      Not looking for a flame war, and I know this is not an acceptable take, but y'all need to learn the difference between conversations about a topic and conversations supporting a topic.
    - mensetmanusman 4y ago
      That assumes an objective ruler who defines what is transphobic.
loa_in_ 4y ago
I like how the most "additional quoted" and "explained" reason has absolutely no merit because there's no automatic fallback between models. It only falls back to older model if you program it to.
mariodiana 4y ago
So, I watched the clip, and honestly that sounds more Norm MacDonald than Jerry Seinfeld.
- endtime 4y ago
  Definitely not Norm. Norm would (and did!) say he loved transgender people, and would leave ambiguous whether he was being candid. (Or he'd pretend to struggle with his love for transgenderism versus his distaste for "psychosexual sadists who slay people" - not looking it up but pretty sure that's an accurate quote. The joke of course being that he is confusing two different categories of people in a shocking way, playing up his clueless character.) He liked bombing, but only with jokes he believed in. He wouldn't tell a meta-joke like this with such a lack of punchline.

cobertos 4y ago

A seemingly innocuous engineering change led to some drastic consequences. Imagine the other more critical production applications who have made the same reasonable engineering decision.

draw_down 4y ago
I dunno about that description. They essentially put a different brain in their guy’s body and he started acting different. One could consider that expected behavior.
- bogwog 4y ago
  [flagged]
- cobertos 4y ago
  This sounds logical, though my brain has always stored Curie as just like a Davinci divided by 2. They're right next two each other in the sandbox drop-down, it _feels_ swappable, in an innocuous way, or at least it did to me until this point.
anononaut 4y ago
I challenge your interpretation of a joke generator making a joke as "drastic consequences." Twitch has arcane rules about what can be streamed. As a streamer you have to tip-toe around topics (or jokes) regarding specific protected classes. This is a matter of arbitrary lines in the sand.
- Macha 4y ago
  For a joke generator, making an offensive joke that gets the generator banned from the primary source it uses to reach users seems pretty high up their on the range of negative possible outcomes. That's the point of the comment, if you were applying AI in more serious scenarios, would this be an AI confidently advising a doctor to overdose a patient, or advising a judge to incarcerate someone based on racist inferences from the set of people currently incarcerated?
  - iso1631 4y ago
    > For a joke generator I'm confused, I thought it was supposed to be generating Seinfeld episodes, not jokes?
    - adamrezich 4y ago
      Seinfeld episodes usually open and close with Jerry doing standup in a club, a format that Nothing, Forever emulates.
  - neuronexmachina 4y ago
    Or mental health: https://gizmodo.com/mental-health-therapy-app-ai-koko-chatgpt-rob-morris-1849965534 https://gizmodo.com/mental-health-therapy-app-ai-koko-chatgp...
- causi 4y ago
  Am I wrong in thinking that getting your feelings hurt by an AI making an offensive joke is the equivalent of thinking you've been doxxed when a random number generator spits out your phone number?
version_five 4y ago
Drastic is a bit strong. A cartoon character on an infinite streaming tv experiment made a bad joke (which appears to be more of a bad meta-joke). The fact that the stakes are so low is presumably why they changed the model. If it was flying an airplane or something that mattered, they probably would have taken more care. The change they made had a completely acceptable risk / consequence tradeoff
- yifanl 4y ago
  Drastic is relative to the expected impact of the application. There isn't much worse that can possibly happen to a bot-generated stream than having the stream taken down by the powers-that-be.
  - timr 4y ago
    This is like saying that having something taken down by a mod on reddit is a "drastic consequence". I mean, sure...the content is removed, but in most cases, the reason for the action is subjective and somewhat arbitrary. To me, this is less a commentary on AI than on the absurd sensitivities of community moderation.
    - madeofpalk 4y ago
      "Website makes moderation decision some people disagree with" is a lot less novel or interesting.
    - yifanl 4y ago
      Again, the drasticness of a consequence is defined by the system you're trying to build for. If you build a bot to automate your streaming so you can have 100% stream uptime and you end up with 0% uptime, that's because your product failed drastically. (Keeping in mind, as arbitrary and subjective the rules of Twitch are, the vast majority of human streamers have never been banned for arbitrary moderation)
    - tinus_hn 4y ago
      For many people Twitch streams are their primary source of income, this kind of ban can have quite a lot of impact. And then besides the ban there is the chilling effect because no one wants to skirt the rules and risk a ban. And that is what allows social justice warriors to dictate what others can make jokes about.
gardenhedge 4y ago
> innocuous engineering change > reasonable engineering decision They swapped out the AI model. That's kind of a big deal.
- cobertos 4y ago
  In my mind, if I look at a drop-down and "model D" is down but "model C" is up and I need the API running NOW, I'm going to change the option to the one that's working. That being said, every company I've been at with a production outage has gone into "fix it and ask questions later" mode. Maybe other companies have more process? But when you put it like you said in your comment, it does feel more weighty
  - gardenhedge 4y ago
    In contrast the (non-AI) companies I've worked with have a rigorous change release process which includes impact assessments, back out plans, getting multiple sign offs including QA testing and post-release testing, etc.
    - cobertos 4y ago
      I mean specifically for a production outage fix. For planned changes, most of the companies I've worked at have similar release processes
hourago 4y ago
> A seemingly innocuous engineering change led to some drastic consequences. If you think that this is "drastic" you should read "Case Study 4: The $440 Million Software Error at Knight Capital". https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-... tldr; At Knight, some new trading software contained a flaw that became apparent only after the software was activated when the New York Stock Exchange (NYSE) opened that day. The errant software sent Knight on a buying spree, snapping up 150 different stocks at a total cost of around $7 billion, all in the first hour of trading.
- KMag 4y ago
  I was developing trading systems at Goldman then. I remember that day. I got a call that evening to come in early the next day to discuss additional preventions and mitigations to implement. Scarier was the day I put out a change request shortly before the end of the day in Asia, went out to dinner and drinks with some colleagues, stopped by my office on the way home, and merged the approved change request, during early morning trading in the US. The next day (in Asia) I woke up to discover Goldman had a roughly 28 million dollar trading loss. I spent a couple of minutes proving to myself that my change couldn't possibly have been the cause, and then realized I would have been woken in the middle of the night by a phone call if there was any possibility that my change had caused the trading loss. Since then, I don't merge code at the end of the day, and try to avoid Fridays. I was already in the habit of not coding if I had consumed any alcohol, but added the habit of not merging if I've consumed any alcohol. There's nothing like a 28 million dollar panic to get your practices in shape.
- cobertos 4y ago
  I meant drastic for their use case. They put all this work into a piece of software specifically for streaming and it gets banned from the streaming platform. Definitely not life altering or a safety issue, but worst case for them, and I can imagine this going similarly wrong in a different piece of software where stakes actually matter
bravetraveler 4y ago
Is changing the source of data really that innocuous? I find it not that surprising that the output changed with the input Especially considering this is the description for the model they left: > higher quality, longer output and better instruction-following and this is what they moved to: > Very capable, but faster and lower cost than Davinci. If nothing else, the adage 'you get what you pay for' works. ref: https://platform.openai.com/docs/models https://platform.openai.com/docs/models
- cobertos 4y ago
  I've only been through a couple production outages, but at those companies, usually we'd get things patched and ask questions later. I could definitely imagine switching to another model that is made to sound like a smaller version of Davinci, at least from how it's been stored in my mind up until now.
  - bravetraveler 4y ago
    I've been there quite a bit, and it's pretty true! It's best to worry about how the ship started sinking after you've saved it :p In that situation I still try to exercise my options a bit... Seeing that this new model is implicitly worse at following directions, knowing that 'production' (this is a Twitch channel) depends on obeying a certain set of rules... switching to it may be ill-advised. I don't want to be seen as overly judgmental - the Twitch channel comment works both ways; derogatory/supportive. I get why one wouldn't really foresee this or even care to. Then I wonder, why not leave it down for a bit? The hype train is fickle but it's not that precious, either.
skim_milk 4y ago
They've been developing this on-the-fly since it gained popularity (and profitability). Basically, they ran fast and broke things, like any good startup. I'm nearly 100% certain that the scripts are generated through prompt engineering, with a random prompt (e.g. tell a joke prompt, talk about a new restaurant prompt) being selected for the scene. From what I can gather they first used the older, cheaper GPT-3 models, only upgrading to davinci-003 when it was profitable. The older GPT-3 models proved fine and didn't generate edgy content for the several months they were up and running. But I think the change that broke the camels back was they added a "2006 Laugh Factory incident with edgy content" prompt and only tested it on the davinci-003 model - the new models having been wiped clean of antisocial training data, while the older smaller models still having contentious content encoded in the model. So, davinci-003 did fine producing "politically aligned" text with the "edgy" prompt because it's "cleaned", but when the openai API for davinci went down the fallback was curie. The older "unclean" curie model combined with an edgy prompt inevitably caused what we saw here.
- ethbr0 4y ago
  My takeaway is that the more we turn to larger and larger models (developed by third parties) that obscure their underlying nuances, the more risky swapping models in a particular domain becomes. Cue MRE/MLOps job listings.

sillysaurusx 4y ago

Is there any info on how long the ban is for? I assume a week, but forsen’s was a month.

nemothekid 4y ago
14 days according to the screenshots posted by the GP
hot_gril 4y ago
0 days if you make a new account

Cthulhu_ 4y ago

This is why, like with every live show, you should have a delay and moderation so you can intervene before unwanted things are broadcasted and get you into trouble.

guywithahat 4y ago

This makes sense because most of the political ChatGPT headlines I see are it refusing to say nice things about republicans or crashing because it's unable to say anything discriminatory. Using a different model here makes sense

deleted 4y ago
[deleted]
pleadthefifth 4y ago
[flagged]

paganel 4y ago

What’s “errant” about it? Lots of transphobic people around, if anything I’d say the AI was closer to “real life” than lots of other AI systems.

GenerocUsername 4y ago
Would it have been transphobic if the sentence had been prefixed with 'some people say...' At a certain level AI trained on inputs is just a big 'some people say' box.
- paganel 4y ago
  People saying transphobic stuff are indeed transphobic.
  - GenerocUsername 4y ago
    Not looking for a flame war, and I know this is not an acceptable take, but y'all need to learn the difference between conversations about a topic and conversations supporting a topic.
  - mensetmanusman 4y ago
    That assumes an objective ruler who defines what is transphobic.

loa_in_ 4y ago

I like how the most "additional quoted" and "explained" reason has absolutely no merit because there's no automatic fallback between models. It only falls back to older model if you program it to.

mariodiana 4y ago

So, I watched the clip, and honestly that sounds more Norm MacDonald than Jerry Seinfeld.

endtime 4y ago
Definitely not Norm. Norm would (and did!) say he loved transgender people, and would leave ambiguous whether he was being candid. (Or he'd pretend to struggle with his love for transgenderism versus his distaste for "psychosexual sadists who slay people" - not looking it up but pretty sure that's an accurate quote. The joke of course being that he is confusing two different categories of people in a shocking way, playing up his clueless character.) He liked bombing, but only with jokes he believed in. He wouldn't tell a meta-joke like this with such a lack of punchline.