11 ms·

A seemingly innocuous engineering change led to some drastic consequences. Imagine the other more critical production applications who have made the same reaso

A seemingly innocuous engineering change led to some drastic consequences. Imagine the other more critical production applications who have made the same reasonable engineering decision.

draw_down 4y ago
I dunno about that description. They essentially put a different brain in their guy’s body and he started acting different. One could consider that expected behavior.
- bogwog 4y ago
  [flagged]
- cobertos 4y ago
  This sounds logical, though my brain has always stored Curie as just like a Davinci divided by 2. They're right next two each other in the sandbox drop-down, it _feels_ swappable, in an innocuous way, or at least it did to me until this point.
anononaut 4y ago
I challenge your interpretation of a joke generator making a joke as "drastic consequences." Twitch has arcane rules about what can be streamed. As a streamer you have to tip-toe around topics (or jokes) regarding specific protected classes. This is a matter of arbitrary lines in the sand.
- Macha 4y ago
  For a joke generator, making an offensive joke that gets the generator banned from the primary source it uses to reach users seems pretty high up their on the range of negative possible outcomes. That's the point of the comment, if you were applying AI in more serious scenarios, would this be an AI confidently advising a doctor to overdose a patient, or advising a judge to incarcerate someone based on racist inferences from the set of people currently incarcerated?
  - iso1631 4y ago
    > For a joke generator I'm confused, I thought it was supposed to be generating Seinfeld episodes, not jokes?
    - adamrezich 4y ago
      Seinfeld episodes usually open and close with Jerry doing standup in a club, a format that Nothing, Forever emulates.
  - neuronexmachina 4y ago
    Or mental health: https://gizmodo.com/mental-health-therapy-app-ai-koko-chatgpt-rob-morris-1849965534 https://gizmodo.com/mental-health-therapy-app-ai-koko-chatgp...
- causi 4y ago
  Am I wrong in thinking that getting your feelings hurt by an AI making an offensive joke is the equivalent of thinking you've been doxxed when a random number generator spits out your phone number?
  - version_five 4y ago
    This is a great analogy! $ seq -w 9999999
    - Pyramus 4y ago
      Isn't the purpose of machine learning to learn something, not nothing/randomness?
      - version_five 4y ago
        The point is that whether it's an RNG or language model or just a program printing a list of numbers like I posted, it's just a state machine spitting stuff out, it has no intention. So it's not really productive to get mad at it just because it produced something you don't like. The language model is spitting out random stuff based on the representation it has learned. It doesn't know what it means, nobody is making any kind of value judgement that results in it saying stuff, so why get offended by it? Edit: Because if our adversaries are the products of inanimate and unthinking evolution, we cannot regard the problem in terms of revenge or payback... that would be no different than whipping the ocean for having sunk a ship and drowned its sailors - Stanislaw Lem, The Invincible
        widowlark 4y ago
        I think it's fair to say that the anger people might feel is probably directed at the creators/owners of the AI, rather than the AI itself
        crote 4y ago
        That's like saying a book has no intention. It's just a bunch of ink and paper - it's completely neutral! However, the AI may not have been trained in a neutral way, and the person using it may not be using it in a neutral way. If you create an AI product, you are responsible for its output. People aren't mad at the model itself, they are mad at its creators. Why did they create an AI which ends up insulting people and breaking the ToS of the platform it is on?
- cobertos 4y ago
  Getting banned feels pretty drastic, especially for the amount of time they put into this project to just have it wiped from Twitch. But perhaps it's not a perma ban? And Twitch's rules only seem as arcane as YouTube and Facebook too. Maybe not Twitter.
version_five 4y ago
Drastic is a bit strong. A cartoon character on an infinite streaming tv experiment made a bad joke (which appears to be more of a bad meta-joke). The fact that the stakes are so low is presumably why they changed the model. If it was flying an airplane or something that mattered, they probably would have taken more care. The change they made had a completely acceptable risk / consequence tradeoff
- yifanl 4y ago
  Drastic is relative to the expected impact of the application. There isn't much worse that can possibly happen to a bot-generated stream than having the stream taken down by the powers-that-be.
  - timr 4y ago
    This is like saying that having something taken down by a mod on reddit is a "drastic consequence". I mean, sure...the content is removed, but in most cases, the reason for the action is subjective and somewhat arbitrary. To me, this is less a commentary on AI than on the absurd sensitivities of community moderation.
    - madeofpalk 4y ago
      "Website makes moderation decision some people disagree with" is a lot less novel or interesting.
    - yifanl 4y ago
      Again, the drasticness of a consequence is defined by the system you're trying to build for. If you build a bot to automate your streaming so you can have 100% stream uptime and you end up with 0% uptime, that's because your product failed drastically. (Keeping in mind, as arbitrary and subjective the rules of Twitch are, the vast majority of human streamers have never been banned for arbitrary moderation)
    - tinus_hn 4y ago
      For many people Twitch streams are their primary source of income, this kind of ban can have quite a lot of impact. And then besides the ban there is the chilling effect because no one wants to skirt the rules and risk a ban. And that is what allows social justice warriors to dictate what others can make jokes about.
gardenhedge 4y ago
> innocuous engineering change > reasonable engineering decision They swapped out the AI model. That's kind of a big deal.
- cobertos 4y ago
  In my mind, if I look at a drop-down and "model D" is down but "model C" is up and I need the API running NOW, I'm going to change the option to the one that's working. That being said, every company I've been at with a production outage has gone into "fix it and ask questions later" mode. Maybe other companies have more process? But when you put it like you said in your comment, it does feel more weighty
  - gardenhedge 4y ago
    In contrast the (non-AI) companies I've worked with have a rigorous change release process which includes impact assessments, back out plans, getting multiple sign offs including QA testing and post-release testing, etc.
    - cobertos 4y ago
      I mean specifically for a production outage fix. For planned changes, most of the companies I've worked at have similar release processes
hourago 4y ago
> A seemingly innocuous engineering change led to some drastic consequences. If you think that this is "drastic" you should read "Case Study 4: The $440 Million Software Error at Knight Capital". https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-... tldr; At Knight, some new trading software contained a flaw that became apparent only after the software was activated when the New York Stock Exchange (NYSE) opened that day. The errant software sent Knight on a buying spree, snapping up 150 different stocks at a total cost of around $7 billion, all in the first hour of trading.
- KMag 4y ago
  I was developing trading systems at Goldman then. I remember that day. I got a call that evening to come in early the next day to discuss additional preventions and mitigations to implement. Scarier was the day I put out a change request shortly before the end of the day in Asia, went out to dinner and drinks with some colleagues, stopped by my office on the way home, and merged the approved change request, during early morning trading in the US. The next day (in Asia) I woke up to discover Goldman had a roughly 28 million dollar trading loss. I spent a couple of minutes proving to myself that my change couldn't possibly have been the cause, and then realized I would have been woken in the middle of the night by a phone call if there was any possibility that my change had caused the trading loss. Since then, I don't merge code at the end of the day, and try to avoid Fridays. I was already in the habit of not coding if I had consumed any alcohol, but added the habit of not merging if I've consumed any alcohol. There's nothing like a 28 million dollar panic to get your practices in shape.
- cobertos 4y ago
  I meant drastic for their use case. They put all this work into a piece of software specifically for streaming and it gets banned from the streaming platform. Definitely not life altering or a safety issue, but worst case for them, and I can imagine this going similarly wrong in a different piece of software where stakes actually matter
bravetraveler 4y ago
Is changing the source of data really that innocuous? I find it not that surprising that the output changed with the input Especially considering this is the description for the model they left: > higher quality, longer output and better instruction-following and this is what they moved to: > Very capable, but faster and lower cost than Davinci. If nothing else, the adage 'you get what you pay for' works. ref: https://platform.openai.com/docs/models https://platform.openai.com/docs/models
- cobertos 4y ago
  I've only been through a couple production outages, but at those companies, usually we'd get things patched and ask questions later. I could definitely imagine switching to another model that is made to sound like a smaller version of Davinci, at least from how it's been stored in my mind up until now.
  - bravetraveler 4y ago
    I've been there quite a bit, and it's pretty true! It's best to worry about how the ship started sinking after you've saved it :p In that situation I still try to exercise my options a bit... Seeing that this new model is implicitly worse at following directions, knowing that 'production' (this is a Twitch channel) depends on obeying a certain set of rules... switching to it may be ill-advised. I don't want to be seen as overly judgmental - the Twitch channel comment works both ways; derogatory/supportive. I get why one wouldn't really foresee this or even care to. Then I wonder, why not leave it down for a bit? The hype train is fickle but it's not that precious, either.
skim_milk 4y ago
They've been developing this on-the-fly since it gained popularity (and profitability). Basically, they ran fast and broke things, like any good startup. I'm nearly 100% certain that the scripts are generated through prompt engineering, with a random prompt (e.g. tell a joke prompt, talk about a new restaurant prompt) being selected for the scene. From what I can gather they first used the older, cheaper GPT-3 models, only upgrading to davinci-003 when it was profitable. The older GPT-3 models proved fine and didn't generate edgy content for the several months they were up and running. But I think the change that broke the camels back was they added a "2006 Laugh Factory incident with edgy content" prompt and only tested it on the davinci-003 model - the new models having been wiped clean of antisocial training data, while the older smaller models still having contentious content encoded in the model. So, davinci-003 did fine producing "politically aligned" text with the "edgy" prompt because it's "cleaned", but when the openai API for davinci went down the fallback was curie. The older "unclean" curie model combined with an edgy prompt inevitably caused what we saw here.
- ethbr0 4y ago
  My takeaway is that the more we turn to larger and larger models (developed by third parties) that obscure their underlying nuances, the more risky swapping models in a particular domain becomes. Cue MRE/MLOps job listings.

A seemingly innocuous engineering change led to some drastic consequences. Imagine the other more critical production applications who have made the same reasonable engineering decision.

draw_down 4y ago

I dunno about that description. They essentially put a different brain in their guy’s body and he started acting different. One could consider that expected behavior.

bogwog 4y ago
[flagged]
cobertos 4y ago
This sounds logical, though my brain has always stored Curie as just like a Davinci divided by 2. They're right next two each other in the sandbox drop-down, it _feels_ swappable, in an innocuous way, or at least it did to me until this point.

anononaut 4y ago

I challenge your interpretation of a joke generator making a joke as "drastic consequences." Twitch has arcane rules about what can be streamed. As a streamer you have to tip-toe around topics (or jokes) regarding specific protected classes. This is a matter of arbitrary lines in the sand.

Macha 4y ago
For a joke generator, making an offensive joke that gets the generator banned from the primary source it uses to reach users seems pretty high up their on the range of negative possible outcomes. That's the point of the comment, if you were applying AI in more serious scenarios, would this be an AI confidently advising a doctor to overdose a patient, or advising a judge to incarcerate someone based on racist inferences from the set of people currently incarcerated?
- iso1631 4y ago
  > For a joke generator I'm confused, I thought it was supposed to be generating Seinfeld episodes, not jokes?
  - adamrezich 4y ago
    Seinfeld episodes usually open and close with Jerry doing standup in a club, a format that Nothing, Forever emulates.
- neuronexmachina 4y ago
  Or mental health: https://gizmodo.com/mental-health-therapy-app-ai-koko-chatgpt-rob-morris-1849965534 https://gizmodo.com/mental-health-therapy-app-ai-koko-chatgp...
causi 4y ago
Am I wrong in thinking that getting your feelings hurt by an AI making an offensive joke is the equivalent of thinking you've been doxxed when a random number generator spits out your phone number?
- version_five 4y ago
  This is a great analogy! $ seq -w 9999999
  - Pyramus 4y ago
    Isn't the purpose of machine learning to learn something, not nothing/randomness?
    - version_five 4y ago
      The point is that whether it's an RNG or language model or just a program printing a list of numbers like I posted, it's just a state machine spitting stuff out, it has no intention. So it's not really productive to get mad at it just because it produced something you don't like. The language model is spitting out random stuff based on the representation it has learned. It doesn't know what it means, nobody is making any kind of value judgement that results in it saying stuff, so why get offended by it? Edit: Because if our adversaries are the products of inanimate and unthinking evolution, we cannot regard the problem in terms of revenge or payback... that would be no different than whipping the ocean for having sunk a ship and drowned its sailors - Stanislaw Lem, The Invincible
      - widowlark 4y ago
        I think it's fair to say that the anger people might feel is probably directed at the creators/owners of the AI, rather than the AI itself
      - crote 4y ago
        That's like saying a book has no intention. It's just a bunch of ink and paper - it's completely neutral! However, the AI may not have been trained in a neutral way, and the person using it may not be using it in a neutral way. If you create an AI product, you are responsible for its output. People aren't mad at the model itself, they are mad at its creators. Why did they create an AI which ends up insulting people and breaking the ToS of the platform it is on?
cobertos 4y ago
Getting banned feels pretty drastic, especially for the amount of time they put into this project to just have it wiped from Twitch. But perhaps it's not a perma ban? And Twitch's rules only seem as arcane as YouTube and Facebook too. Maybe not Twitter.

version_five 4y ago

Drastic is a bit strong. A cartoon character on an infinite streaming tv experiment made a bad joke (which appears to be more of a bad meta-joke). The fact that the stakes are so low is presumably why they changed the model. If it was flying an airplane or something that mattered, they probably would have taken more care. The change they made had a completely acceptable risk / consequence tradeoff

yifanl 4y ago
Drastic is relative to the expected impact of the application. There isn't much worse that can possibly happen to a bot-generated stream than having the stream taken down by the powers-that-be.
- timr 4y ago
  This is like saying that having something taken down by a mod on reddit is a "drastic consequence". I mean, sure...the content is removed, but in most cases, the reason for the action is subjective and somewhat arbitrary. To me, this is less a commentary on AI than on the absurd sensitivities of community moderation.
  - madeofpalk 4y ago
    "Website makes moderation decision some people disagree with" is a lot less novel or interesting.
  - yifanl 4y ago
    Again, the drasticness of a consequence is defined by the system you're trying to build for. If you build a bot to automate your streaming so you can have 100% stream uptime and you end up with 0% uptime, that's because your product failed drastically. (Keeping in mind, as arbitrary and subjective the rules of Twitch are, the vast majority of human streamers have never been banned for arbitrary moderation)
  - tinus_hn 4y ago
    For many people Twitch streams are their primary source of income, this kind of ban can have quite a lot of impact. And then besides the ban there is the chilling effect because no one wants to skirt the rules and risk a ban. And that is what allows social justice warriors to dictate what others can make jokes about.

gardenhedge 4y ago

> innocuous engineering change > reasonable engineering decision They swapped out the AI model. That's kind of a big deal.

cobertos 4y ago
In my mind, if I look at a drop-down and "model D" is down but "model C" is up and I need the API running NOW, I'm going to change the option to the one that's working. That being said, every company I've been at with a production outage has gone into "fix it and ask questions later" mode. Maybe other companies have more process? But when you put it like you said in your comment, it does feel more weighty
- gardenhedge 4y ago
  In contrast the (non-AI) companies I've worked with have a rigorous change release process which includes impact assessments, back out plans, getting multiple sign offs including QA testing and post-release testing, etc.
  - cobertos 4y ago
    I mean specifically for a production outage fix. For planned changes, most of the companies I've worked at have similar release processes

hourago 4y ago

> A seemingly innocuous engineering change led to some drastic consequences. If you think that this is "drastic" you should read "Case Study 4: The $440 Million Software Error at Knight Capital". https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-... tldr; At Knight, some new trading software contained a flaw that became apparent only after the software was activated when the New York Stock Exchange (NYSE) opened that day. The errant software sent Knight on a buying spree, snapping up 150 different stocks at a total cost of around $7 billion, all in the first hour of trading.

KMag 4y ago
I was developing trading systems at Goldman then. I remember that day. I got a call that evening to come in early the next day to discuss additional preventions and mitigations to implement. Scarier was the day I put out a change request shortly before the end of the day in Asia, went out to dinner and drinks with some colleagues, stopped by my office on the way home, and merged the approved change request, during early morning trading in the US. The next day (in Asia) I woke up to discover Goldman had a roughly 28 million dollar trading loss. I spent a couple of minutes proving to myself that my change couldn't possibly have been the cause, and then realized I would have been woken in the middle of the night by a phone call if there was any possibility that my change had caused the trading loss. Since then, I don't merge code at the end of the day, and try to avoid Fridays. I was already in the habit of not coding if I had consumed any alcohol, but added the habit of not merging if I've consumed any alcohol. There's nothing like a 28 million dollar panic to get your practices in shape.
cobertos 4y ago
I meant drastic for their use case. They put all this work into a piece of software specifically for streaming and it gets banned from the streaming platform. Definitely not life altering or a safety issue, but worst case for them, and I can imagine this going similarly wrong in a different piece of software where stakes actually matter

bravetraveler 4y ago

Is changing the source of data really that innocuous? I find it not that surprising that the output changed with the input Especially considering this is the description for the model they left: > higher quality, longer output and better instruction-following and this is what they moved to: > Very capable, but faster and lower cost than Davinci. If nothing else, the adage 'you get what you pay for' works. ref: https://platform.openai.com/docs/models https://platform.openai.com/docs/models

cobertos 4y ago
I've only been through a couple production outages, but at those companies, usually we'd get things patched and ask questions later. I could definitely imagine switching to another model that is made to sound like a smaller version of Davinci, at least from how it's been stored in my mind up until now.
- bravetraveler 4y ago
  I've been there quite a bit, and it's pretty true! It's best to worry about how the ship started sinking after you've saved it :p In that situation I still try to exercise my options a bit... Seeing that this new model is implicitly worse at following directions, knowing that 'production' (this is a Twitch channel) depends on obeying a certain set of rules... switching to it may be ill-advised. I don't want to be seen as overly judgmental - the Twitch channel comment works both ways; derogatory/supportive. I get why one wouldn't really foresee this or even care to. Then I wonder, why not leave it down for a bit? The hype train is fickle but it's not that precious, either.

skim_milk 4y ago

They've been developing this on-the-fly since it gained popularity (and profitability). Basically, they ran fast and broke things, like any good startup. I'm nearly 100% certain that the scripts are generated through prompt engineering, with a random prompt (e.g. tell a joke prompt, talk about a new restaurant prompt) being selected for the scene. From what I can gather they first used the older, cheaper GPT-3 models, only upgrading to davinci-003 when it was profitable. The older GPT-3 models proved fine and didn't generate edgy content for the several months they were up and running. But I think the change that broke the camels back was they added a "2006 Laugh Factory incident with edgy content" prompt and only tested it on the davinci-003 model - the new models having been wiped clean of antisocial training data, while the older smaller models still having contentious content encoded in the model. So, davinci-003 did fine producing "politically aligned" text with the "edgy" prompt because it's "cleaned", but when the openai API for davinci went down the fallback was curie. The older "unclean" curie model combined with an edgy prompt inevitably caused what we saw here.

ethbr0 4y ago
My takeaway is that the more we turn to larger and larger models (developed by third parties) that obscure their underlying nuances, the more risky swapping models in a particular domain becomes. Cue MRE/MLOps job listings.