3 ms·

This behavior is by design

33 points by msephton 2y ago

rubatuga 2y ago
Agreed with everything until the last bit. Runaway AI models are a possibility I still consider
- msephton 2y ago
  (author) I wondered if I should even add the last section. My thinking was that if I didn't the discussion would centre on its omission, and if I did the discussion would centre on its inclusion. Catch-22.
- aa-jv 2y ago
  > Runaway AI models The greatest danger is that humans will forget how to verify that we aren't an AI. If the AI is sentinent, and somehow attempts to prevent us from talking to each other, unless we are talking through it, then we get into trouble. Since AI is nothing without social media, one could say that these models would be meme powered, leading humans (hopefully) to recognize the cliché and pastiche of the entity, and turn it off - probably by simply ceasing to talk to it, in as many ways as humanly possible. The danger is that we won't recognize the need to do this, and the whole world turns into a massive computer. Not just every single human, but every single thing involved in human life...
llamaimperative 2y ago
The world is absolutely chock full of systems composed entirely of perfectly sensible, well-considered decisions that nonetheless yield unexpected and undesirable effects.
- msephton 2y ago
  (author) I do mention this briefly in the article. Did you think I should have written more on this? I can't tell by your comment.
  - llamaimperative 2y ago
    It seems you're mostly referring to positive-outcome errors, which is not really what I'm referring to (i.e. emergent behaviors, and especially maladaptive emergent behaviors). For what it's worth I enjoyed the article and am mostly taking issue with the implication of the last section: > The fundamental truth remains: behind every line of code, every pixel, and every feature, there are human decisions shaping our experiences. This is true, but does not mean that those human decisions intended to shape our experiences actually ended up shaping them in the intended way, especially in aggregate.
    - msephton 2y ago
      Right, but they were still shaped by human decisions.
      - llamaimperative 2y ago
        It reads to me like you're saying, "fret not, even mostly inscrutable things were shaped by human decisions." Is that right? The more correct statement is: "You might need to fret even if inscrutable things were shaped by human decisions, because outcomes frequently end up unrelated or totally contrary to the human intent behind those decisions even in total absence of malice or even mistakes."
        msephton 2y ago
        I think we're talking about your personal take after reading of the post? Maybe. I specifically mention bugs and glitches, unintended consequences, happy accidents (unintended things that become intended by virtue of them being kept rather than discarded), unequal weight given to decisions, and more. Of course I don't use your exact wording, but it's all there. I try to keep it neutral, rather positive vs negative outcomes, just "decisions".
        spacebacon 2y ago
        I have a feel for the tone llamaimperative is referencing. The choice to reflect from a more critical point of view has inspired further useful dialog in interpreting your contents for me personally. Good work, do better may be the deeper message here. In a nice way.
- kevincox 2y ago
  Reminds me of a bug that I field for my podcatcher. It has an automatic playlist feature where it automatically queues up next episodes based on some simple rules. One of the rules is "alternate podcasts" to ensure that you rotate through podcasts rather than listening ot just one because it happens to be the oldest or similar. I noticed that I was always listening to the same two podcasts despite having this feature enabled. It turns out that whenever it downloaded a new episode it would regenerate the playlist and forget about where it was in the rotation. So when I finished and episode it would download the next one and reorder the queue, putting the other podcast next. I filed that to the developer and they said "working as intended", when you change the set of downloaded podcasts the playlist is regenerated. And said they wouldn't fix it. But this behaviour is clearly not what it should be, it defeats most of the point of the "alternate podcasts" flag (It only works correctly if you only ever have two eligible podcasts or disable auto-download)
  - llamaimperative 2y ago
    Yep! That's an excellent example of how this can happen even with a very small system with very few people/motivations/constraints interacting.
  - Nannooskeeska 2y ago
    I had literally this exact same scenario happen but with Podcast Addict. It's still my favorite podcast app by far but the auto re-sorting threw me off for a while until I got used to it.
    - kevincox 2y ago
      This was Podcast Addict.
ibash 2y ago
I think “this behavior is by design” is often an excuse. Building quality software is hard because it’s the mistakes and oversights that get noticed. It’s much easier for someone to claim it’s “by design” than to try and fix it.
- msephton 2y ago
  (author) me too, for a while. I always thought it was an excuse back in the 1990s when it was the response to most problems with Windows. But in the years since I have grown to appreciate that whilst it can be used to explain away a bug, it can also justify something that has had a lot of thought put behind it that you might not agree with. The use of "This behavior is by design" as the title of my blog post was intentional, so that it could be interpreted as either positive or negative by the reader.
prometheus76 2y ago
Have you ever seen Sorcerer's Apprentice from Fantasia? I haven't seen the movie, and the original poem it's based on is in French, so I only have seen it in the Disney form. That being said, it's the perfect analogy for what is a deliberate decision by the sorcerer's apprentice to enchant a broom and get it to start bringing water in from the well outside into the cistern in the room. The process starts, and everything is going well, but it's taking too long. So he finds a way to make two brooms. Then ten...etc. Things quickly get out of hand and he can't find how to stop the brooms, and the whole room gets filled with water (chaos). At the last minute, the actual sorcerer appears and sets everything aright, and chastises the apprentice for his foolhardy curiosity in meddling with things he doesn't understand. We could use some of that humility with regard to interfering with complex systems. Most of the effects of perturbations of a complex or chaotic system are unpredictable, and great care should be exercised before making a change, and even with great care, most of the effects will be surprising and unpredictable. So yes, on a superficial level, all decisions in software are deliberate, but the way those decisions interact, and the outcomes of those decisions are unpredictable and largely unknown when the decisions are made. Also, and you mentioned this in the article, many of the decisions are not made deliberately, after much reflection, but rather, on the fly and in the moment. To hand-wave that all away by saying "but a human made it" seems pretty disingenuous, or at least extremely naive.
- msephton 2y ago
  (author) I disagree, as you might expect. :) I'm an independent developer who makes video games, one of which - YOYOZO - was voted a GOTY in 2023. I can explain every single line of that source code and every parameter, variable and constant. I could also explain the physics system I created, my graphics rendering, replay system, scoring calculations, and so on. Nothing is in my code by chance and everything was written by me. I can appreciate the that some people find this difficult to believe, even more so if that type of from-scratch solo-development seems alien. Mine is a bit of a special case because I code games from a blank file without using engines like Godot, Unreal, Unity, Game Maker. Could every game developer explain every part of the game they're making? Of course not. But if one person can claim it then we know it can be true. I agree with you that some systems can be so complex that it would take a huge effort to understand or predict all possible outcomes, like breaking encryption. But that's not all systems. So I think it's not so black and white. It can possible and impossible, easy and difficult, obviously human and done by design or less obviously human with the intent shrouded in complexity. It doesn't have to be one or the other, it can be both.
Avisan 2y ago
It's easy to forget that even with the rise of AI and automation, human decisions are still at the core of software design. The reminder to consider the intent behind design choices is valuable. It helps us appreciate the complexities and nuances that go into creating software

33 points by msephton 2y ago

rubatuga 2y ago

Agreed with everything until the last bit. Runaway AI models are a possibility I still consider

msephton 2y ago
(author) I wondered if I should even add the last section. My thinking was that if I didn't the discussion would centre on its omission, and if I did the discussion would centre on its inclusion. Catch-22.
aa-jv 2y ago
> Runaway AI models The greatest danger is that humans will forget how to verify that we aren't an AI. If the AI is sentinent, and somehow attempts to prevent us from talking to each other, unless we are talking through it, then we get into trouble. Since AI is nothing without social media, one could say that these models would be meme powered, leading humans (hopefully) to recognize the cliché and pastiche of the entity, and turn it off - probably by simply ceasing to talk to it, in as many ways as humanly possible. The danger is that we won't recognize the need to do this, and the whole world turns into a massive computer. Not just every single human, but every single thing involved in human life...

llamaimperative 2y ago

The world is absolutely chock full of systems composed entirely of perfectly sensible, well-considered decisions that nonetheless yield unexpected and undesirable effects.

msephton 2y ago
(author) I do mention this briefly in the article. Did you think I should have written more on this? I can't tell by your comment.
- llamaimperative 2y ago
  It seems you're mostly referring to positive-outcome errors, which is not really what I'm referring to (i.e. emergent behaviors, and especially maladaptive emergent behaviors). For what it's worth I enjoyed the article and am mostly taking issue with the implication of the last section: > The fundamental truth remains: behind every line of code, every pixel, and every feature, there are human decisions shaping our experiences. This is true, but does not mean that those human decisions intended to shape our experiences actually ended up shaping them in the intended way, especially in aggregate.
  - msephton 2y ago
    Right, but they were still shaped by human decisions.
    - llamaimperative 2y ago
      It reads to me like you're saying, "fret not, even mostly inscrutable things were shaped by human decisions." Is that right? The more correct statement is: "You might need to fret even if inscrutable things were shaped by human decisions, because outcomes frequently end up unrelated or totally contrary to the human intent behind those decisions even in total absence of malice or even mistakes."
      - msephton 2y ago
        I think we're talking about your personal take after reading of the post? Maybe. I specifically mention bugs and glitches, unintended consequences, happy accidents (unintended things that become intended by virtue of them being kept rather than discarded), unequal weight given to decisions, and more. Of course I don't use your exact wording, but it's all there. I try to keep it neutral, rather positive vs negative outcomes, just "decisions".
        spacebacon 2y ago
        I have a feel for the tone llamaimperative is referencing. The choice to reflect from a more critical point of view has inspired further useful dialog in interpreting your contents for me personally. Good work, do better may be the deeper message here. In a nice way.
kevincox 2y ago
Reminds me of a bug that I field for my podcatcher. It has an automatic playlist feature where it automatically queues up next episodes based on some simple rules. One of the rules is "alternate podcasts" to ensure that you rotate through podcasts rather than listening ot just one because it happens to be the oldest or similar. I noticed that I was always listening to the same two podcasts despite having this feature enabled. It turns out that whenever it downloaded a new episode it would regenerate the playlist and forget about where it was in the rotation. So when I finished and episode it would download the next one and reorder the queue, putting the other podcast next. I filed that to the developer and they said "working as intended", when you change the set of downloaded podcasts the playlist is regenerated. And said they wouldn't fix it. But this behaviour is clearly not what it should be, it defeats most of the point of the "alternate podcasts" flag (It only works correctly if you only ever have two eligible podcasts or disable auto-download)
- llamaimperative 2y ago
  Yep! That's an excellent example of how this can happen even with a very small system with very few people/motivations/constraints interacting.
- Nannooskeeska 2y ago
  I had literally this exact same scenario happen but with Podcast Addict. It's still my favorite podcast app by far but the auto re-sorting threw me off for a while until I got used to it.
  - kevincox 2y ago
    This was Podcast Addict.

ibash 2y ago

I think “this behavior is by design” is often an excuse. Building quality software is hard because it’s the mistakes and oversights that get noticed. It’s much easier for someone to claim it’s “by design” than to try and fix it.

msephton 2y ago
(author) me too, for a while. I always thought it was an excuse back in the 1990s when it was the response to most problems with Windows. But in the years since I have grown to appreciate that whilst it can be used to explain away a bug, it can also justify something that has had a lot of thought put behind it that you might not agree with. The use of "This behavior is by design" as the title of my blog post was intentional, so that it could be interpreted as either positive or negative by the reader.

prometheus76 2y ago

Have you ever seen Sorcerer's Apprentice from Fantasia? I haven't seen the movie, and the original poem it's based on is in French, so I only have seen it in the Disney form. That being said, it's the perfect analogy for what is a deliberate decision by the sorcerer's apprentice to enchant a broom and get it to start bringing water in from the well outside into the cistern in the room. The process starts, and everything is going well, but it's taking too long. So he finds a way to make two brooms. Then ten...etc. Things quickly get out of hand and he can't find how to stop the brooms, and the whole room gets filled with water (chaos). At the last minute, the actual sorcerer appears and sets everything aright, and chastises the apprentice for his foolhardy curiosity in meddling with things he doesn't understand. We could use some of that humility with regard to interfering with complex systems. Most of the effects of perturbations of a complex or chaotic system are unpredictable, and great care should be exercised before making a change, and even with great care, most of the effects will be surprising and unpredictable. So yes, on a superficial level, all decisions in software are deliberate, but the way those decisions interact, and the outcomes of those decisions are unpredictable and largely unknown when the decisions are made. Also, and you mentioned this in the article, many of the decisions are not made deliberately, after much reflection, but rather, on the fly and in the moment. To hand-wave that all away by saying "but a human made it" seems pretty disingenuous, or at least extremely naive.

msephton 2y ago
(author) I disagree, as you might expect. :) I'm an independent developer who makes video games, one of which - YOYOZO - was voted a GOTY in 2023. I can explain every single line of that source code and every parameter, variable and constant. I could also explain the physics system I created, my graphics rendering, replay system, scoring calculations, and so on. Nothing is in my code by chance and everything was written by me. I can appreciate the that some people find this difficult to believe, even more so if that type of from-scratch solo-development seems alien. Mine is a bit of a special case because I code games from a blank file without using engines like Godot, Unreal, Unity, Game Maker. Could every game developer explain every part of the game they're making? Of course not. But if one person can claim it then we know it can be true. I agree with you that some systems can be so complex that it would take a huge effort to understand or predict all possible outcomes, like breaking encryption. But that's not all systems. So I think it's not so black and white. It can possible and impossible, easy and difficult, obviously human and done by design or less obviously human with the intent shrouded in complexity. It doesn't have to be one or the other, it can be both.

Avisan 2y ago

It's easy to forget that even with the rise of AI and automation, human decisions are still at the core of software design. The reminder to consider the intent behind design choices is valuable. It helps us appreciate the complexities and nuances that go into creating software