11 ms·

It’s always some external thing: “the algorithm”, as if that doesn’t just mean “the way we programmed it”. Computing systems are do not act on their own. That

It’s always some external thing: “the algorithm”, as if that doesn’t just mean “the way we programmed it”. Computing systems are do not act on their own. That the system did this was because it was allowed to, or more accurately, told to.

throwawaygh 6y ago
> It’s always some external thing: “the algorithm”, as if that doesn’t just mean “the way we programmed it”. This is true. > Computing systems are do not act on their own. That doesn't mean that computer systems behave the way that we intend them to behave, or even that we really fully understand our own intent! How much of your own code have you formally verified? (https://en.wikipedia.org/wiki/Formal_verification https://en.wikipedia.org/wiki/Formal_verification) How much of your own code even has a precise enough purpose that the spec of what it's supposed to do is shorter than the length of the code? Such that you could even in theory formally verify that the implementation is in some sense "correct"? For that matter.... how much of your code has a precise enough purpose that the spec of what it's supposed to do can be written down in formal language at all? And actually... how much of your code has a precise enough purpose that the spec of what it's supposed to do can be written down in ENGLISH at all? I don't think something like an "extremism filter" can ever be implemented in a bug-free way, because I don't think there's a precise enough definition of what "bug-free" would even mean. The problem of people blaming bad outcomes on "the algorithm" is real, and organizations should take responsibility for misclassifications generated by code that they own and operate. It's unhelpful to pretend like engineers and the organizations they work for have zero agency. However, it's equally unhelpful to pretend like buggy behavior aligns with the intent of the engineer/organization.
- stretchcat 6y ago
  When software goes wrong, the company that decided to create and operate that software for profit is responsible for whatever negative impact that software might have, regardless of whether these problems were foreseen. (And in the case of automated moderation, the software getting something wrong really should be considered foreseen consequence anyway. Facebook knew, or should have known, that these systems would have false positives.)
  - throwawaygh 6y ago
    Yes, "organizations should take responsibility for misclassifications generated by code that they own and operate." But also, "the computer was told to do the buggy thing" is misleading because it suggests mens rea.
    - stretchcat 6y ago
      I don't think mens rea is particularly relevant when it comes to the actions of corporations. It's not at all clear to me what it would mean for a corporation to have intent. We could certainly say a 'corporation intended' to do something if we found an email from the CEO commanding it, that much is trivial. But what if we think of a corporation as a 'slow AI' or 'China brain'? Might the corporation, viewed that way, have intent that transcends the individual thoughts and desires of the constituent employees? The system may be structured in a way that rewards low level employees for doing things the individual executives would never consider acceptable (for instance, 'lying to the executives'), and which the low level employees don't think is a good idea either. With all the humans personally opposed to some business practice, it might still occur due to the structure of the system. If we personify that system, we could perhaps say the system itself has a will of its own, with objectives alien to any of the humans involved. A harsh example far worse than Facebook: the Functionalism-Intentionalism debate (https://en.m.wikipedia.org/wiki/Functionalism–intentionalism_debate https://en.m.wikipedia.org/wiki/Functionalism–intentionalism...) At the end of the day, does it really matter? Whether you subscribe to functionalism or intentionalism in that case, the horrible end result is the same and people need to be held accountable for it.
      - throwawaygh 6y ago
        > I don't think mens rea is particularly relevant when it comes to the actions of corporations. It's not at all clear to me what it would mean for a corporation to have intent. But we do distinguish in regulated engineering disciplines. A contractor that intentionally uses shoddy materials in a bridge or tunnel is treated differently from a contractor who simply fails to implement legally required QC is treated different from a contractor who implemented legally required QC but laid bad concrete due to operator error that wasn't captured by that legal requirement. > At the end of the day, does it really matter? Yes, and the reason it matters is immediate when you try to answer the following questions. > Whether you subscribe to functionalism or intentionalism in that case, the horrible end result is the same and people need to be held accountable for it. 1. Held accountable how? 2. Toward what end? -- Facebook has good engineers, but they're not gods. It is not possible to write a perfect -- or perhaps even passably good -- "political extremism" filter. Shifting blame from "the algorithm" to "the engineers" or "the corporation" completely misses the whole fucking point: there is no spec for "perfectly functional political extremism filter". 0.000% of the people calling for "politically neutral moderation" have any god damn clue how to define the thing that are asking for, even in a natural language, let alone a language precise enough to implement. Just look at the rhetoric used. People blame in passive voice "corporations" and "engineers", while criticizing others for using passive voice to blame "the algorithm". I'm not saying that we don't need a base level of QC and corporate responsibility. I even think Software people should be folded into Professional Engineering with all of the personal responsibility that entails! But we need to be very realistic about the fact that engineers are not Gods who can Solve Politics. Hell, even if you erase inherent political tension this is still an impossible task. Can you write down a filter that's perfectly biased toward liberal speech? No. So, again, Held accountable how? Toward what end? Unless your answer is "purposefully kill all social media including HN", mens rea matters.
- TeMPOraL 6y ago
  > That doesn't mean that computer systems behave the way that we intend them to behave, or even that we really fully understand our own intent! Sure. Neither do children, nor pets, nor farm animals, nor automated control systems. The common thread connecting these examples is: people who are responsible for managing children/animals/PID controllers bear responsibility for the action of these systems, whatever the actions are. Same reasoning should apply to algorithms: don't deploy them unless you're prepared to be held accountable for what they end up doing.
  - deleted 6y ago
    [deleted]
AurelioB 6y ago
Sure, makers should always be responsible & accountable for how a system behaves, but this seems like a textbook Hanlon's razor example
- HarryHirsch 6y ago
  People generally misunderstand Hanlon's Razor. It's immaterial if someone is evil or just stupid, the only thing that matters is that they should not be making decisions about other people. (Charles Williams' All Hallows' Eve is an extended meditation about just that.)
  - splistud 6y ago
    Well if it's stupid, and not evil, they weren't making a decision, right?
    - Veen 6y ago
      They were making a decision, but it’s not malice because they’re incapable of foreseeing the outcomes of their decisions. In either case, they shouldn’t be in charge of anything important.
      - amanaplanacanal 6y ago
        > they shouldn’t be in charge of anything important And in this case, it appears that they aren’t.
threeseed 6y ago
Often these are machine learning models which are not explicitly programmed and do act on their own. It could easily be a model that takes in features such as "has an admin banned this page" or "has an overwhelming number of users reported this page" and then based on that it identifies problematic keywords that it uses to ban pages. And then it relies on humans to manually interject when the model fails as it sometimes will.
- tpxl 6y ago
  > Often these are machine learning models which are not explicitly programmed and do act on their own. That were programmed to make decision no one understands and to act on their own.
- sudosysgen 6y ago
  Why not put the 1% largest pages up for manual review? I think it there is some merit to the idea that they are sometimes made overly broad to give cover for some actions by blaming them on the automated system and ignoring pleas for reinstation.
- ClumsyPilot 6y ago
  The law and the rest of society does not have to accomodate unpredictable and buggy ML models -in fact its the other way round. We are bot obligated to suffer damage just because FB relies on ML.
  - hakfoo 6y ago
    Somehow, the big players have managed to convince us to suck up the damage. Correctly moderating Facebook, blocking recalled/bogus/questionable goods on Amazon, or keeping Google's search index pruned of scams and misinformation, are fundamentally problems that requires armies of trained humans, well-thought-out and transparent process, and potentially independent or state oversight to keep it all on the rails. It's much cheaper for them to buy racks of servers, and occasionally shrug their shoulders and blame algorithms. I'm amazed that legislatures throughout the world have been willing to accept that behaviour for so long. It's not like the problems can be explained away with "it's an insignificant business with no political or commercial influence", or "it's a 6-month-old startup firm that made beginner mistakes because they couldn't afford to do things properly". Break them all up. Even in a world without Google and Facebook, enterprising nerds will be encoding H.265 via smoke signals to ensure we get our cat videos.

throwawaygh 6y ago

> It’s always some external thing: “the algorithm”, as if that doesn’t just mean “the way we programmed it”. This is true. > Computing systems are do not act on their own. That doesn't mean that computer systems behave the way that we intend them to behave, or even that we really fully understand our own intent! How much of your own code have you formally verified? (https://en.wikipedia.org/wiki/Formal_verification https://en.wikipedia.org/wiki/Formal_verification) How much of your own code even has a precise enough purpose that the spec of what it's supposed to do is shorter than the length of the code? Such that you could even in theory formally verify that the implementation is in some sense "correct"? For that matter.... how much of your code has a precise enough purpose that the spec of what it's supposed to do can be written down in formal language at all? And actually... how much of your code has a precise enough purpose that the spec of what it's supposed to do can be written down in ENGLISH at all? I don't think something like an "extremism filter" can ever be implemented in a bug-free way, because I don't think there's a precise enough definition of what "bug-free" would even mean. The problem of people blaming bad outcomes on "the algorithm" is real, and organizations should take responsibility for misclassifications generated by code that they own and operate. It's unhelpful to pretend like engineers and the organizations they work for have zero agency. However, it's equally unhelpful to pretend like buggy behavior aligns with the intent of the engineer/organization.

stretchcat 6y ago
When software goes wrong, the company that decided to create and operate that software for profit is responsible for whatever negative impact that software might have, regardless of whether these problems were foreseen. (And in the case of automated moderation, the software getting something wrong really should be considered foreseen consequence anyway. Facebook knew, or should have known, that these systems would have false positives.)
- throwawaygh 6y ago
  Yes, "organizations should take responsibility for misclassifications generated by code that they own and operate." But also, "the computer was told to do the buggy thing" is misleading because it suggests mens rea.
  - stretchcat 6y ago
    I don't think mens rea is particularly relevant when it comes to the actions of corporations. It's not at all clear to me what it would mean for a corporation to have intent. We could certainly say a 'corporation intended' to do something if we found an email from the CEO commanding it, that much is trivial. But what if we think of a corporation as a 'slow AI' or 'China brain'? Might the corporation, viewed that way, have intent that transcends the individual thoughts and desires of the constituent employees? The system may be structured in a way that rewards low level employees for doing things the individual executives would never consider acceptable (for instance, 'lying to the executives'), and which the low level employees don't think is a good idea either. With all the humans personally opposed to some business practice, it might still occur due to the structure of the system. If we personify that system, we could perhaps say the system itself has a will of its own, with objectives alien to any of the humans involved. A harsh example far worse than Facebook: the Functionalism-Intentionalism debate (https://en.m.wikipedia.org/wiki/Functionalism–intentionalism_debate https://en.m.wikipedia.org/wiki/Functionalism–intentionalism...) At the end of the day, does it really matter? Whether you subscribe to functionalism or intentionalism in that case, the horrible end result is the same and people need to be held accountable for it.
    - throwawaygh 6y ago
      > I don't think mens rea is particularly relevant when it comes to the actions of corporations. It's not at all clear to me what it would mean for a corporation to have intent. But we do distinguish in regulated engineering disciplines. A contractor that intentionally uses shoddy materials in a bridge or tunnel is treated differently from a contractor who simply fails to implement legally required QC is treated different from a contractor who implemented legally required QC but laid bad concrete due to operator error that wasn't captured by that legal requirement. > At the end of the day, does it really matter? Yes, and the reason it matters is immediate when you try to answer the following questions. > Whether you subscribe to functionalism or intentionalism in that case, the horrible end result is the same and people need to be held accountable for it. 1. Held accountable how? 2. Toward what end? -- Facebook has good engineers, but they're not gods. It is not possible to write a perfect -- or perhaps even passably good -- "political extremism" filter. Shifting blame from "the algorithm" to "the engineers" or "the corporation" completely misses the whole fucking point: there is no spec for "perfectly functional political extremism filter". 0.000% of the people calling for "politically neutral moderation" have any god damn clue how to define the thing that are asking for, even in a natural language, let alone a language precise enough to implement. Just look at the rhetoric used. People blame in passive voice "corporations" and "engineers", while criticizing others for using passive voice to blame "the algorithm". I'm not saying that we don't need a base level of QC and corporate responsibility. I even think Software people should be folded into Professional Engineering with all of the personal responsibility that entails! But we need to be very realistic about the fact that engineers are not Gods who can Solve Politics. Hell, even if you erase inherent political tension this is still an impossible task. Can you write down a filter that's perfectly biased toward liberal speech? No. So, again, Held accountable how? Toward what end? Unless your answer is "purposefully kill all social media including HN", mens rea matters.
TeMPOraL 6y ago
> That doesn't mean that computer systems behave the way that we intend them to behave, or even that we really fully understand our own intent! Sure. Neither do children, nor pets, nor farm animals, nor automated control systems. The common thread connecting these examples is: people who are responsible for managing children/animals/PID controllers bear responsibility for the action of these systems, whatever the actions are. Same reasoning should apply to algorithms: don't deploy them unless you're prepared to be held accountable for what they end up doing.
- deleted 6y ago
  [deleted]

AurelioB 6y ago

Sure, makers should always be responsible & accountable for how a system behaves, but this seems like a textbook Hanlon's razor example

HarryHirsch 6y ago
People generally misunderstand Hanlon's Razor. It's immaterial if someone is evil or just stupid, the only thing that matters is that they should not be making decisions about other people. (Charles Williams' All Hallows' Eve is an extended meditation about just that.)
- splistud 6y ago
  Well if it's stupid, and not evil, they weren't making a decision, right?
  - Veen 6y ago
    They were making a decision, but it’s not malice because they’re incapable of foreseeing the outcomes of their decisions. In either case, they shouldn’t be in charge of anything important.
    - amanaplanacanal 6y ago
      > they shouldn’t be in charge of anything important And in this case, it appears that they aren’t.

threeseed 6y ago

Often these are machine learning models which are not explicitly programmed and do act on their own. It could easily be a model that takes in features such as "has an admin banned this page" or "has an overwhelming number of users reported this page" and then based on that it identifies problematic keywords that it uses to ban pages. And then it relies on humans to manually interject when the model fails as it sometimes will.

tpxl 6y ago
> Often these are machine learning models which are not explicitly programmed and do act on their own. That were programmed to make decision no one understands and to act on their own.
sudosysgen 6y ago
Why not put the 1% largest pages up for manual review? I think it there is some merit to the idea that they are sometimes made overly broad to give cover for some actions by blaming them on the automated system and ignoring pleas for reinstation.
ClumsyPilot 6y ago
The law and the rest of society does not have to accomodate unpredictable and buggy ML models -in fact its the other way round. We are bot obligated to suffer damage just because FB relies on ML.
- hakfoo 6y ago
  Somehow, the big players have managed to convince us to suck up the damage. Correctly moderating Facebook, blocking recalled/bogus/questionable goods on Amazon, or keeping Google's search index pruned of scams and misinformation, are fundamentally problems that requires armies of trained humans, well-thought-out and transparent process, and potentially independent or state oversight to keep it all on the rails. It's much cheaper for them to buy racks of servers, and occasionally shrug their shoulders and blame algorithms. I'm amazed that legislatures throughout the world have been willing to accept that behaviour for so long. It's not like the problems can be explained away with "it's an insignificant business with no political or commercial influence", or "it's a 6-month-old startup firm that made beginner mistakes because they couldn't afford to do things properly". Break them all up. Even in a world without Google and Facebook, enterprising nerds will be encoding H.265 via smoke signals to ensure we get our cat videos.