5 ms·

> It’s always some external thing: “the algorithm”, as if that doesn’t just mean “the way we programmed it”. This is true. > Computing systems are do not act

> It’s always some external thing: “the algorithm”, as if that doesn’t just mean “the way we programmed it”. This is true. > Computing systems are do not act on their own. That doesn't mean that computer systems behave the way that we intend them to behave, or even that we really fully understand our own intent! How much of your own code have you formally verified? (https://en.wikipedia.org/wiki/Formal_verification https://en.wikipedia.org/wiki/Formal_verification) How much of your own code even has a precise enough purpose that the spec of what it's supposed to do is shorter than the length of the code? Such that you could even in theory formally verify that the implementation is in some sense "correct"? For that matter.... how much of your code has a precise enough purpose that the spec of what it's supposed to do can be written down in formal language at all? And actually... how much of your code has a precise enough purpose that the spec of what it's supposed to do can be written down in ENGLISH at all? I don't think something like an "extremism filter" can ever be implemented in a bug-free way, because I don't think there's a precise enough definition of what "bug-free" would even mean. The problem of people blaming bad outcomes on "the algorithm" is real, and organizations should take responsibility for misclassifications generated by code that they own and operate. It's unhelpful to pretend like engineers and the organizations they work for have zero agency. However, it's equally unhelpful to pretend like buggy behavior aligns with the intent of the engineer/organization.

stretchcat 6y ago
When software goes wrong, the company that decided to create and operate that software for profit is responsible for whatever negative impact that software might have, regardless of whether these problems were foreseen. (And in the case of automated moderation, the software getting something wrong really should be considered foreseen consequence anyway. Facebook knew, or should have known, that these systems would have false positives.)
- throwawaygh 6y ago
  Yes, "organizations should take responsibility for misclassifications generated by code that they own and operate." But also, "the computer was told to do the buggy thing" is misleading because it suggests mens rea.
  - stretchcat 6y ago
    I don't think mens rea is particularly relevant when it comes to the actions of corporations. It's not at all clear to me what it would mean for a corporation to have intent. We could certainly say a 'corporation intended' to do something if we found an email from the CEO commanding it, that much is trivial. But what if we think of a corporation as a 'slow AI' or 'China brain'? Might the corporation, viewed that way, have intent that transcends the individual thoughts and desires of the constituent employees? The system may be structured in a way that rewards low level employees for doing things the individual executives would never consider acceptable (for instance, 'lying to the executives'), and which the low level employees don't think is a good idea either. With all the humans personally opposed to some business practice, it might still occur due to the structure of the system. If we personify that system, we could perhaps say the system itself has a will of its own, with objectives alien to any of the humans involved. A harsh example far worse than Facebook: the Functionalism-Intentionalism debate (https://en.m.wikipedia.org/wiki/Functionalism–intentionalism_debate https://en.m.wikipedia.org/wiki/Functionalism–intentionalism...) At the end of the day, does it really matter? Whether you subscribe to functionalism or intentionalism in that case, the horrible end result is the same and people need to be held accountable for it.
    - throwawaygh 6y ago
      > I don't think mens rea is particularly relevant when it comes to the actions of corporations. It's not at all clear to me what it would mean for a corporation to have intent. But we do distinguish in regulated engineering disciplines. A contractor that intentionally uses shoddy materials in a bridge or tunnel is treated differently from a contractor who simply fails to implement legally required QC is treated different from a contractor who implemented legally required QC but laid bad concrete due to operator error that wasn't captured by that legal requirement. > At the end of the day, does it really matter? Yes, and the reason it matters is immediate when you try to answer the following questions. > Whether you subscribe to functionalism or intentionalism in that case, the horrible end result is the same and people need to be held accountable for it. 1. Held accountable how? 2. Toward what end? -- Facebook has good engineers, but they're not gods. It is not possible to write a perfect -- or perhaps even passably good -- "political extremism" filter. Shifting blame from "the algorithm" to "the engineers" or "the corporation" completely misses the whole fucking point: there is no spec for "perfectly functional political extremism filter". 0.000% of the people calling for "politically neutral moderation" have any god damn clue how to define the thing that are asking for, even in a natural language, let alone a language precise enough to implement. Just look at the rhetoric used. People blame in passive voice "corporations" and "engineers", while criticizing others for using passive voice to blame "the algorithm". I'm not saying that we don't need a base level of QC and corporate responsibility. I even think Software people should be folded into Professional Engineering with all of the personal responsibility that entails! But we need to be very realistic about the fact that engineers are not Gods who can Solve Politics. Hell, even if you erase inherent political tension this is still an impossible task. Can you write down a filter that's perfectly biased toward liberal speech? No. So, again, Held accountable how? Toward what end? Unless your answer is "purposefully kill all social media including HN", mens rea matters.
TeMPOraL 6y ago
> That doesn't mean that computer systems behave the way that we intend them to behave, or even that we really fully understand our own intent! Sure. Neither do children, nor pets, nor farm animals, nor automated control systems. The common thread connecting these examples is: people who are responsible for managing children/animals/PID controllers bear responsibility for the action of these systems, whatever the actions are. Same reasoning should apply to algorithms: don't deploy them unless you're prepared to be held accountable for what they end up doing.
- deleted 6y ago
  [deleted]

stretchcat 6y ago

When software goes wrong, the company that decided to create and operate that software for profit is responsible for whatever negative impact that software might have, regardless of whether these problems were foreseen. (And in the case of automated moderation, the software getting something wrong really should be considered foreseen consequence anyway. Facebook knew, or should have known, that these systems would have false positives.)

throwawaygh 6y ago
Yes, "organizations should take responsibility for misclassifications generated by code that they own and operate." But also, "the computer was told to do the buggy thing" is misleading because it suggests mens rea.
- stretchcat 6y ago
  I don't think mens rea is particularly relevant when it comes to the actions of corporations. It's not at all clear to me what it would mean for a corporation to have intent. We could certainly say a 'corporation intended' to do something if we found an email from the CEO commanding it, that much is trivial. But what if we think of a corporation as a 'slow AI' or 'China brain'? Might the corporation, viewed that way, have intent that transcends the individual thoughts and desires of the constituent employees? The system may be structured in a way that rewards low level employees for doing things the individual executives would never consider acceptable (for instance, 'lying to the executives'), and which the low level employees don't think is a good idea either. With all the humans personally opposed to some business practice, it might still occur due to the structure of the system. If we personify that system, we could perhaps say the system itself has a will of its own, with objectives alien to any of the humans involved. A harsh example far worse than Facebook: the Functionalism-Intentionalism debate (https://en.m.wikipedia.org/wiki/Functionalism–intentionalism_debate https://en.m.wikipedia.org/wiki/Functionalism–intentionalism...) At the end of the day, does it really matter? Whether you subscribe to functionalism or intentionalism in that case, the horrible end result is the same and people need to be held accountable for it.
  - throwawaygh 6y ago
    > I don't think mens rea is particularly relevant when it comes to the actions of corporations. It's not at all clear to me what it would mean for a corporation to have intent. But we do distinguish in regulated engineering disciplines. A contractor that intentionally uses shoddy materials in a bridge or tunnel is treated differently from a contractor who simply fails to implement legally required QC is treated different from a contractor who implemented legally required QC but laid bad concrete due to operator error that wasn't captured by that legal requirement. > At the end of the day, does it really matter? Yes, and the reason it matters is immediate when you try to answer the following questions. > Whether you subscribe to functionalism or intentionalism in that case, the horrible end result is the same and people need to be held accountable for it. 1. Held accountable how? 2. Toward what end? -- Facebook has good engineers, but they're not gods. It is not possible to write a perfect -- or perhaps even passably good -- "political extremism" filter. Shifting blame from "the algorithm" to "the engineers" or "the corporation" completely misses the whole fucking point: there is no spec for "perfectly functional political extremism filter". 0.000% of the people calling for "politically neutral moderation" have any god damn clue how to define the thing that are asking for, even in a natural language, let alone a language precise enough to implement. Just look at the rhetoric used. People blame in passive voice "corporations" and "engineers", while criticizing others for using passive voice to blame "the algorithm". I'm not saying that we don't need a base level of QC and corporate responsibility. I even think Software people should be folded into Professional Engineering with all of the personal responsibility that entails! But we need to be very realistic about the fact that engineers are not Gods who can Solve Politics. Hell, even if you erase inherent political tension this is still an impossible task. Can you write down a filter that's perfectly biased toward liberal speech? No. So, again, Held accountable how? Toward what end? Unless your answer is "purposefully kill all social media including HN", mens rea matters.

TeMPOraL 6y ago

> That doesn't mean that computer systems behave the way that we intend them to behave, or even that we really fully understand our own intent! Sure. Neither do children, nor pets, nor farm animals, nor automated control systems. The common thread connecting these examples is: people who are responsible for managing children/animals/PID controllers bear responsibility for the action of these systems, whatever the actions are. Same reasoning should apply to algorithms: don't deploy them unless you're prepared to be held accountable for what they end up doing.

deleted 6y ago
[deleted]