3 ms·

When software goes wrong, the company that decided to create and operate that software for profit is responsible for whatever negative impact that software migh

When software goes wrong, the company that decided to create and operate that software for profit is responsible for whatever negative impact that software might have, regardless of whether these problems were foreseen. (And in the case of automated moderation, the software getting something wrong really should be considered foreseen consequence anyway. Facebook knew, or should have known, that these systems would have false positives.)

throwawaygh 6y ago
Yes, "organizations should take responsibility for misclassifications generated by code that they own and operate." But also, "the computer was told to do the buggy thing" is misleading because it suggests mens rea.
- stretchcat 6y ago
  I don't think mens rea is particularly relevant when it comes to the actions of corporations. It's not at all clear to me what it would mean for a corporation to have intent. We could certainly say a 'corporation intended' to do something if we found an email from the CEO commanding it, that much is trivial. But what if we think of a corporation as a 'slow AI' or 'China brain'? Might the corporation, viewed that way, have intent that transcends the individual thoughts and desires of the constituent employees? The system may be structured in a way that rewards low level employees for doing things the individual executives would never consider acceptable (for instance, 'lying to the executives'), and which the low level employees don't think is a good idea either. With all the humans personally opposed to some business practice, it might still occur due to the structure of the system. If we personify that system, we could perhaps say the system itself has a will of its own, with objectives alien to any of the humans involved. A harsh example far worse than Facebook: the Functionalism-Intentionalism debate (https://en.m.wikipedia.org/wiki/Functionalism–intentionalism_debate https://en.m.wikipedia.org/wiki/Functionalism–intentionalism...) At the end of the day, does it really matter? Whether you subscribe to functionalism or intentionalism in that case, the horrible end result is the same and people need to be held accountable for it.
  - throwawaygh 6y ago
    > I don't think mens rea is particularly relevant when it comes to the actions of corporations. It's not at all clear to me what it would mean for a corporation to have intent. But we do distinguish in regulated engineering disciplines. A contractor that intentionally uses shoddy materials in a bridge or tunnel is treated differently from a contractor who simply fails to implement legally required QC is treated different from a contractor who implemented legally required QC but laid bad concrete due to operator error that wasn't captured by that legal requirement. > At the end of the day, does it really matter? Yes, and the reason it matters is immediate when you try to answer the following questions. > Whether you subscribe to functionalism or intentionalism in that case, the horrible end result is the same and people need to be held accountable for it. 1. Held accountable how? 2. Toward what end? -- Facebook has good engineers, but they're not gods. It is not possible to write a perfect -- or perhaps even passably good -- "political extremism" filter. Shifting blame from "the algorithm" to "the engineers" or "the corporation" completely misses the whole fucking point: there is no spec for "perfectly functional political extremism filter". 0.000% of the people calling for "politically neutral moderation" have any god damn clue how to define the thing that are asking for, even in a natural language, let alone a language precise enough to implement. Just look at the rhetoric used. People blame in passive voice "corporations" and "engineers", while criticizing others for using passive voice to blame "the algorithm". I'm not saying that we don't need a base level of QC and corporate responsibility. I even think Software people should be folded into Professional Engineering with all of the personal responsibility that entails! But we need to be very realistic about the fact that engineers are not Gods who can Solve Politics. Hell, even if you erase inherent political tension this is still an impossible task. Can you write down a filter that's perfectly biased toward liberal speech? No. So, again, Held accountable how? Toward what end? Unless your answer is "purposefully kill all social media including HN", mens rea matters.
    - stretchcat 6y ago
      > 1. Held accountable how? 2. Toward what end? In the case of the Nazis, the answers are 1. Executed. 2. Justice. For Facebook, I think executions should be off the table, but the second answer is the same. Facebook's engineers aren't gods, I get that. Knowing their engineers aren't gods, Facebook proceeded to use them to create imperfect but profitable systems anyway. Systems they knew or should have known would harm society while enriching their shareholders. This is facebook's crime. If they couldn't create moderation systems that operate well at a massive scale, they never should have operated at that scale in the first place. They couldn't, should have known they couldn't, but tried anyway.

throwawaygh 6y ago

Yes, "organizations should take responsibility for misclassifications generated by code that they own and operate." But also, "the computer was told to do the buggy thing" is misleading because it suggests mens rea.

stretchcat 6y ago
I don't think mens rea is particularly relevant when it comes to the actions of corporations. It's not at all clear to me what it would mean for a corporation to have intent. We could certainly say a 'corporation intended' to do something if we found an email from the CEO commanding it, that much is trivial. But what if we think of a corporation as a 'slow AI' or 'China brain'? Might the corporation, viewed that way, have intent that transcends the individual thoughts and desires of the constituent employees? The system may be structured in a way that rewards low level employees for doing things the individual executives would never consider acceptable (for instance, 'lying to the executives'), and which the low level employees don't think is a good idea either. With all the humans personally opposed to some business practice, it might still occur due to the structure of the system. If we personify that system, we could perhaps say the system itself has a will of its own, with objectives alien to any of the humans involved. A harsh example far worse than Facebook: the Functionalism-Intentionalism debate (https://en.m.wikipedia.org/wiki/Functionalism–intentionalism_debate https://en.m.wikipedia.org/wiki/Functionalism–intentionalism...) At the end of the day, does it really matter? Whether you subscribe to functionalism or intentionalism in that case, the horrible end result is the same and people need to be held accountable for it.
- throwawaygh 6y ago
  > I don't think mens rea is particularly relevant when it comes to the actions of corporations. It's not at all clear to me what it would mean for a corporation to have intent. But we do distinguish in regulated engineering disciplines. A contractor that intentionally uses shoddy materials in a bridge or tunnel is treated differently from a contractor who simply fails to implement legally required QC is treated different from a contractor who implemented legally required QC but laid bad concrete due to operator error that wasn't captured by that legal requirement. > At the end of the day, does it really matter? Yes, and the reason it matters is immediate when you try to answer the following questions. > Whether you subscribe to functionalism or intentionalism in that case, the horrible end result is the same and people need to be held accountable for it. 1. Held accountable how? 2. Toward what end? -- Facebook has good engineers, but they're not gods. It is not possible to write a perfect -- or perhaps even passably good -- "political extremism" filter. Shifting blame from "the algorithm" to "the engineers" or "the corporation" completely misses the whole fucking point: there is no spec for "perfectly functional political extremism filter". 0.000% of the people calling for "politically neutral moderation" have any god damn clue how to define the thing that are asking for, even in a natural language, let alone a language precise enough to implement. Just look at the rhetoric used. People blame in passive voice "corporations" and "engineers", while criticizing others for using passive voice to blame "the algorithm". I'm not saying that we don't need a base level of QC and corporate responsibility. I even think Software people should be folded into Professional Engineering with all of the personal responsibility that entails! But we need to be very realistic about the fact that engineers are not Gods who can Solve Politics. Hell, even if you erase inherent political tension this is still an impossible task. Can you write down a filter that's perfectly biased toward liberal speech? No. So, again, Held accountable how? Toward what end? Unless your answer is "purposefully kill all social media including HN", mens rea matters.
  - stretchcat 6y ago
    > 1. Held accountable how? 2. Toward what end? In the case of the Nazis, the answers are 1. Executed. 2. Justice. For Facebook, I think executions should be off the table, but the second answer is the same. Facebook's engineers aren't gods, I get that. Knowing their engineers aren't gods, Facebook proceeded to use them to create imperfect but profitable systems anyway. Systems they knew or should have known would harm society while enriching their shareholders. This is facebook's crime. If they couldn't create moderation systems that operate well at a massive scale, they never should have operated at that scale in the first place. They couldn't, should have known they couldn't, but tried anyway.