6 ms·

I find that people in tech, especially in my own field of data science, treat ethics as a relatively simple problem. If they were building a technical product,

I find that people in tech, especially in my own field of data science, treat ethics as a relatively simple problem. If they were building a technical product, they would at least make an appearance of checking assumptions, and worrying about breadth and depth of stakeholder buy-in, and building tests for unintended consequences. But when it comes to building an ethical "product" to guide their work, it's like all of the best-practices go by the wayside. For example: https://www.wired.com/story/should-data-scientists-adhere-to-a-hippocratic-oath/ https://www.wired.com/story/should-data-scientists-adhere-to... Data for Democracy came up with an ethical code for data scientists and is now talking it up asking people to sign on to it. The thing is basically the product a few months of working groups plus a day-long hackathon, and it's already been put out on the market, so to speak. So it's not surprising that they produced something that could, in many ways, actually run counter to their goals. I firmly believe the ethical code as written is itself unethical: https://towardsdatascience.com/an-ethical-code-cant-be-about-ethics-66acaea6f16f https://towardsdatascience.com/an-ethical-code-cant-be-about... What I've found amazing is that the community that built the D4D ethical code has been entirely unwilling to invite criticism of the code, or even engage with those who question it. The fact that willingness to entertain criticism and consider unintended consequences is actually one of the pillars of their ethical code makes it doubly troubling.

IntronExon 9y ago
I’ve been saying this a lot lately, but software needs a professional associstion (like the AMA) with teeth. Ethics are not easy, and without collective governing power, they simply won’t exist. https://news.ycombinator.com/item?id=16423328 https://news.ycombinator.com/item?id=16423328 https://news.ycombinator.com/item?id=16420659 https://news.ycombinator.com/item?id=16420659
- whataretensors 9y ago
  I disagree. Not in principle, but because I think it won't work. If an organization has teeth in this market it will be turned into a weapon to gate out or remove competition.
  - IntronExon 9y ago
    What makes software so different from every other endeavor in that regard?
    - whataretensors 9y ago
      Scope
      - IntronExon 9y ago
        Medicine encompasses everything from practice to research, humans to non-human animals, procedures and devices to drugs and therapies, and touches nearly every human alive. Engineering is even more diverse and penetrating, and software is merely a subset. It ain’t scope.
    - Bartweiss 9y ago
      If I understand your question: nothing, professional associations become anti-competitive gatekeepers in every endeavor. Roughly 30% of US workers have mandatory occupational licensing before they can do their jobs. Licensed services have roughly 15% higher prices than nonlicensed ones. Most of those workers are not in safety critical jobs - on average an interior designer faces more stringent licensing requirements than an EMT. Unions and professional associations have consistently been the main lobbyists advocating professional licensing as a way to keep wages high (by keeping supply low) and guarantee a role for the organization. Even in medicine, the number of residency slots is controlled in a way that creates a stable, minor shortage of doctors. I'm not against all licensing and oversight, obviously. I'd rather have a cap on residencies than untrained doctors, if those are the only two choices. But the history of 'every other endeavor' looks to me like a lesson in how oversight organizations will cause exactly this problem.
      - IntronExon 9y ago
        There is always a trade off, but each occupation learns the hard way thst it’s worth it. Hopefully software will figure that out in time to regulate itself before the government steps in with agendas and a heavy hand.
- schaunwheeler 9y ago
  A think a governing body solves the wrong problem. Most of the ethical issues that have arisen around data science in particular are actually competency issues. It's not that someone deliberately set out to target already-vulnerable populations. It's that they designed systems without building in checks for unintended consequences. A governing body accomplishes what any legal framework accomplishes: it provides instruction and incentives to limit liability. A legal framework is different than an ethical framework. What we need is an ethical framework. The original Hippocratic Oath did a fairly good job of this by stipulating ways that a doctor could prove his competence. Doctors who adhered to the oath weren't better doctors because they had some kind of internal moral compass or external adjudicating body. They were better doctors because only the doctors who had competency to spare were willing to make the sacrifices that adherence to the oath required. https://hackernoon.com/on-the-difficulty-of-creating-a-data-science-code-of-ethics-9ab44f71af52 https://hackernoon.com/on-the-difficulty-of-creating-a-data-... Ethical problems get solved (as much as that type of problem can ever really be "solved") by individual practitioners refusing to work with other individual practitioners who refuse to adhere to some basic best practices. That creates a network of competent and trustworthy individuals, who are still totally fallible, but who put their own practice up toe constant public scrutiny. Ethical problems are problems of systemic risk. Systemic risk doesn't get solved through organizations.
  - s73v3r_ 9y ago
    The governing body would not just be there to arbitrate, advocate, or negotiate. They would also be there for promoting training and sharing of knowledge.
    - mr_spothawk 9y ago
      i want to stress that i'm not challenging the person, just the text. > The governing body would not just be there to arbitrate, advocate, or negotiate. They would also be there for promoting training and sharing of knowledge. this reads like duct-tape solutions. it's always the case that a governing body grows and grows and gets in the way. "it's not just there to XYZ, it also BLAHBLAHBLAH". a governing body is one possible solution to the problem of people not understanding ethics (or how to figure out what ethics are), that I don't think will solve the larger problem. a governing body will need to be run by practitioners of governance, and enforced by practitioners of enforcement. ethics needs to be practiced everywhere, all the time. and a governing body won't make that happen. people need to be learning about why their activities are "good" or "bad" or "questionable". in my (simplified version) view: we're not lacking incentive to work (money!), we're lacking incentive to not work (people shouting at us for being assholes for doing asshole work [e.g. bombs, baby killing, evil algos]). we need to cultivate behaviors of ethical culture... instead we're cultivating behaviors that promote short-term-ism (buy your way to happiness, bomb your way to safety, build your dream home on loan!, opportunity cost services saving us from having to cleaning up after ourselves, trash removal services that vanish things away to faraway dumps we never think of)
ssivark 9y ago
I think the “problem” is that ethical questions and nuanced thinking generally get in the way of “getting things done”. It seems that the software/startup/disruption community glorifies moving fast, shipping an MVP and then iterating on improving measurable metrics. None of these are conducive to dialectic and thinking carefully about consequences. Heck, the popularity of weak/dynamic typing and REPL-it-till-it-works shows that many programmers today aren’t willing to reason about effects even within their own code... leave alone societal consequences! Nothing wrong with that style of programming as such, but I think it’s a symptom of the ratio of thinking-to-doing that many find comfortable. Software is unique in that it gives us the flexibility to fix bugs after shipping, unlike any other engineering field. Over the last few decades, we seem to have gotten increasingly carried away with emphasis on that feature. Also has to do with the incentive structure encouraged in SV.
- schaunwheeler 9y ago
  I agree with this. There comes a point (much sooner than most people like to recognize) when quality and safety get sacrificed to just getting things done. However, I think that's a manageable risk. Most focus - and tooling - centered on analytic/technical design rather than implementation has the potential to catch problematic implementations before they're shipped. https://towardsdatascience.com/data-is-a-stakeholder-31bfdb650af0 https://towardsdatascience.com/data-is-a-stakeholder-31bfdb6... It's not fool-proof, of course, and I think a change in perspective away from just moving fast and breaking things would be quite healthy for the industry as a whole.
- s73v3r_ 9y ago
  I think the problem is, you can fix technical bugs after shipping. It's much, much harder to fix ethical bugs.
  - schaunwheeler 9y ago
    Exactly. The cost of fixing a bug after the product is out the door is much higher when the bug is ethical rather than technical. Therefore, don't move the product out the door so quickly. Ethical QA takes time.

IntronExon 9y ago

I’ve been saying this a lot lately, but software needs a professional associstion (like the AMA) with teeth. Ethics are not easy, and without collective governing power, they simply won’t exist. https://news.ycombinator.com/item?id=16423328 https://news.ycombinator.com/item?id=16423328 https://news.ycombinator.com/item?id=16420659 https://news.ycombinator.com/item?id=16420659

whataretensors 9y ago
I disagree. Not in principle, but because I think it won't work. If an organization has teeth in this market it will be turned into a weapon to gate out or remove competition.
- IntronExon 9y ago
  What makes software so different from every other endeavor in that regard?
  - whataretensors 9y ago
    Scope
    - IntronExon 9y ago
      Medicine encompasses everything from practice to research, humans to non-human animals, procedures and devices to drugs and therapies, and touches nearly every human alive. Engineering is even more diverse and penetrating, and software is merely a subset. It ain’t scope.
  - Bartweiss 9y ago
    If I understand your question: nothing, professional associations become anti-competitive gatekeepers in every endeavor. Roughly 30% of US workers have mandatory occupational licensing before they can do their jobs. Licensed services have roughly 15% higher prices than nonlicensed ones. Most of those workers are not in safety critical jobs - on average an interior designer faces more stringent licensing requirements than an EMT. Unions and professional associations have consistently been the main lobbyists advocating professional licensing as a way to keep wages high (by keeping supply low) and guarantee a role for the organization. Even in medicine, the number of residency slots is controlled in a way that creates a stable, minor shortage of doctors. I'm not against all licensing and oversight, obviously. I'd rather have a cap on residencies than untrained doctors, if those are the only two choices. But the history of 'every other endeavor' looks to me like a lesson in how oversight organizations will cause exactly this problem.
    - IntronExon 9y ago
      There is always a trade off, but each occupation learns the hard way thst it’s worth it. Hopefully software will figure that out in time to regulate itself before the government steps in with agendas and a heavy hand.
schaunwheeler 9y ago
A think a governing body solves the wrong problem. Most of the ethical issues that have arisen around data science in particular are actually competency issues. It's not that someone deliberately set out to target already-vulnerable populations. It's that they designed systems without building in checks for unintended consequences. A governing body accomplishes what any legal framework accomplishes: it provides instruction and incentives to limit liability. A legal framework is different than an ethical framework. What we need is an ethical framework. The original Hippocratic Oath did a fairly good job of this by stipulating ways that a doctor could prove his competence. Doctors who adhered to the oath weren't better doctors because they had some kind of internal moral compass or external adjudicating body. They were better doctors because only the doctors who had competency to spare were willing to make the sacrifices that adherence to the oath required. https://hackernoon.com/on-the-difficulty-of-creating-a-data-science-code-of-ethics-9ab44f71af52 https://hackernoon.com/on-the-difficulty-of-creating-a-data-... Ethical problems get solved (as much as that type of problem can ever really be "solved") by individual practitioners refusing to work with other individual practitioners who refuse to adhere to some basic best practices. That creates a network of competent and trustworthy individuals, who are still totally fallible, but who put their own practice up toe constant public scrutiny. Ethical problems are problems of systemic risk. Systemic risk doesn't get solved through organizations.
- s73v3r_ 9y ago
  The governing body would not just be there to arbitrate, advocate, or negotiate. They would also be there for promoting training and sharing of knowledge.
  - mr_spothawk 9y ago
    i want to stress that i'm not challenging the person, just the text. > The governing body would not just be there to arbitrate, advocate, or negotiate. They would also be there for promoting training and sharing of knowledge. this reads like duct-tape solutions. it's always the case that a governing body grows and grows and gets in the way. "it's not just there to XYZ, it also BLAHBLAHBLAH". a governing body is one possible solution to the problem of people not understanding ethics (or how to figure out what ethics are), that I don't think will solve the larger problem. a governing body will need to be run by practitioners of governance, and enforced by practitioners of enforcement. ethics needs to be practiced everywhere, all the time. and a governing body won't make that happen. people need to be learning about why their activities are "good" or "bad" or "questionable". in my (simplified version) view: we're not lacking incentive to work (money!), we're lacking incentive to not work (people shouting at us for being assholes for doing asshole work [e.g. bombs, baby killing, evil algos]). we need to cultivate behaviors of ethical culture... instead we're cultivating behaviors that promote short-term-ism (buy your way to happiness, bomb your way to safety, build your dream home on loan!, opportunity cost services saving us from having to cleaning up after ourselves, trash removal services that vanish things away to faraway dumps we never think of)

ssivark 9y ago

I think the “problem” is that ethical questions and nuanced thinking generally get in the way of “getting things done”. It seems that the software/startup/disruption community glorifies moving fast, shipping an MVP and then iterating on improving measurable metrics. None of these are conducive to dialectic and thinking carefully about consequences. Heck, the popularity of weak/dynamic typing and REPL-it-till-it-works shows that many programmers today aren’t willing to reason about effects even within their own code... leave alone societal consequences! Nothing wrong with that style of programming as such, but I think it’s a symptom of the ratio of thinking-to-doing that many find comfortable. Software is unique in that it gives us the flexibility to fix bugs after shipping, unlike any other engineering field. Over the last few decades, we seem to have gotten increasingly carried away with emphasis on that feature. Also has to do with the incentive structure encouraged in SV.

schaunwheeler 9y ago
I agree with this. There comes a point (much sooner than most people like to recognize) when quality and safety get sacrificed to just getting things done. However, I think that's a manageable risk. Most focus - and tooling - centered on analytic/technical design rather than implementation has the potential to catch problematic implementations before they're shipped. https://towardsdatascience.com/data-is-a-stakeholder-31bfdb650af0 https://towardsdatascience.com/data-is-a-stakeholder-31bfdb6... It's not fool-proof, of course, and I think a change in perspective away from just moving fast and breaking things would be quite healthy for the industry as a whole.
s73v3r_ 9y ago
I think the problem is, you can fix technical bugs after shipping. It's much, much harder to fix ethical bugs.
- schaunwheeler 9y ago
  Exactly. The cost of fixing a bug after the product is out the door is much higher when the bug is ethical rather than technical. Therefore, don't move the product out the door so quickly. Ethical QA takes time.