3 ms·

i think we can all agree that black boxes are not desirable. But let's not jump the gun here, and assume ill-intent.... to explain something that can be attrib

i think we can all agree that black boxes are not desirable. But let's not jump the gun here, and assume ill-intent.... to explain something that can be attributed to negligence, or simple human error.

anamexis 6y ago
Why shouldn't we assume ill-intent? The precedent and the incentives are there.
kspacewalk2 6y ago
I agree that we shouldn't assume ill intent, but we should recognize it is a very real possibility. Ill intent comes in when you realize the simple human error. Alone, in your office, poring over code or reviewing past files. Then you weigh the impact on your career and reputation of honestly fessing up, and put it off for a while, then a while longer.
IgorPartola 6y ago
When your negligence results in people wrongfully spending decades in prison I would argue that it ought to be criminal negligence. If your software does not produce results that are accurate, precise, and have confidence intervals included, how OK are you with people going to prison based on them?
- caconym_ 6y ago
  Yeah, there need to be consequences for this. The alternative is empowering people and corporations to destroy lives with impunity in pursuit of their own goals, as long as they can later claim it wasn’t intentional. What really pisses me off is how we fail again and again to ask the question of whether these entities should have been doing what they were doing in the first place, if it even can be done safely. Facebook, Equifax, police surveillance and misconduct, totally unaccountable “forensics” techniques—in all these cases where incalculable damage is done in totally predictable ways, we only address it reactively (if we address it at all) and completely fail to fix or even really consider the root cause(s). It’s like we have this huge blind spot where we take as a given that established entities doing thing(s) just have a right to keep doing them, regardless of any other factors. In this case, forensics “experts” (essentially witch doctors) and fly-by-night black-box “labs” with unproven methodologies an undergrad intern might have developed in a week or two are institutionalized in our courts. This has happened because there is zero oversight and zero accountability for their claims, and lives are destroyed as a result. It is insane not to pursue criminal charges for these indiscriminate arsonists of justice—if they can’t do what they claim to do fairly and accurately, they should not do it. They don’t have a right to “try” just because it will make them money, regardless of the consequences.
  - tsimionescu 6y ago
    A lot of people respond to this as if the fact that it would be nice if forensics worked justifies treating them as if they do. 'What do you want them to do, stop looking for fingerprints and DNA evidence?' Yes! Just like I don't want them to use 'lie detectors' or hire psychics.
    - caconym_ 6y ago
      Exactly! It’s pure wishful thinking, and in a lot of cases those wishes are (at best) frighteningly amoral in context. It’s like a hack to get around the whole “reasonable doubt” thing: just sequester all that pesky doubt in a black box so that people can’t see it, and if anyone asks just play dumb.
- crazygringo 6y ago
  The quality standards need to be set by the courts or by leglislation. No software is perfect. It's unreasonable for any bug to be considered criminal negligence, or else the entire software development profession would be in prison. Instead, software that can result in life or death or prison scenarios needs to have tested, documented, verified quality controls, just like we do for vehicles. But that means courts or the legislature have to decide what those standards are.
- deleted 6y ago
  [deleted]
- drstewart 6y ago
  >I would argue that it ought to be criminal negligence I would like a thorough analysis of everything you've ever learned that led you to this conclusion, and if we find one mistake in your lifetime of learning then this is also criminal negligence on your part.

anamexis 6y ago

Why shouldn't we assume ill-intent? The precedent and the incentives are there.

kspacewalk2 6y ago

I agree that we shouldn't assume ill intent, but we should recognize it is a very real possibility. Ill intent comes in when you realize the simple human error. Alone, in your office, poring over code or reviewing past files. Then you weigh the impact on your career and reputation of honestly fessing up, and put it off for a while, then a while longer.

IgorPartola 6y ago

When your negligence results in people wrongfully spending decades in prison I would argue that it ought to be criminal negligence. If your software does not produce results that are accurate, precise, and have confidence intervals included, how OK are you with people going to prison based on them?

caconym_ 6y ago
Yeah, there need to be consequences for this. The alternative is empowering people and corporations to destroy lives with impunity in pursuit of their own goals, as long as they can later claim it wasn’t intentional. What really pisses me off is how we fail again and again to ask the question of whether these entities should have been doing what they were doing in the first place, if it even can be done safely. Facebook, Equifax, police surveillance and misconduct, totally unaccountable “forensics” techniques—in all these cases where incalculable damage is done in totally predictable ways, we only address it reactively (if we address it at all) and completely fail to fix or even really consider the root cause(s). It’s like we have this huge blind spot where we take as a given that established entities doing thing(s) just have a right to keep doing them, regardless of any other factors. In this case, forensics “experts” (essentially witch doctors) and fly-by-night black-box “labs” with unproven methodologies an undergrad intern might have developed in a week or two are institutionalized in our courts. This has happened because there is zero oversight and zero accountability for their claims, and lives are destroyed as a result. It is insane not to pursue criminal charges for these indiscriminate arsonists of justice—if they can’t do what they claim to do fairly and accurately, they should not do it. They don’t have a right to “try” just because it will make them money, regardless of the consequences.
- tsimionescu 6y ago
  A lot of people respond to this as if the fact that it would be nice if forensics worked justifies treating them as if they do. 'What do you want them to do, stop looking for fingerprints and DNA evidence?' Yes! Just like I don't want them to use 'lie detectors' or hire psychics.
  - caconym_ 6y ago
    Exactly! It’s pure wishful thinking, and in a lot of cases those wishes are (at best) frighteningly amoral in context. It’s like a hack to get around the whole “reasonable doubt” thing: just sequester all that pesky doubt in a black box so that people can’t see it, and if anyone asks just play dumb.
crazygringo 6y ago
The quality standards need to be set by the courts or by leglislation. No software is perfect. It's unreasonable for any bug to be considered criminal negligence, or else the entire software development profession would be in prison. Instead, software that can result in life or death or prison scenarios needs to have tested, documented, verified quality controls, just like we do for vehicles. But that means courts or the legislature have to decide what those standards are.
deleted 6y ago
[deleted]
drstewart 6y ago
>I would argue that it ought to be criminal negligence I would like a thorough analysis of everything you've ever learned that led you to this conclusion, and if we find one mistake in your lifetime of learning then this is also criminal negligence on your part.