3 ms·

A tool designed to find people guilty is biased to find people guilty. As far as I know it is fairly easy to take a generic dna sequencer meant for healtcare d

by text70 6y ago

A tool designed to find people guilty is biased to find people guilty. As far as I know it is fairly easy to take a generic dna sequencer meant for healtcare diagnostics, and repurpose it for STR analysis. The only major difference between the healthcare versions and the forensic versions is the software i/o.

sdflhasjd 6y ago
> A tool designed to find people guilty is biased to find people guilty. I don't see those particular issues make it biased, just inaccurate - it could go either way.
- radu_floricica 6y ago
  And yet somehow whenever you take a closer look at mislabeled product prices, the average is always in favor of the store. And that's far from the only industry. Complex tools are the product of many thousands of individual decisions taken by humans, humans aware of who's the paying client.
  - sdflhasjd 6y ago
    I still believe Hanlons razor applies. I've seen products that have serious performance affecting bugs caused by similar mistakes.
    - ncallaway 6y ago
      I still think—even when applying Hanlon's razor—there's an imbalance in incentives that leads to a weight in favor of the interests of the party paying for the test. Take the store pricing example. Suppose the store's pricing & labeling process produce an equal number of bugs at checkout in favor of the store and in opposition to the store. The store is heavily incentivized to detect the errors that are opposed to them. They are much less likely to detect the errors in their favor. Consider the manager that looks at the cash at the end of the day and notices they are $500 short. They likely dig hard to find the root cause of the issue, detect the pricing disparity and correct it. Now consider the manager that is $500 over at the end of the day. They are much more likely to say: "that's weird", shrug their shoulders and move on. The same applies to forensic tools. Even if they originally produced bugs in both directions, their own internal QA and the market of police officers are likely to work hard to detect bugs that make them less likely to allow them to make an arrest. The net result is that the tools end up with a bias in one direction, even if the original developers made an equal number of mistakes in both directions.
      - jfengel 6y ago
        Most store managers get as grumpy about overages as undercounts. They mean that some customer got shortchanged. For $500, it probably means a lot of customers got shortchanged, or something even worse is going on. That makes customers grumpy, and it affects your future. There are plenty of lazy managers who would sweep it under the rug once. But if it happens more than once, it can become their job on the line. They start looking for who's counting wrong. And if they can't figure that out, they get really worried. I have no idea about police officers and prosecutors. But store managers care about accuracy of counts, not just profits.
  - boyesm 6y ago
    > And yet somehow whenever you take a closer look at mislabeled product prices, the average is always in favor of the store. What is this based on?
    - sdenton4 6y ago
      I had a 'fun' experience along these lines with health insurance and medical bills a couple years ago. I can confirm that in our case at least, /every/ error we found was not in our favor, and took usually about an hour on the phone to get fixed. The somewhat-less-malicious interpretation is that the companies have a strong incentive to detect + fix errors that cost them money. Meanwhile, consumers are a) non-centralized, uncoordinated, and often unaware of errors, and b) have no way to fix systemic issues that impact them. And the companies therefore have no /real/ incentive to fix systemic problems. It is literally more profitable to fix the bills of the few people who complain, as they still make money on the remainder who don't notice the errors in the first place. (on edit; exactly what the other comment one subthread over said. :P )
    - ClumsyPilot 6y ago
      decades of data?
  - jstanley 6y ago
    > And yet somehow whenever you take a closer look at mislabeled product prices, the average is always in favor of the store. This could just as easily be selection bias: the errors in favour of the customer are less likely to get reported by customers.
- bgirard 6y ago
  When running an experiment and following poor practices (i.e. p-hacking), results that fit the hypothesis will be accepted more readily and negative results will be debugged or re-ran more often. i.e. The initial error may be randomly distributed. But the follow-up on the error will have a lot of bias.

by text70 6y ago

sdflhasjd 6y ago

> A tool designed to find people guilty is biased to find people guilty. I don't see those particular issues make it biased, just inaccurate - it could go either way.

radu_floricica 6y ago
And yet somehow whenever you take a closer look at mislabeled product prices, the average is always in favor of the store. And that's far from the only industry. Complex tools are the product of many thousands of individual decisions taken by humans, humans aware of who's the paying client.
- sdflhasjd 6y ago
  I still believe Hanlons razor applies. I've seen products that have serious performance affecting bugs caused by similar mistakes.
  - ncallaway 6y ago
    I still think—even when applying Hanlon's razor—there's an imbalance in incentives that leads to a weight in favor of the interests of the party paying for the test. Take the store pricing example. Suppose the store's pricing & labeling process produce an equal number of bugs at checkout in favor of the store and in opposition to the store. The store is heavily incentivized to detect the errors that are opposed to them. They are much less likely to detect the errors in their favor. Consider the manager that looks at the cash at the end of the day and notices they are $500 short. They likely dig hard to find the root cause of the issue, detect the pricing disparity and correct it. Now consider the manager that is $500 over at the end of the day. They are much more likely to say: "that's weird", shrug their shoulders and move on. The same applies to forensic tools. Even if they originally produced bugs in both directions, their own internal QA and the market of police officers are likely to work hard to detect bugs that make them less likely to allow them to make an arrest. The net result is that the tools end up with a bias in one direction, even if the original developers made an equal number of mistakes in both directions.
    - jfengel 6y ago
      Most store managers get as grumpy about overages as undercounts. They mean that some customer got shortchanged. For $500, it probably means a lot of customers got shortchanged, or something even worse is going on. That makes customers grumpy, and it affects your future. There are plenty of lazy managers who would sweep it under the rug once. But if it happens more than once, it can become their job on the line. They start looking for who's counting wrong. And if they can't figure that out, they get really worried. I have no idea about police officers and prosecutors. But store managers care about accuracy of counts, not just profits.
- boyesm 6y ago
  > And yet somehow whenever you take a closer look at mislabeled product prices, the average is always in favor of the store. What is this based on?
  - sdenton4 6y ago
    I had a 'fun' experience along these lines with health insurance and medical bills a couple years ago. I can confirm that in our case at least, /every/ error we found was not in our favor, and took usually about an hour on the phone to get fixed. The somewhat-less-malicious interpretation is that the companies have a strong incentive to detect + fix errors that cost them money. Meanwhile, consumers are a) non-centralized, uncoordinated, and often unaware of errors, and b) have no way to fix systemic issues that impact them. And the companies therefore have no /real/ incentive to fix systemic problems. It is literally more profitable to fix the bills of the few people who complain, as they still make money on the remainder who don't notice the errors in the first place. (on edit; exactly what the other comment one subthread over said. :P )
  - ClumsyPilot 6y ago
    decades of data?
- jstanley 6y ago
  > And yet somehow whenever you take a closer look at mislabeled product prices, the average is always in favor of the store. This could just as easily be selection bias: the errors in favour of the customer are less likely to get reported by customers.
bgirard 6y ago
When running an experiment and following poor practices (i.e. p-hacking), results that fit the hypothesis will be accepted more readily and negative results will be debugged or re-ran more often. i.e. The initial error may be randomly distributed. But the follow-up on the error will have a lot of bias.