6 ms·

This is similar to a case a few years back where someone asked to see the source code of a breathalyser that had found them guilty. AFAIR, the breathalyser was

This is similar to a case a few years back where someone asked to see the source code of a breathalyser that had found them guilty. AFAIR, the breathalyser was incorrectly averaging the readings, giving disproportional weight to the first reading. I don't know if it was enough to rule in their favour, but I'm sure it called the data into question Edit: Looks like it was a Draeger breathalyser https://www.schneier.com/blog/archives/2009/05/software_proble.html https://www.schneier.com/blog/archives/2009/05/software_prob...

dathinab 6y ago
I wouldn't be surprised if "incorrectly averaging" and similar are very common software errors. The reasons are manifold, including: - Normalized values need to be averaged differently the absolute values. - Floating point has limited precision, even just correctly summing/multiplying numbers need special care if you care about correctness. Results can, in the worst case, be of by a massive amount. Often you don't need to care about it so it's not uncommon for especially junior programmers to be not so aware about it. I mean in the last 3 years of working as a professional software engineer/developer I didn't need any of this at all, but once I do I know what to look out for.
text70 6y ago
A tool designed to find people guilty is biased to find people guilty. As far as I know it is fairly easy to take a generic dna sequencer meant for healtcare diagnostics, and repurpose it for STR analysis. The only major difference between the healthcare versions and the forensic versions is the software i/o.
- sdflhasjd 6y ago
  > A tool designed to find people guilty is biased to find people guilty. I don't see those particular issues make it biased, just inaccurate - it could go either way.
  - radu_floricica 6y ago
    And yet somehow whenever you take a closer look at mislabeled product prices, the average is always in favor of the store. And that's far from the only industry. Complex tools are the product of many thousands of individual decisions taken by humans, humans aware of who's the paying client.
    - sdflhasjd 6y ago
      I still believe Hanlons razor applies. I've seen products that have serious performance affecting bugs caused by similar mistakes.
      - ncallaway 6y ago
        I still think—even when applying Hanlon's razor—there's an imbalance in incentives that leads to a weight in favor of the interests of the party paying for the test. Take the store pricing example. Suppose the store's pricing & labeling process produce an equal number of bugs at checkout in favor of the store and in opposition to the store. The store is heavily incentivized to detect the errors that are opposed to them. They are much less likely to detect the errors in their favor. Consider the manager that looks at the cash at the end of the day and notices they are $500 short. They likely dig hard to find the root cause of the issue, detect the pricing disparity and correct it. Now consider the manager that is $500 over at the end of the day. They are much more likely to say: "that's weird", shrug their shoulders and move on. The same applies to forensic tools. Even if they originally produced bugs in both directions, their own internal QA and the market of police officers are likely to work hard to detect bugs that make them less likely to allow them to make an arrest. The net result is that the tools end up with a bias in one direction, even if the original developers made an equal number of mistakes in both directions.
        jfengel 6y ago
        Most store managers get as grumpy about overages as undercounts. They mean that some customer got shortchanged. For $500, it probably means a lot of customers got shortchanged, or something even worse is going on. That makes customers grumpy, and it affects your future. There are plenty of lazy managers who would sweep it under the rug once. But if it happens more than once, it can become their job on the line. They start looking for who's counting wrong. And if they can't figure that out, they get really worried. I have no idea about police officers and prosecutors. But store managers care about accuracy of counts, not just profits.
throw2838 6y ago
Draeger requires calibration prior each use for temperature and humidity. It is easy to get it thrown out as evidence.
- lwigo 6y ago
  When the officer shows up and swears under oath that he "calibrated it that morning" it's not so easy. :shrug:
  - jdironman 6y ago
    Temps vary throughout the day.
    - lwigo 6y ago
      A rural small town judge probably doesn't have the best insight into that, just that Corporal Bubba said he calibrated it so it must've been right. I know, this is exceedingly cynical.
      - klyrs 6y ago
        Your "Corporal Bubba" isn't just cynical, it leans heavily upon a harmful stereotype that folks in small towns are uneducated simpletons. Stop the polarization, please
        owl57 6y ago
        Why wouldn't they be? I went to university with people from all over Russia. They don't tend to return to their home towns after getting educated. Does this work differently in America?
        zymhan 6y ago
        > Does this work differently in America? No, not really.
ClumsyPilot 6y ago
This is also similar to how Toyota hilled people with control software that would cause car to accelerate randomly. The software audit team concluded that they could not find the bug, but the code was totally unreadable and terrible. They settled. https://www.nytimes.com/2013/10/26/business/toyota-agrees-to-settlement-in-fatal-acceleration-crash.html#:~:text=Toyota%20agreed%20on%20Friday%20to,trials%20over%20the%20acceleration%20issue https://www.nytimes.com/2013/10/26/business/toyota-agrees-to.... Also let's remember that a company in UK was selling fake bomb detectors to Israeli and other militaries, and it took them more than 10 years to notice! https://www.bbc.co.uk/news/uk-29459896 https://www.bbc.co.uk/news/uk-29459896 There needs to be proper scrutiny into these things, I could start some random 'deep learning to find criminals' company tomorrow, and have less regulation than a car mechanic
- abandonliberty 6y ago
  Every attempt to replicate this has failed, and other vehicle manufacturers have just as many 'sudden acceleration events'. https://en.wikipedia.org/wiki/Sudden_unintended_acceleration https://en.wikipedia.org/wiki/Sudden_unintended_acceleration Occam's razer points to people just hitting the wrong pedal or people's floor mats getting stuck. The reason Toyota ate it in the press for this was competitive.
  - appleflaxen 6y ago
    > The reason Toyota ate it in the press for this was competitive. can you explain what you mean? I don't understand this sentence.
    - roryokane 6y ago
      I think they mean this: Toyota “ate it” (an idiom meaning to take losses) in the press by having bad stories written about them. These stories were written for a reason. This reason was not that Toyota deserving those stories, but that Toyota’s competition encouraged those stories.
- azernik 6y ago

dathinab 6y ago

I wouldn't be surprised if "incorrectly averaging" and similar are very common software errors. The reasons are manifold, including: - Normalized values need to be averaged differently the absolute values. - Floating point has limited precision, even just correctly summing/multiplying numbers need special care if you care about correctness. Results can, in the worst case, be of by a massive amount. Often you don't need to care about it so it's not uncommon for especially junior programmers to be not so aware about it. I mean in the last 3 years of working as a professional software engineer/developer I didn't need any of this at all, but once I do I know what to look out for.

text70 6y ago

A tool designed to find people guilty is biased to find people guilty. As far as I know it is fairly easy to take a generic dna sequencer meant for healtcare diagnostics, and repurpose it for STR analysis. The only major difference between the healthcare versions and the forensic versions is the software i/o.

sdflhasjd 6y ago
> A tool designed to find people guilty is biased to find people guilty. I don't see those particular issues make it biased, just inaccurate - it could go either way.
- radu_floricica 6y ago
  And yet somehow whenever you take a closer look at mislabeled product prices, the average is always in favor of the store. And that's far from the only industry. Complex tools are the product of many thousands of individual decisions taken by humans, humans aware of who's the paying client.
  - sdflhasjd 6y ago
    I still believe Hanlons razor applies. I've seen products that have serious performance affecting bugs caused by similar mistakes.
    - ncallaway 6y ago
      I still think—even when applying Hanlon's razor—there's an imbalance in incentives that leads to a weight in favor of the interests of the party paying for the test. Take the store pricing example. Suppose the store's pricing & labeling process produce an equal number of bugs at checkout in favor of the store and in opposition to the store. The store is heavily incentivized to detect the errors that are opposed to them. They are much less likely to detect the errors in their favor. Consider the manager that looks at the cash at the end of the day and notices they are $500 short. They likely dig hard to find the root cause of the issue, detect the pricing disparity and correct it. Now consider the manager that is $500 over at the end of the day. They are much more likely to say: "that's weird", shrug their shoulders and move on. The same applies to forensic tools. Even if they originally produced bugs in both directions, their own internal QA and the market of police officers are likely to work hard to detect bugs that make them less likely to allow them to make an arrest. The net result is that the tools end up with a bias in one direction, even if the original developers made an equal number of mistakes in both directions.
      - jfengel 6y ago
        Most store managers get as grumpy about overages as undercounts. They mean that some customer got shortchanged. For $500, it probably means a lot of customers got shortchanged, or something even worse is going on. That makes customers grumpy, and it affects your future. There are plenty of lazy managers who would sweep it under the rug once. But if it happens more than once, it can become their job on the line. They start looking for who's counting wrong. And if they can't figure that out, they get really worried. I have no idea about police officers and prosecutors. But store managers care about accuracy of counts, not just profits.

throw2838 6y ago

Draeger requires calibration prior each use for temperature and humidity. It is easy to get it thrown out as evidence.

lwigo 6y ago
When the officer shows up and swears under oath that he "calibrated it that morning" it's not so easy. :shrug:
- jdironman 6y ago
  Temps vary throughout the day.
  - lwigo 6y ago
    A rural small town judge probably doesn't have the best insight into that, just that Corporal Bubba said he calibrated it so it must've been right. I know, this is exceedingly cynical.
    - klyrs 6y ago
      Your "Corporal Bubba" isn't just cynical, it leans heavily upon a harmful stereotype that folks in small towns are uneducated simpletons. Stop the polarization, please
      - owl57 6y ago
        Why wouldn't they be? I went to university with people from all over Russia. They don't tend to return to their home towns after getting educated. Does this work differently in America?
        zymhan 6y ago
        > Does this work differently in America? No, not really.

ClumsyPilot 6y ago

This is also similar to how Toyota hilled people with control software that would cause car to accelerate randomly. The software audit team concluded that they could not find the bug, but the code was totally unreadable and terrible. They settled. https://www.nytimes.com/2013/10/26/business/toyota-agrees-to-settlement-in-fatal-acceleration-crash.html#:~:text=Toyota%20agreed%20on%20Friday%20to,trials%20over%20the%20acceleration%20issue https://www.nytimes.com/2013/10/26/business/toyota-agrees-to.... Also let's remember that a company in UK was selling fake bomb detectors to Israeli and other militaries, and it took them more than 10 years to notice! https://www.bbc.co.uk/news/uk-29459896 https://www.bbc.co.uk/news/uk-29459896 There needs to be proper scrutiny into these things, I could start some random 'deep learning to find criminals' company tomorrow, and have less regulation than a car mechanic

abandonliberty 6y ago
Every attempt to replicate this has failed, and other vehicle manufacturers have just as many 'sudden acceleration events'. https://en.wikipedia.org/wiki/Sudden_unintended_acceleration https://en.wikipedia.org/wiki/Sudden_unintended_acceleration Occam's razer points to people just hitting the wrong pedal or people's floor mats getting stuck. The reason Toyota ate it in the press for this was competitive.
- appleflaxen 6y ago
  > The reason Toyota ate it in the press for this was competitive. can you explain what you mean? I don't understand this sentence.
  - roryokane 6y ago
    I think they mean this: Toyota “ate it” (an idiom meaning to take losses) in the press by having bad stories written about them. These stories were written for a reason. This reason was not that Toyota deserving those stories, but that Toyota’s competition encouraged those stories.
azernik 6y ago