8 ms·

Sounds like it’s well-intentioned, but I’m afraid that given the incentives of all parties involved here, and the complexity of the technology, there will be a

by woah 7y ago

Sounds like it’s well-intentioned, but I’m afraid that given the incentives of all parties involved here, and the complexity of the technology, there will be a lot of false positives. It’s obvious that the police will want a device that produces more convictions, no point in disputing that. I’m wondering if you can elaborate on what incentives your company has to produce accurate devices, against your customer’s unstated preferences. What’s keeping you in check? Are you worried about class action lawsuits from victims of false positives, if it were independently proven that your device is overly sensitive?

mitchellst 7y ago
> It’s obvious that the police will want a device that produces more convictions, no point in disputing that. Is that obvious? If this were as overwhelming an incentive as you say, we wouldn't have accurate breathalyzers for alcohol. But we do, and it's simple to see why. 1) Cops aren't the only customers; medical establishments, individuals, and workplaces also have a legitimate interest in measuring impairment. 2) Tech like this doesn't remain novel forever, it can be checked for accuracy, and if you made a device that exaggerated impairment your reputation would tank, your company name would become a political hot potato, and your sales would never materialize. 3) There is definitely a point in disputing the idea that cops are indiscriminate gangsters roaming the streets trying to lock up any citizen they don't like the look of. Police have their personal and institutional biases, but by and large, most are interested in tools that allow them to safely and fairly administrate the rule of law. Police don't have much career or institutional incentive to arrest someone for a DUI at a traffic stop or not—they just stop bad drivers and check. It's prosecutors who are trying to rack up conviction numbers as a matter of career advancement and political viability. But even that gets checked: their work will be subject to an adversarial legal system, at which point any evidence that the device doesn't work would come out in court and tank their conviction rate. So... where do you see perverse incentives here, again?
- edmundsauto 7y ago
  Do we have accurate breathalyzers for alcohol? AFAIK, the manufacturers aggressively resist requirements to publish their source code or calibrations for testing. Also, AIUI, the devices require recalibration for accuracy, and it's not clear that police stations perform such maintenance. No idea how rampant this is, every system will have flaws. But I question whether we can use the alcohol breathalyzer example without acknowledging there are significant unanswered questions w/r/t accuracy.
  - ac29 7y ago
    If breathalyzers were inaccurate, uncalibrated, or unmaintained it would be trivial to get any conviction relying on their measurements thrown out.
    - monocasa 7y ago
      What makes you say that?
    - edmundsauto 7y ago
      Only if you're allowed to test the devices in question, and the courts believe your experts over the manufacturers. This doesn't prove my point, but manufacturers do aggressively fight against researchers identifying flaws in their products. Given the high stakes (+ the fact this isn't a security issue, really), it seems like a major smell to me. There are reports out there of people convicted, who are blocked from examining the devices used as the primary piece of evidence against them. Also, after years of effort to discredit breathalyzer results, convictions ARE being overturned regularly. NJ overturns > 27k convictions because of breathalyzer issues - https://www.nytimes.com/2018/11/13/nyregion/nj-dwi-convictions-court.html https://www.nytimes.com/2018/11/13/nyregion/nj-dwi-convictio... MS judge suspends admission of breathalyzer evidence -- https://www.bostonherald.com/2019/01/10/breathalyzers-temporarily-not-admissible-in-court-judge-rules/ https://www.bostonherald.com/2019/01/10/breathalyzers-tempor... More relevant info - https://www.zdnet.com/article/draeger-breathalyzer-breath-test-convictions/ https://www.zdnet.com/article/draeger-breathalyzer-breath-te...
    - jayd16 7y ago
- alex_young 7y ago
  The National Highway Traffic Safety Administration report to Congress [0] seems pretty clear in terms of the art of the science and the impairment observed from national data. I would avoid incarceration of anyone from a test like this without significant and clear data that there is an actual problem we need to resolve. There are plenty of well proven driving problems we could focus on correcting. [0] https://www.nhtsa.gov/sites/nhtsa.dot.gov/files/documents/812440-marijuana-impaired-driving-report-to-congress.pdf https://www.nhtsa.gov/sites/nhtsa.dot.gov/files/documents/81...
chadash 7y ago
> What’s keeping you in check? It's going to be fairly easy to test this. You take one group of people who haven't smoked in the last few hours and you test them. Make sure it doesn't go off. You take another group who have smoked (I'm sure plenty of college students would be happy to sign up for this trial) and you test them, make sure it does go off. It's pretty simple. It's a machine with an empirical number, so you don't even have to do this double blind. And if it doesn't give good results? It won't hold up in court. Every lawyer defending someone with a positive result (whether false positive or true positive) will argue that the device isn't accurate and the case gets tossed (or even the ability to use these as evidence). If it doesn't hold up in court, police really won't have much incentive to use these anyway.
- wolco 7y ago
  It is not that simple. Within the control group of non-smokers you need people who smoked yesterday or 12 hours before to determine if the testing is successful. But even that is too simplist. You need to activity give a specific dose to a group with similiar profiles. You need to test with different profile groups. Remember we are testing currently high not has been high.
  - wysifnwyg 7y ago
    The basic functionality of a breathalyzer is to take a given volume of an exhale and measure the concentration of a given particle as an indicator of intoxication. So long as an individual's results are greater than or equal to 0.08 then it becomes an unacceptable risk and they are considered intoxicated. If this process is going to test in a similar way, there is no need to test specific doses or profiles because the concentration isn't variable, it's a static amount. With alcohol, we don't consider weight, or what they had to drink, or how long they've drank so why should this be any different? All that needs to be defined is what the law considers a concentration which should represent a level where the risk reaches an unacceptable percentage.
    - chadash 7y ago
      I think maybe you misunderstood @wolco's point? I think what they are saying is that, for example, you might do all of your testing on college age males and find that it doesn't give any false positives for people who smoked yesterday but not today. But you still need to test it on 60 year old women to make sure it doesn't give false positives there. In other words, you need to test it for varying lengths of time and also to make sure that it has consistently acceptable results by race, sex, weight, age, etc.
ruffrey 7y ago
It makes me wonder how breathalyzers work. If you fail, I assume the police still need to analyze a urine sample to prevent false positives. But probable cause is established with a breathalyzer failure.
- deleted 7y ago
  [deleted]
- SannTek 7y ago
  It depends on the jurisdiction. In general, the police begin with a roadside test. This is either an standard field sobriety test (walking in a straight line, touching your nose, etc) or a screening breathalyzer. If the person fails, they are arrested and taken to the station for further testing. This is either performed using a more accurate breathalyzer, or a blood test. In general, suspects can choose which test they want (blood or breath).
dragonwriter 7y ago
> It’s obvious that the police will want a device that produces more convictions If a defendant ever challenges the use of the device, an inaccurate (particularly, false-positive-producing) device is more likely to be thrown out discrediting the device class, not just the individual case, and potentially forcing after-the-fact dismissal of prior convictions. So, avoidable false-positives isn't necessarily an approach consistent with maximizing convictions.
- jakelazaroff 7y ago
  Unfortunately, it doesn't seem to be that simple in practice; this comment by edmundsato [1] has a lot of good info. After-the-fact dismissal of convictions may seem like things eventually work out, but remember that in the meantime innocent people's lives are upended as they're thrown into the justice system. [1] https://news.ycombinator.com/item?id=20718599 https://news.ycombinator.com/item?id=20718599
  - dragonwriter 7y ago
    > After-the-fact dismissal of convictions may seem like things eventually work out No, it doesn't. But, if the police motivation is maximizing convictions that stick, it's a potentiality that reduces the degree to which a system that produces false positives meets that goal. (Unless you have a system that is as comprehensive a conspiracy as the FBI crime lab wholesale fabrication of the entire field of fiber analysis where essentially all the “experts” in the field were part of the scam...and even that eventually collapsed.)
    - jakelazaroff 7y ago
      My point is that before these scams "eventually collapse", people's real lives are caught in the crossfire. It doesn't even have to be a big conspiracy theory. Here is an instance of a breathalyzer manufacturer pressuring researchers to stop researching whether their devices are faulty [1]. Are they actually delivering false positives? Who knows! We need to bake oversight and accountability in from the jump, not try to back our way in once too many people's lives are disrupted. [1] https://www.zdnet.com/article/draeger-breathalyzer-breath-test-convictions/ https://www.zdnet.com/article/draeger-breathalyzer-breath-te...
SannTek 7y ago
You raise a good point. Interestingly, your comments run contrary to what a lot of law enforcement officers have said to us. In general, the police are hyper vigilant about buying only devices that are independently validated to be very accurate. Every conversation we have had has eventually lead to "is it NHTSA approved?". The reason for this need for third party validation is that the police are incredibly court room sensitive. If there is any chance a defense attorney would be able to pull out a study showing low specificity or sensitivity for a device, the police will simply not buy it. Third party validation gives them that guarantee. You are right that the police want more convictions (or less time consuming convictions), but the way they do that is by having very accurate devices that are defensible in court, not inaccurate devices that risk cases being thrown out. That is why making sure our device has low false positive is very important.
- neuralzen 7y ago
  I admire what you're doing, and hope you can adhere to scientific integrity while developing and selling your product, but that simply isn't true. At least in Washington state. A few years ago Washington State purchased a whole bunch of Draeger 9510 (alcohol breathalyzer) units without ever having tested them. After two years of sitting unused, the warranties were going to expire and they finally got around to testing them, and purchasing extended warranties at that point. However their initial tests failed multiple accuracy tests to the point that the state's lab halted testing, since it was so off. A few months later Draeger patched the firmware and OS application (the breath/alcohol measurement firmware being written by a single contractor in Texas, and versioned as 0.7) and while testing looked better, it still wasn't accurate and they just pushed it out the door for deployment...they had already bought them, after all. I'll link the records below...they didn't even have the correct versions of the software and firmware accounted for and signed off, which led to other problems (and Draegar has had other problems of accuracy in other court cases, such as Florida, but that's a side tangent). My point is, a lot of state governments (at least in the US) spend money first and verify later, and 'verify' can be a damnably loose term. Loose enough to violate WAC 448-15-020 (in Washington state's case) which requires a 'reasonable degree of scientific accuracy'. EDIT: Apparently the original link (below) is no longer available in WSP's records, which is interesting. You can view the document via archive.org, here https://web.archive.org/web/20170211114622/http://www.wsp.wa.gov/breathtest/docs/webdms/Draeger/Valid_Docs/Draeger%20Validation%20Approval.pdf https://web.archive.org/web/20170211114622/http://www.wsp.wa... original (defunct) http://www.wsp.wa.gov/breathtest/docs/webdms/Draeger/Valid_Docs/Draeger%20Validation%20Approval.pdf http://www.wsp.wa.gov/breathtest/docs/webdms/Draeger/Valid_D...
Daniel3 7y ago
"It’s obvious that the police will want a device that produces more convictions, no point in disputing that." Uh, why couldn't we talk about it as adults? OP's answer below is excellent. Police want a device that will make their job most effective and profitable, with the lowest margin not only for legal repercussions, but also the lowest possibility of people like you deciding that all police officers are pure evil who just want as many convictions as possible. Doesn't everybody want their job to be like that? BLM has made some good progress (and more is needed yet) scaring some pure evil people who happen to be police officers into wanting what's good and right. Probably not prudent to worry about a new technology that might be oversensitive, especially when the first time this thing is used in a conviction by a person with a rich dad, they'll spend as much as possible throwing doubt on the research supporting it, or how the officer was trained to use it, or etc etc etc, in order to build plausible deniability in court and get the case thrown out.

by woah 7y ago

mitchellst 7y ago

> It’s obvious that the police will want a device that produces more convictions, no point in disputing that. Is that obvious? If this were as overwhelming an incentive as you say, we wouldn't have accurate breathalyzers for alcohol. But we do, and it's simple to see why. 1) Cops aren't the only customers; medical establishments, individuals, and workplaces also have a legitimate interest in measuring impairment. 2) Tech like this doesn't remain novel forever, it can be checked for accuracy, and if you made a device that exaggerated impairment your reputation would tank, your company name would become a political hot potato, and your sales would never materialize. 3) There is definitely a point in disputing the idea that cops are indiscriminate gangsters roaming the streets trying to lock up any citizen they don't like the look of. Police have their personal and institutional biases, but by and large, most are interested in tools that allow them to safely and fairly administrate the rule of law. Police don't have much career or institutional incentive to arrest someone for a DUI at a traffic stop or not—they just stop bad drivers and check. It's prosecutors who are trying to rack up conviction numbers as a matter of career advancement and political viability. But even that gets checked: their work will be subject to an adversarial legal system, at which point any evidence that the device doesn't work would come out in court and tank their conviction rate. So... where do you see perverse incentives here, again?

edmundsauto 7y ago
Do we have accurate breathalyzers for alcohol? AFAIK, the manufacturers aggressively resist requirements to publish their source code or calibrations for testing. Also, AIUI, the devices require recalibration for accuracy, and it's not clear that police stations perform such maintenance. No idea how rampant this is, every system will have flaws. But I question whether we can use the alcohol breathalyzer example without acknowledging there are significant unanswered questions w/r/t accuracy.
- ac29 7y ago
  If breathalyzers were inaccurate, uncalibrated, or unmaintained it would be trivial to get any conviction relying on their measurements thrown out.
  - monocasa 7y ago
    What makes you say that?
  - edmundsauto 7y ago
    Only if you're allowed to test the devices in question, and the courts believe your experts over the manufacturers. This doesn't prove my point, but manufacturers do aggressively fight against researchers identifying flaws in their products. Given the high stakes (+ the fact this isn't a security issue, really), it seems like a major smell to me. There are reports out there of people convicted, who are blocked from examining the devices used as the primary piece of evidence against them. Also, after years of effort to discredit breathalyzer results, convictions ARE being overturned regularly. NJ overturns > 27k convictions because of breathalyzer issues - https://www.nytimes.com/2018/11/13/nyregion/nj-dwi-convictions-court.html https://www.nytimes.com/2018/11/13/nyregion/nj-dwi-convictio... MS judge suspends admission of breathalyzer evidence -- https://www.bostonherald.com/2019/01/10/breathalyzers-temporarily-not-admissible-in-court-judge-rules/ https://www.bostonherald.com/2019/01/10/breathalyzers-tempor... More relevant info - https://www.zdnet.com/article/draeger-breathalyzer-breath-test-convictions/ https://www.zdnet.com/article/draeger-breathalyzer-breath-te...
  - jayd16 7y ago
alex_young 7y ago
The National Highway Traffic Safety Administration report to Congress [0] seems pretty clear in terms of the art of the science and the impairment observed from national data. I would avoid incarceration of anyone from a test like this without significant and clear data that there is an actual problem we need to resolve. There are plenty of well proven driving problems we could focus on correcting. [0] https://www.nhtsa.gov/sites/nhtsa.dot.gov/files/documents/812440-marijuana-impaired-driving-report-to-congress.pdf https://www.nhtsa.gov/sites/nhtsa.dot.gov/files/documents/81...

chadash 7y ago

> What’s keeping you in check? It's going to be fairly easy to test this. You take one group of people who haven't smoked in the last few hours and you test them. Make sure it doesn't go off. You take another group who have smoked (I'm sure plenty of college students would be happy to sign up for this trial) and you test them, make sure it does go off. It's pretty simple. It's a machine with an empirical number, so you don't even have to do this double blind. And if it doesn't give good results? It won't hold up in court. Every lawyer defending someone with a positive result (whether false positive or true positive) will argue that the device isn't accurate and the case gets tossed (or even the ability to use these as evidence). If it doesn't hold up in court, police really won't have much incentive to use these anyway.

wolco 7y ago
It is not that simple. Within the control group of non-smokers you need people who smoked yesterday or 12 hours before to determine if the testing is successful. But even that is too simplist. You need to activity give a specific dose to a group with similiar profiles. You need to test with different profile groups. Remember we are testing currently high not has been high.
- wysifnwyg 7y ago
  The basic functionality of a breathalyzer is to take a given volume of an exhale and measure the concentration of a given particle as an indicator of intoxication. So long as an individual's results are greater than or equal to 0.08 then it becomes an unacceptable risk and they are considered intoxicated. If this process is going to test in a similar way, there is no need to test specific doses or profiles because the concentration isn't variable, it's a static amount. With alcohol, we don't consider weight, or what they had to drink, or how long they've drank so why should this be any different? All that needs to be defined is what the law considers a concentration which should represent a level where the risk reaches an unacceptable percentage.
  - chadash 7y ago
    I think maybe you misunderstood @wolco's point? I think what they are saying is that, for example, you might do all of your testing on college age males and find that it doesn't give any false positives for people who smoked yesterday but not today. But you still need to test it on 60 year old women to make sure it doesn't give false positives there. In other words, you need to test it for varying lengths of time and also to make sure that it has consistently acceptable results by race, sex, weight, age, etc.

ruffrey 7y ago

It makes me wonder how breathalyzers work. If you fail, I assume the police still need to analyze a urine sample to prevent false positives. But probable cause is established with a breathalyzer failure.

deleted 7y ago
[deleted]
SannTek 7y ago
It depends on the jurisdiction. In general, the police begin with a roadside test. This is either an standard field sobriety test (walking in a straight line, touching your nose, etc) or a screening breathalyzer. If the person fails, they are arrested and taken to the station for further testing. This is either performed using a more accurate breathalyzer, or a blood test. In general, suspects can choose which test they want (blood or breath).

dragonwriter 7y ago

> It’s obvious that the police will want a device that produces more convictions If a defendant ever challenges the use of the device, an inaccurate (particularly, false-positive-producing) device is more likely to be thrown out discrediting the device class, not just the individual case, and potentially forcing after-the-fact dismissal of prior convictions. So, avoidable false-positives isn't necessarily an approach consistent with maximizing convictions.

jakelazaroff 7y ago
Unfortunately, it doesn't seem to be that simple in practice; this comment by edmundsato [1] has a lot of good info. After-the-fact dismissal of convictions may seem like things eventually work out, but remember that in the meantime innocent people's lives are upended as they're thrown into the justice system. [1] https://news.ycombinator.com/item?id=20718599 https://news.ycombinator.com/item?id=20718599
- dragonwriter 7y ago
  > After-the-fact dismissal of convictions may seem like things eventually work out No, it doesn't. But, if the police motivation is maximizing convictions that stick, it's a potentiality that reduces the degree to which a system that produces false positives meets that goal. (Unless you have a system that is as comprehensive a conspiracy as the FBI crime lab wholesale fabrication of the entire field of fiber analysis where essentially all the “experts” in the field were part of the scam...and even that eventually collapsed.)
  - jakelazaroff 7y ago
    My point is that before these scams "eventually collapse", people's real lives are caught in the crossfire. It doesn't even have to be a big conspiracy theory. Here is an instance of a breathalyzer manufacturer pressuring researchers to stop researching whether their devices are faulty [1]. Are they actually delivering false positives? Who knows! We need to bake oversight and accountability in from the jump, not try to back our way in once too many people's lives are disrupted. [1] https://www.zdnet.com/article/draeger-breathalyzer-breath-test-convictions/ https://www.zdnet.com/article/draeger-breathalyzer-breath-te...

SannTek 7y ago

You raise a good point. Interestingly, your comments run contrary to what a lot of law enforcement officers have said to us. In general, the police are hyper vigilant about buying only devices that are independently validated to be very accurate. Every conversation we have had has eventually lead to "is it NHTSA approved?". The reason for this need for third party validation is that the police are incredibly court room sensitive. If there is any chance a defense attorney would be able to pull out a study showing low specificity or sensitivity for a device, the police will simply not buy it. Third party validation gives them that guarantee. You are right that the police want more convictions (or less time consuming convictions), but the way they do that is by having very accurate devices that are defensible in court, not inaccurate devices that risk cases being thrown out. That is why making sure our device has low false positive is very important.

neuralzen 7y ago
I admire what you're doing, and hope you can adhere to scientific integrity while developing and selling your product, but that simply isn't true. At least in Washington state. A few years ago Washington State purchased a whole bunch of Draeger 9510 (alcohol breathalyzer) units without ever having tested them. After two years of sitting unused, the warranties were going to expire and they finally got around to testing them, and purchasing extended warranties at that point. However their initial tests failed multiple accuracy tests to the point that the state's lab halted testing, since it was so off. A few months later Draeger patched the firmware and OS application (the breath/alcohol measurement firmware being written by a single contractor in Texas, and versioned as 0.7) and while testing looked better, it still wasn't accurate and they just pushed it out the door for deployment...they had already bought them, after all. I'll link the records below...they didn't even have the correct versions of the software and firmware accounted for and signed off, which led to other problems (and Draegar has had other problems of accuracy in other court cases, such as Florida, but that's a side tangent). My point is, a lot of state governments (at least in the US) spend money first and verify later, and 'verify' can be a damnably loose term. Loose enough to violate WAC 448-15-020 (in Washington state's case) which requires a 'reasonable degree of scientific accuracy'. EDIT: Apparently the original link (below) is no longer available in WSP's records, which is interesting. You can view the document via archive.org, here https://web.archive.org/web/20170211114622/http://www.wsp.wa.gov/breathtest/docs/webdms/Draeger/Valid_Docs/Draeger%20Validation%20Approval.pdf https://web.archive.org/web/20170211114622/http://www.wsp.wa... original (defunct) http://www.wsp.wa.gov/breathtest/docs/webdms/Draeger/Valid_Docs/Draeger%20Validation%20Approval.pdf http://www.wsp.wa.gov/breathtest/docs/webdms/Draeger/Valid_D...

Daniel3 7y ago

"It’s obvious that the police will want a device that produces more convictions, no point in disputing that." Uh, why couldn't we talk about it as adults? OP's answer below is excellent. Police want a device that will make their job most effective and profitable, with the lowest margin not only for legal repercussions, but also the lowest possibility of people like you deciding that all police officers are pure evil who just want as many convictions as possible. Doesn't everybody want their job to be like that? BLM has made some good progress (and more is needed yet) scaring some pure evil people who happen to be police officers into wanting what's good and right. Probably not prudent to worry about a new technology that might be oversensitive, especially when the first time this thing is used in a conviction by a person with a rich dad, they'll spend as much as possible throwing doubt on the research supporting it, or how the officer was trained to use it, or etc etc etc, in order to build plausible deniability in court and get the case thrown out.