13 ms·

Ouch: > Over the years, ShotSpotter’s claims about its accuracy have increased, from 80 percent accurate to 90 percent accurate to 97 percent accurate. Accordi

by dlgeek 5y ago

Ouch: > Over the years, ShotSpotter’s claims about its accuracy have increased, from 80 percent accurate to 90 percent accurate to 97 percent accurate. According to Greene, those numbers aren’t actually calculated by engineers, though. > “Our guarantee was put together by our sales and marketing department, not our engineers,” Greene told a San Francisco court in 2017. “We need to give them [customers] a number … We have to tell them something. … It’s not perfect. The dot on the map is simply a starting point.”

ghostly_s 5y ago
In fact, 89% of ShotSpotter deployments turn up no evidence of gun-related crime: [1] 1. https://endpolicesurveillance.com/ https://endpolicesurveillance.com/
- bpodgursky 5y ago
  I'm not sure why this is an important stat. Isn't it good if there aren't gun crimes happening in a lot of places?
  - sodality2 5y ago
    Deployments I assume means police response, not deploying the system.
  - Cycl0ps 5y ago
    Check the link. What it's saying is that most times the police receive a notice of a gunshot, it's a false positive. In Chicago this leads to 60 dead-end responses each day. Very few systems are worth that trouble
    - handmodel 5y ago
      I 100% dont think it should be used as evidence but this seems like a decent rate for responding. There's 12,000 total officers in Chicago. Even if only 3,000 are on duty at any time that's not a huge inconvenience if there are still even dozens of cases a year you get to the scene 5 minutes before you would otherwise. If you track the number of calls to 911 that are "dead-ends" because there are no arrests I'm sure it would be high too. It also doesnt say the error rate for when the police aren't called. If criminals know that if they shoot a gun there is a 50% chance a police car will drive through 5 minutes later - wouldn't that deter you?
      - Cycl0ps 5y ago
        Yeah, in fairness the steam controller had its perks. I didn't see the appeal of the two pads, but it was the first I can recall to do half/full pull triggers. Their customization also allowed you to do some crazy things. Personal favorite of mine was setting the gyroscope to act as right joystick, but only while zoomed in. This let you fine adjust your aim with about as much precision as with a mouse. In the end though every game had to be translated to the controller and there was always enough of a change to make it feel unintuitive. If Valve had paid an intern to go through the top games each day and make official layouts for games that roughly lined up with each other I'd probably be using one now.
- adolph 5y ago
  > 89% of ShotSpotter deployments turn up no evidence of gun-related crime If ShotSpotter is limited to registering the the sound of gunshots, then it is limited to firearm-discharge-related crime which is a subset of gun-related crime.
- meowface 5y ago
  That site says 14% of alerts result in a case report being filed and 10% result in a gun-related case report. But even a 10% alert -> case report rate is better than I would've expected for a big city. That still seems like it'd be extremely useful, even if it means some wasted man hours. And that's case reports - not necessarily detection true positives. If someone fires a gun at the sky and picks up the shell casings and drives away, with no witnesses or cameras and an inconclusive/indiscernible audio recording, police likely won't be able to file a report. Also, I'm guessing there are some confidence indicators for each alert: those case report rates probably don't capture the full story even when accounting for hypothetical inconclusive true positives. Police are likely (and should be) obligated to investigate every detection, including every detection rated as low-confidence. Plus that's just for Chicago. I doubt they actually achieve their claimed 97% accuracy anywhere, but it could be higher in other areas. Especially areas where loud noises are less common. The main issue seems to be when it's used as evidence. I think it shouldn't ever be admissible as evidence of anything. And to mitigate some of the other issues that site raises, it'd be nice if there were some kind of "no fishing expeditions" law, where police responding to a gunfire alert must exclusively look for evidence of a fired gun in that vicinity and not use it as an excuse to scavenge for low-hanging fruit by harassing / arresting people in the area for unrelated matters.
  - giantg2 5y ago
    "That still seems like it'd be extremely useful, even if it means some wasted man hours." I think the least wasteful approach would be to build rapport with the community and rely on their reports. Probably not an easy thing, but the most valuable things rarely are.
    - whimsicalism 5y ago
      Still doesn't stop the phenomena of everyone thinking "oh, someone else will call." Shotspotter is a very useful tool IMO.
      - pessimizer 5y ago
        > the phenomena of everyone thinking "oh, someone else will call." This generally doesn't happen. It originated as a slander against the people who lived around where Kitty Genovese was killed, as an after-the-fact excuse for the lack of police response. > While there was no question that the attack occurred, and that some neighbors ignored cries for help, the portrayal of 38 witnesses as fully aware and unresponsive was erroneous. The article grossly exaggerated the number of witnesses and what they had perceived. None saw the attack in its entirety. Only a few had glimpsed parts of it, or recognized the cries for help. Many thought they had heard lovers or drunks quarreling. There were two attacks, not three. And afterward, two people did call the police. A 70-year-old woman ventured out and cradled the dying victim in her arms until they arrived. Ms. Genovese died on the way to a hospital. https://en.wikipedia.org/wiki/Murder_of_Kitty_Genovese#Accuracy_of_original_reports https://en.wikipedia.org/wiki/Murder_of_Kitty_Genovese#Accur...
- GhostVII 5y ago
  Having an 11% success rate seems pretty impressive to me. Much better than the rate of false panic alarms, for example. And I'm sure a lot of those ShotSpotter alerts were accurate, but the police didn't get there soon enough.
  - handmodel 5y ago
    I think for the purposes of sending a car over to drive through a street its pretty good. The article discusses instances where it is used in trial and for that it is obviously pretty weak - surprised it can be used at all.
    - beerandt 5y ago
      I can see a use to confirm timing, or approx location, or as backing up a witnesses testimony. But it shouldn't be used as stand alone evidence.
  - giantg2 5y ago
    It shouldn't have anything to do with getting there soon enough. If they find shell casings, victims, or other evidence that a crime occurred (usually even discharging a firearm in a major city is considered a crime with very limited defenses), then they should still be collecting evidence and logging a report. Now, they may not be doing that because they don't want to have high crime numbers that are unsolved, or even dragging down the reputation of the safer areas (I've heard of officers doing this).
    - GhostVII 5y ago
      I was thinking of cases where someone fires a few shots and then everyone runs off - odds are the shell casings aren't going to be seen unless they do a careful sweep of the area, and the victims all left pretty quickly. No idea how often that actually happens though.
      - giantg2 5y ago
        Yeah, but if the people running off aren't calling in, then the technology isn't going to make any impact until the public relations change.
BoorishBears 5y ago
That's the ouch? Not literally rewriting history to create evidence for a murder trial? > But after the 11:46 p.m. alert came in, a ShotSpotter analyst manually overrode the algorithms and “reclassified” the sound as a gunshot. Then, months later and after “post-processing,” another ShotSpotter analyst changed the alert’s coordinates to a location on South Stony Island Drive near where Williams’ car was seen on camera. > Motherboard’s review of court documents from the Williams case and other trials in Chicago and New York State, including testimony from ShotSpotter’s favored expert witness, suggests that the company’s analysts frequently modify alerts at the request of police departments—some of which appear to be grasping for evidence that supports their narrative of events. Marketing games are absolutely nothing compared to this.
- klyrs 5y ago
  This sounds suspiciously like evidence tampering. If only the legal system would be so dogged in pursuit of justice for the cops & analysts involved in this conspiracy.
pjungwir 5y ago
Any time I see "accuracy" reported as a single number, I assume it's B.S. and "for marketing purposes only". If you're serious, give me the precision and recall.
- the_snooze 5y ago
  >If you're serious, give me the precision and recall. The thing is decision-makers have no idea (and no care to know) what those are.
  - wcerfgba 5y ago
    Maybe these people shouldn't be decision-makers then. We need both greater transparency in decision-making processes, and greater education amongst the population about statistics, causal inference, and the scientific method. In order for the citizenry to hold decision-makers to account and obtain a higher standard we need: 1. Citizens to have the time and motivation to seek out and examine primary evidence. 2. Citizens to have the skills to understand and critique that evidence. 3. Robust mechanisms to allow citizens to replace people in power.
    - kube-system 5y ago
      This is an inherent problem with the specialization of labor. Those who run police departments have more relevant specializations to their job role than information science. This isn't a problem with statistics or science in particular -- it's an issue with every specialization. This is only going to get worse as popular opinion of the role of higher education is starting to shift from a more liberal-arts focus to a job-placement focus.
  - anigbrowl 5y ago
    Correct and that's unlikely to change, but it should be grounds to throw out the prosecution. Yes, this will make it harder to convict real criminals. Too bad.
- drc500free 5y ago
  Precision is an attribute of the deployed system. The same tech in an affluent suburb will have different precision deployed in an inner city. Ran into this exact problem in biometrics, where NIST publishes tech error rates like FNMR and FMR, and system owners don't even know precision is a thing. They expect the vendor to tell them if it will work, when it depends how many true and false events you throw at the system.
cryptonector 5y ago
This is all devastating. These kinds of devices should use a a cryptographic tramper-evident log, and those logs should be available to the public in real time for archival purposes. That's the only way to avoid tampering. The same should be used for things like, e.g., rolls of who voted at every precinct.
LegitShady 5y ago
That sounds like fraud

by dlgeek 5y ago

ghostly_s 5y ago

In fact, 89% of ShotSpotter deployments turn up no evidence of gun-related crime: [1] 1. https://endpolicesurveillance.com/ https://endpolicesurveillance.com/

bpodgursky 5y ago
I'm not sure why this is an important stat. Isn't it good if there aren't gun crimes happening in a lot of places?
- sodality2 5y ago
  Deployments I assume means police response, not deploying the system.
- Cycl0ps 5y ago
  Check the link. What it's saying is that most times the police receive a notice of a gunshot, it's a false positive. In Chicago this leads to 60 dead-end responses each day. Very few systems are worth that trouble
  - handmodel 5y ago
    I 100% dont think it should be used as evidence but this seems like a decent rate for responding. There's 12,000 total officers in Chicago. Even if only 3,000 are on duty at any time that's not a huge inconvenience if there are still even dozens of cases a year you get to the scene 5 minutes before you would otherwise. If you track the number of calls to 911 that are "dead-ends" because there are no arrests I'm sure it would be high too. It also doesnt say the error rate for when the police aren't called. If criminals know that if they shoot a gun there is a 50% chance a police car will drive through 5 minutes later - wouldn't that deter you?
    - Cycl0ps 5y ago
      Yeah, in fairness the steam controller had its perks. I didn't see the appeal of the two pads, but it was the first I can recall to do half/full pull triggers. Their customization also allowed you to do some crazy things. Personal favorite of mine was setting the gyroscope to act as right joystick, but only while zoomed in. This let you fine adjust your aim with about as much precision as with a mouse. In the end though every game had to be translated to the controller and there was always enough of a change to make it feel unintuitive. If Valve had paid an intern to go through the top games each day and make official layouts for games that roughly lined up with each other I'd probably be using one now.
adolph 5y ago
> 89% of ShotSpotter deployments turn up no evidence of gun-related crime If ShotSpotter is limited to registering the the sound of gunshots, then it is limited to firearm-discharge-related crime which is a subset of gun-related crime.
meowface 5y ago
That site says 14% of alerts result in a case report being filed and 10% result in a gun-related case report. But even a 10% alert -> case report rate is better than I would've expected for a big city. That still seems like it'd be extremely useful, even if it means some wasted man hours. And that's case reports - not necessarily detection true positives. If someone fires a gun at the sky and picks up the shell casings and drives away, with no witnesses or cameras and an inconclusive/indiscernible audio recording, police likely won't be able to file a report. Also, I'm guessing there are some confidence indicators for each alert: those case report rates probably don't capture the full story even when accounting for hypothetical inconclusive true positives. Police are likely (and should be) obligated to investigate every detection, including every detection rated as low-confidence. Plus that's just for Chicago. I doubt they actually achieve their claimed 97% accuracy anywhere, but it could be higher in other areas. Especially areas where loud noises are less common. The main issue seems to be when it's used as evidence. I think it shouldn't ever be admissible as evidence of anything. And to mitigate some of the other issues that site raises, it'd be nice if there were some kind of "no fishing expeditions" law, where police responding to a gunfire alert must exclusively look for evidence of a fired gun in that vicinity and not use it as an excuse to scavenge for low-hanging fruit by harassing / arresting people in the area for unrelated matters.
- giantg2 5y ago
  "That still seems like it'd be extremely useful, even if it means some wasted man hours." I think the least wasteful approach would be to build rapport with the community and rely on their reports. Probably not an easy thing, but the most valuable things rarely are.
  - whimsicalism 5y ago
    Still doesn't stop the phenomena of everyone thinking "oh, someone else will call." Shotspotter is a very useful tool IMO.
    - pessimizer 5y ago
      > the phenomena of everyone thinking "oh, someone else will call." This generally doesn't happen. It originated as a slander against the people who lived around where Kitty Genovese was killed, as an after-the-fact excuse for the lack of police response. > While there was no question that the attack occurred, and that some neighbors ignored cries for help, the portrayal of 38 witnesses as fully aware and unresponsive was erroneous. The article grossly exaggerated the number of witnesses and what they had perceived. None saw the attack in its entirety. Only a few had glimpsed parts of it, or recognized the cries for help. Many thought they had heard lovers or drunks quarreling. There were two attacks, not three. And afterward, two people did call the police. A 70-year-old woman ventured out and cradled the dying victim in her arms until they arrived. Ms. Genovese died on the way to a hospital. https://en.wikipedia.org/wiki/Murder_of_Kitty_Genovese#Accuracy_of_original_reports https://en.wikipedia.org/wiki/Murder_of_Kitty_Genovese#Accur...
GhostVII 5y ago
Having an 11% success rate seems pretty impressive to me. Much better than the rate of false panic alarms, for example. And I'm sure a lot of those ShotSpotter alerts were accurate, but the police didn't get there soon enough.
- handmodel 5y ago
  I think for the purposes of sending a car over to drive through a street its pretty good. The article discusses instances where it is used in trial and for that it is obviously pretty weak - surprised it can be used at all.
  - beerandt 5y ago
    I can see a use to confirm timing, or approx location, or as backing up a witnesses testimony. But it shouldn't be used as stand alone evidence.
- giantg2 5y ago
  It shouldn't have anything to do with getting there soon enough. If they find shell casings, victims, or other evidence that a crime occurred (usually even discharging a firearm in a major city is considered a crime with very limited defenses), then they should still be collecting evidence and logging a report. Now, they may not be doing that because they don't want to have high crime numbers that are unsolved, or even dragging down the reputation of the safer areas (I've heard of officers doing this).
  - GhostVII 5y ago
    I was thinking of cases where someone fires a few shots and then everyone runs off - odds are the shell casings aren't going to be seen unless they do a careful sweep of the area, and the victims all left pretty quickly. No idea how often that actually happens though.
    - giantg2 5y ago
      Yeah, but if the people running off aren't calling in, then the technology isn't going to make any impact until the public relations change.

BoorishBears 5y ago

That's the ouch? Not literally rewriting history to create evidence for a murder trial? > But after the 11:46 p.m. alert came in, a ShotSpotter analyst manually overrode the algorithms and “reclassified” the sound as a gunshot. Then, months later and after “post-processing,” another ShotSpotter analyst changed the alert’s coordinates to a location on South Stony Island Drive near where Williams’ car was seen on camera. > Motherboard’s review of court documents from the Williams case and other trials in Chicago and New York State, including testimony from ShotSpotter’s favored expert witness, suggests that the company’s analysts frequently modify alerts at the request of police departments—some of which appear to be grasping for evidence that supports their narrative of events. Marketing games are absolutely nothing compared to this.

klyrs 5y ago
This sounds suspiciously like evidence tampering. If only the legal system would be so dogged in pursuit of justice for the cops & analysts involved in this conspiracy.

pjungwir 5y ago

Any time I see "accuracy" reported as a single number, I assume it's B.S. and "for marketing purposes only". If you're serious, give me the precision and recall.

the_snooze 5y ago
>If you're serious, give me the precision and recall. The thing is decision-makers have no idea (and no care to know) what those are.
- wcerfgba 5y ago
  Maybe these people shouldn't be decision-makers then. We need both greater transparency in decision-making processes, and greater education amongst the population about statistics, causal inference, and the scientific method. In order for the citizenry to hold decision-makers to account and obtain a higher standard we need: 1. Citizens to have the time and motivation to seek out and examine primary evidence. 2. Citizens to have the skills to understand and critique that evidence. 3. Robust mechanisms to allow citizens to replace people in power.
  - kube-system 5y ago
    This is an inherent problem with the specialization of labor. Those who run police departments have more relevant specializations to their job role than information science. This isn't a problem with statistics or science in particular -- it's an issue with every specialization. This is only going to get worse as popular opinion of the role of higher education is starting to shift from a more liberal-arts focus to a job-placement focus.
- anigbrowl 5y ago
  Correct and that's unlikely to change, but it should be grounds to throw out the prosecution. Yes, this will make it harder to convict real criminals. Too bad.
drc500free 5y ago
Precision is an attribute of the deployed system. The same tech in an affluent suburb will have different precision deployed in an inner city. Ran into this exact problem in biometrics, where NIST publishes tech error rates like FNMR and FMR, and system owners don't even know precision is a thing. They expect the vendor to tell them if it will work, when it depends how many true and false events you throw at the system.

cryptonector 5y ago

This is all devastating. These kinds of devices should use a a cryptographic tramper-evident log, and those logs should be available to the public in real time for archival purposes. That's the only way to avoid tampering. The same should be used for things like, e.g., rolls of who voted at every precinct.

LegitShady 5y ago

That sounds like fraud