9 ms·

The article says that employees of the AI company (ShotSpotter) manually reviewed and classified the sounds as gunshots: > records showed that ShotSpotter actu

The article says that employees of the AI company (ShotSpotter) manually reviewed and classified the sounds as gunshots: > records showed that ShotSpotter actually initially picked up what sounded like a firework a mile away, and this was later reclassified by ShotSpotter staff to be a gunshot at the intersection where and when Williams was seen on camera. So the AI didn’t even make the call. The staff did, manually. I assume that means the actual audio is available and entered into evidence? If humans are making the call then blaming AI seems like a stretch. That’s almost like blaming the motion detection algorithms for triggering video recordings that were later reviewed by humans. It’s still humans reviewing the recordings and making decisions.

jtbayly 5y ago
It may not be right to blame AI, but it's certainly the case that they were presenting it as "the system says there was a shot fired here." The bigger point is that the "system" can't be trusted. And the fact that they need to manually classify things is proof of that fact. (It's way more complicated than just an audio recording. It's recordings from numerous locations, attempting to triangulate the original location, without being confused by echoes and other things.)
- PragmaticPulp 5y ago
  > It may not be right to blame AI, but it's certainly the case that they were presenting it as "the system says there was a shot fired here." I think a crucial distinction is to clarify who “they” is: Who was making the claim? From the article, it seems that the prosecutors may have been the ones exaggerating the evidence: > Prosecutors said ShotSpotter picked up a gunshot sound where Williams was seen on surveillance camera footage in his car, putting it all forward as proof that Williams shot Herring right there and then. Naturally the defense wanted to see the actual evidence, so that’s what they got down to. I know everyone wants to vilify AI, but I think the actual bad actor in this article might be the prosecutor trying to misrepresent the evidence.
  - Aerroon 5y ago
    >I think a crucial distinction is to clarify who “they” is: Who was making the claim? From the article, it seems that the prosecutors may have been the ones exaggerating the evidence: The prosecutors are the "they". The further removed you get from people who know the 'AI' the more the validity of the evidence provided by the AI relies merely on what others have said. The people working on the system know that it can be faulty. Other software developers have an idea that it could be faulty. Prosecutors are told that it's alright. They turn around and argue it in a court of law. And the further you get the less people have any idea that the system is faulty. Down the line people will go "the machine says so, therefore it must be so".
    - ClumsyPilot 5y ago
      >Down the line people will go "the machine says so, therefore it must be so". Thats the real feature of AI, devision laundering: a flawed decision made by someone for their own gain now appears impartial and trustworthy, and come wothout accountability
  - jtbayly 5y ago
    Sure. But the point is that they misrepresent the evidence by pointing to the system as trustworthy. Even without tampering with the evidence the system isn’t trustworthy beyond a reasonable shadow of a doubt.
stefan_ 5y ago
That's the point? These systems invariably turn out to not be any sort of AI at all, but sales people ready to make a call any way their customer (police) wants it made. Later it's presented as AI to a jury. It's basically the same scam as lots of forensics, junk science to get a desired result.
- giantg2 5y ago
  I generally agree, but wouldn't necessarily say a lot of forensics is junk science. The science is generally sound (procedures and theory). Usually it's the prosecutors that present the forensics in a way that misrepresents what the results actually mean. Or lab techs make mistakes or perform misconduct (How to Fix a Drug Scandal on Netflix is interesting).
  - wyager 5y ago
    Fingerprinting is junk science. Polygraphs are junk science. Tons of things that have been or are admissible as evidence are junk science.
    - giantg2 5y ago
      I wouldn't call polygraphs forensics. Most states do not allow them as evidence. Do you have a link for the fingerprinting claim?
      - monetus 5y ago
        (Not who you responded to) https://www.crammlawfirm.com/fingerprint-match-jails-innocent-man/ https://www.crammlawfirm.com/fingerprint-match-jails-innocen... This isn't the case I originally recalled, and doesn't say how many points were matched, but I remember a 5 point match being used to convict a person who was later proven innocent.
        newswasboring 5y ago
        That shows its flawed, not junk.
helsinkiandrew 5y ago
It's scary that prosecutors thought that the ShotSpotter evidence is suitable for a court. It's amazing technology that can help police get to a crime scene quickly but is far too easy to spoof or get the wrong answer due to reflections and other sounds etc. Hopefully cases like this will eventually stop the system being used in court.
- SamoyedFurFluff 5y ago
  I mean, drug dogs are still a thing even though we know they tend to respond to owners desires regardless of if there’s drugs present.
  - yawaworht1978 5y ago
    Wow i have never heard of that. Curious to see a link/article on the issue, if you can provide.
    - DaftDank 5y ago
      This is a decent summary of court cases related to the dogs: https://www.nlrg.com/criminal-law-legal-research/bid/84670/CRIMINAL-LAW-UPDATE-Reliability-of-Narcotics-Dogs-to-Be-Revisited-by-U-S-Supreme-Court https://www.nlrg.com/criminal-law-legal-research/bid/84670/C... I think this part is pretty key, and has always been a big open question in my mind: "The court further recognized a potential problem in assessing reliability related to handler cuing, as even well-trained dogs could respond to subconscious cues from their handlers. For this reason, the court determined that a critical factor in determining reliability is the record of false positive alerts made by the dog." The subconscious cues, or even deliberate cues that only the handler and dog know, could allow an officer with ill intentions to make some kind of signal only known to him and the dog, which causes the dog to make an "alert", giving the cop probable cause.
      - leephillips 5y ago
        Some departments deploy dogs that alert on all vehicles when they want an excuse to search. In at least one department, this was referred to as “probable cause on four legs”. If judges continue to allow this quality of evidence, the police will continue to abuse the legal environment that creates.
nerdponx 5y ago
> Crucially, Williams' lawyers – public defenders Lisa Boughton and Brendan Max – said records showed that ShotSpotter actually initially picked up what sounded like a firework a mile away, and this was later reclassified by ShotSpotter staff to be a gunshot at the intersection where and when Williams was seen on camera. ShotSpotter strongly insisted it had not improperly altered any data to favor the police's case, and said that regardless of the initial real-time alert, its evidence of the gunshot was the result of follow-up forensic analysis, which was submitted to the courts. "Strongly insisted" links to: https://www.theregister.com/2021/08/02/nvidia_cuda_openai/ https://www.theregister.com/2021/08/02/nvidia_cuda_openai/ > One of the pieces of evidence against Williams claims ShotSpotter's sensors in Chicago identified gunfire where surveillance cameras had seen Williams stop his car by a south-side Chicago block, right when and where the cops said Herring was shot. > However, Williams’ lawyer submitted paperwork [PDF] claiming ShotSpotter actually detected a firework a mile away from that location, and that ShotSpotter later reclassified the bang as a gunshot and the location as being where Williams was seen on camera, Vice first reported. > Williams' lawyer demanded the court hold an inquiry into the ShotSpotter evidence, and the prosecutors simply withdrew it. > ShotSpotter responded by denying at length it improperly altered any data or evidence, and hit back at any suggestion it had done so to help the police make a case. It said its software generates real-time alerts automatically, and staff later analyze the microphone readings to submit forensic reports for the courts, and these final reports can therefore differ from the initial alerts. > "The idea that ShotSpotter 'alters' or 'fabricates' evidence in any way is an outrageous lie and would be a criminal offense," it said in a statement. "We follow the facts and data for our forensic analysis. Period." > Update: The case against Williams was dismissed by the judge at the request of the prosecution, which admitted it now had insufficient evidence. Williams had spent the best part of a year in jail awaiting trial.
- merrywhether 5y ago
  Wow, we need a civil rights lawyer to represent Williams in a wrongful imprisonment suit against the city, if not also a civil suit against ShotSpotter (or maybe even a criminal complaint for knowingly providing false testimony?). People shouldn’t be able to just dust their hands of this.
coldtea 5y ago
>So the AI didn’t even make the call. The staff did, manually. (a) The staff of the company who makes the AI, which would have an incentive to make the AI's original judgement appear accurate? (b) The staff of a company paid by the state/police, which aims to please its customers? (c) Was the staff trained/experts in recognizing such sounds, or were they random AI company employees, which just made a judgement call? If so, was this call presented as AI-supported to the trial, and was it sold as anything beyond "some devs, no gun experts, mind you, heard this recording and said this must be a gunshot".
- dylan604 5y ago
  It could come at the behest of police as in "it would be really helpful to our case if there was ShotSpotter evidence to support our cased". It could come from the fact that ShotSpotter is losing customers, and willing to do anything to make the product "feel" more valuable to the police to ensure cash cows keep producing. The modern criminal investigation has so much negative baggage attached, from historical shenianigans, that it's easy to find ways to question the prosecution's case.
emodendroket 5y ago
If manual reclassification is routine then what does that say about the AI?
- CrazyStat 5y ago
  That it doesn't give the answers the customers (police and prosecutors) want. Which is mostly unrelated to whether or not it's accurate.
  - emodendroket 5y ago
    I think it’s related. If it is good enough to put someone in jail there shouldn’t be any kind of need for manual overrides at all.
    - CrazyStat 5y ago
      "Good" in what sense? Police and prosecutors don't necessarily care about their evidence being good in the sense of accurate. It just needs to convince a jury. There is a very, very, very long history of police and prosecutors using bad (in the sense of accuracy) evidence to put people in jail.
      - emodendroket 5y ago
        I’m making a normative claim. Obviously it does not describe reality.
kbenson 5y ago
There was a submission here a few weeks back about ShitSpotter and how they seem willing to put in overrides at request of the police without necessarily having a basis for it. If that's what happened in this circumstances it would be, police check ShoySpotter for more evidence, ShotSpotter hasn't automatically marked anything, police request manual review for gunshot they are sure must be there, ShotSpotter manual review process comes up with something and marks it as a shot with a human override. There's a lot of places bias, or outright corruption, could cause problems in that process, if it's what happened here.
mola 5y ago
It's the AI companies to blame, always. Obviously the algorithm is without blame, it's not a blamelable entity.
FireBeyond 5y ago
What almost invariably happens in cases like this or the cell phone surveillance boxes (Stingrays) is that if the underlying technology or behavior is challenged or questioned strongly enough, the evidence introduced by them, if not the case itself, will be withdrawn, in order to try to avoid any negative precedent against the device or technology being set.
chacham15 5y ago
This is missing a key crucial insight: the overall number of evaluated cases increases when AI is helping meaning that borderline cases which may have been ignored prior are now being evaluated. Said another way, with AI handling the obvious cases, the number of borderline cases being evaluated by humans increases. With humans having large amounts of inaccuracy, borderline cases before which may have been ignored due to not having manpower to evaluate are now being evaluated. This can and does lead to many more erroneous accusations despite the fact that all the borderline evaluations are being handled by humans. Another factor here is that simply evaluating more data is also likely to lead to more erroneous evaluations since random chance has a much larger impact.
smsm42 5y ago
But this is so convenient! "Your honor, this is the AI algorithm that said it was a gunshot, AI can't lie and is based on pure Science!" Yeah, later it turns out people made that decision because police asked them to do it, but the defense may not know it.
jrochkind1 5y ago
It sounds like they reclassified it after the police said "Gee, are you sure you don't have a gunshot? in a slightly different location?" I wonder how much of "AI" introduced in court (or for a warrant) involves some element of individual human attention though... They just call it "AI" without qualification until they get caught (if even then). Would it be better though if it had been purely the output of a machine learning algorithm? (Which after all is not infallible). In that case, the guy probably would have been convicted on that alone...

jtbayly 5y ago

It may not be right to blame AI, but it's certainly the case that they were presenting it as "the system says there was a shot fired here." The bigger point is that the "system" can't be trusted. And the fact that they need to manually classify things is proof of that fact. (It's way more complicated than just an audio recording. It's recordings from numerous locations, attempting to triangulate the original location, without being confused by echoes and other things.)

PragmaticPulp 5y ago
> It may not be right to blame AI, but it's certainly the case that they were presenting it as "the system says there was a shot fired here." I think a crucial distinction is to clarify who “they” is: Who was making the claim? From the article, it seems that the prosecutors may have been the ones exaggerating the evidence: > Prosecutors said ShotSpotter picked up a gunshot sound where Williams was seen on surveillance camera footage in his car, putting it all forward as proof that Williams shot Herring right there and then. Naturally the defense wanted to see the actual evidence, so that’s what they got down to. I know everyone wants to vilify AI, but I think the actual bad actor in this article might be the prosecutor trying to misrepresent the evidence.
- Aerroon 5y ago
  >I think a crucial distinction is to clarify who “they” is: Who was making the claim? From the article, it seems that the prosecutors may have been the ones exaggerating the evidence: The prosecutors are the "they". The further removed you get from people who know the 'AI' the more the validity of the evidence provided by the AI relies merely on what others have said. The people working on the system know that it can be faulty. Other software developers have an idea that it could be faulty. Prosecutors are told that it's alright. They turn around and argue it in a court of law. And the further you get the less people have any idea that the system is faulty. Down the line people will go "the machine says so, therefore it must be so".
  - ClumsyPilot 5y ago
    >Down the line people will go "the machine says so, therefore it must be so". Thats the real feature of AI, devision laundering: a flawed decision made by someone for their own gain now appears impartial and trustworthy, and come wothout accountability
- jtbayly 5y ago
  Sure. But the point is that they misrepresent the evidence by pointing to the system as trustworthy. Even without tampering with the evidence the system isn’t trustworthy beyond a reasonable shadow of a doubt.

stefan_ 5y ago

That's the point? These systems invariably turn out to not be any sort of AI at all, but sales people ready to make a call any way their customer (police) wants it made. Later it's presented as AI to a jury. It's basically the same scam as lots of forensics, junk science to get a desired result.

giantg2 5y ago
I generally agree, but wouldn't necessarily say a lot of forensics is junk science. The science is generally sound (procedures and theory). Usually it's the prosecutors that present the forensics in a way that misrepresents what the results actually mean. Or lab techs make mistakes or perform misconduct (How to Fix a Drug Scandal on Netflix is interesting).
- wyager 5y ago
  Fingerprinting is junk science. Polygraphs are junk science. Tons of things that have been or are admissible as evidence are junk science.
  - giantg2 5y ago
    I wouldn't call polygraphs forensics. Most states do not allow them as evidence. Do you have a link for the fingerprinting claim?
    - monetus 5y ago
      (Not who you responded to) https://www.crammlawfirm.com/fingerprint-match-jails-innocent-man/ https://www.crammlawfirm.com/fingerprint-match-jails-innocen... This isn't the case I originally recalled, and doesn't say how many points were matched, but I remember a 5 point match being used to convict a person who was later proven innocent.
      - newswasboring 5y ago
        That shows its flawed, not junk.

helsinkiandrew 5y ago

It's scary that prosecutors thought that the ShotSpotter evidence is suitable for a court. It's amazing technology that can help police get to a crime scene quickly but is far too easy to spoof or get the wrong answer due to reflections and other sounds etc. Hopefully cases like this will eventually stop the system being used in court.

SamoyedFurFluff 5y ago
I mean, drug dogs are still a thing even though we know they tend to respond to owners desires regardless of if there’s drugs present.
- yawaworht1978 5y ago
  Wow i have never heard of that. Curious to see a link/article on the issue, if you can provide.
  - DaftDank 5y ago
    This is a decent summary of court cases related to the dogs: https://www.nlrg.com/criminal-law-legal-research/bid/84670/CRIMINAL-LAW-UPDATE-Reliability-of-Narcotics-Dogs-to-Be-Revisited-by-U-S-Supreme-Court https://www.nlrg.com/criminal-law-legal-research/bid/84670/C... I think this part is pretty key, and has always been a big open question in my mind: "The court further recognized a potential problem in assessing reliability related to handler cuing, as even well-trained dogs could respond to subconscious cues from their handlers. For this reason, the court determined that a critical factor in determining reliability is the record of false positive alerts made by the dog." The subconscious cues, or even deliberate cues that only the handler and dog know, could allow an officer with ill intentions to make some kind of signal only known to him and the dog, which causes the dog to make an "alert", giving the cop probable cause.
    - leephillips 5y ago
      Some departments deploy dogs that alert on all vehicles when they want an excuse to search. In at least one department, this was referred to as “probable cause on four legs”. If judges continue to allow this quality of evidence, the police will continue to abuse the legal environment that creates.

nerdponx 5y ago

> Crucially, Williams' lawyers – public defenders Lisa Boughton and Brendan Max – said records showed that ShotSpotter actually initially picked up what sounded like a firework a mile away, and this was later reclassified by ShotSpotter staff to be a gunshot at the intersection where and when Williams was seen on camera. ShotSpotter strongly insisted it had not improperly altered any data to favor the police's case, and said that regardless of the initial real-time alert, its evidence of the gunshot was the result of follow-up forensic analysis, which was submitted to the courts. "Strongly insisted" links to: https://www.theregister.com/2021/08/02/nvidia_cuda_openai/ https://www.theregister.com/2021/08/02/nvidia_cuda_openai/ > One of the pieces of evidence against Williams claims ShotSpotter's sensors in Chicago identified gunfire where surveillance cameras had seen Williams stop his car by a south-side Chicago block, right when and where the cops said Herring was shot. > However, Williams’ lawyer submitted paperwork [PDF] claiming ShotSpotter actually detected a firework a mile away from that location, and that ShotSpotter later reclassified the bang as a gunshot and the location as being where Williams was seen on camera, Vice first reported. > Williams' lawyer demanded the court hold an inquiry into the ShotSpotter evidence, and the prosecutors simply withdrew it. > ShotSpotter responded by denying at length it improperly altered any data or evidence, and hit back at any suggestion it had done so to help the police make a case. It said its software generates real-time alerts automatically, and staff later analyze the microphone readings to submit forensic reports for the courts, and these final reports can therefore differ from the initial alerts. > "The idea that ShotSpotter 'alters' or 'fabricates' evidence in any way is an outrageous lie and would be a criminal offense," it said in a statement. "We follow the facts and data for our forensic analysis. Period." > Update: The case against Williams was dismissed by the judge at the request of the prosecution, which admitted it now had insufficient evidence. Williams had spent the best part of a year in jail awaiting trial.

merrywhether 5y ago
Wow, we need a civil rights lawyer to represent Williams in a wrongful imprisonment suit against the city, if not also a civil suit against ShotSpotter (or maybe even a criminal complaint for knowingly providing false testimony?). People shouldn’t be able to just dust their hands of this.

coldtea 5y ago

>So the AI didn’t even make the call. The staff did, manually. (a) The staff of the company who makes the AI, which would have an incentive to make the AI's original judgement appear accurate? (b) The staff of a company paid by the state/police, which aims to please its customers? (c) Was the staff trained/experts in recognizing such sounds, or were they random AI company employees, which just made a judgement call? If so, was this call presented as AI-supported to the trial, and was it sold as anything beyond "some devs, no gun experts, mind you, heard this recording and said this must be a gunshot".

dylan604 5y ago
It could come at the behest of police as in "it would be really helpful to our case if there was ShotSpotter evidence to support our cased". It could come from the fact that ShotSpotter is losing customers, and willing to do anything to make the product "feel" more valuable to the police to ensure cash cows keep producing. The modern criminal investigation has so much negative baggage attached, from historical shenianigans, that it's easy to find ways to question the prosecution's case.

emodendroket 5y ago

If manual reclassification is routine then what does that say about the AI?

CrazyStat 5y ago
That it doesn't give the answers the customers (police and prosecutors) want. Which is mostly unrelated to whether or not it's accurate.
- emodendroket 5y ago
  I think it’s related. If it is good enough to put someone in jail there shouldn’t be any kind of need for manual overrides at all.
  - CrazyStat 5y ago
    "Good" in what sense? Police and prosecutors don't necessarily care about their evidence being good in the sense of accurate. It just needs to convince a jury. There is a very, very, very long history of police and prosecutors using bad (in the sense of accuracy) evidence to put people in jail.
    - emodendroket 5y ago
      I’m making a normative claim. Obviously it does not describe reality.

kbenson 5y ago

There was a submission here a few weeks back about ShitSpotter and how they seem willing to put in overrides at request of the police without necessarily having a basis for it. If that's what happened in this circumstances it would be, police check ShoySpotter for more evidence, ShotSpotter hasn't automatically marked anything, police request manual review for gunshot they are sure must be there, ShotSpotter manual review process comes up with something and marks it as a shot with a human override. There's a lot of places bias, or outright corruption, could cause problems in that process, if it's what happened here.

mola 5y ago

It's the AI companies to blame, always. Obviously the algorithm is without blame, it's not a blamelable entity.

FireBeyond 5y ago

What almost invariably happens in cases like this or the cell phone surveillance boxes (Stingrays) is that if the underlying technology or behavior is challenged or questioned strongly enough, the evidence introduced by them, if not the case itself, will be withdrawn, in order to try to avoid any negative precedent against the device or technology being set.

chacham15 5y ago

This is missing a key crucial insight: the overall number of evaluated cases increases when AI is helping meaning that borderline cases which may have been ignored prior are now being evaluated. Said another way, with AI handling the obvious cases, the number of borderline cases being evaluated by humans increases. With humans having large amounts of inaccuracy, borderline cases before which may have been ignored due to not having manpower to evaluate are now being evaluated. This can and does lead to many more erroneous accusations despite the fact that all the borderline evaluations are being handled by humans. Another factor here is that simply evaluating more data is also likely to lead to more erroneous evaluations since random chance has a much larger impact.

smsm42 5y ago

But this is so convenient! "Your honor, this is the AI algorithm that said it was a gunshot, AI can't lie and is based on pure Science!" Yeah, later it turns out people made that decision because police asked them to do it, but the defense may not know it.

jrochkind1 5y ago

It sounds like they reclassified it after the police said "Gee, are you sure you don't have a gunshot? in a slightly different location?" I wonder how much of "AI" introduced in court (or for a warrant) involves some element of individual human attention though... They just call it "AI" without qualification until they get caught (if even then). Would it be better though if it had been purely the output of a machine learning algorithm? (Which after all is not infallible). In that case, the guy probably would have been convicted on that alone...