3 ms·

This hardly needs to be said here, but there should have been human review of the AI output before taking any drastic action. That would (I assume, though since

by rmunn 10mo ago

This hardly needs to be said here, but there should have been human review of the AI output before taking any drastic action. That would (I assume, though since I can't read the original article I don't know if that assumption is correct) have immediately let them know that the alleged "gun" was nothing of the sort, and avoided the massive disruption of a totally unnecessary lockdown.

nucleardog 10mo ago
Not sure how much that would help overall. Unless it's completely clear that it's not a gun, the reviewer is essentially always going to pull the alarm. The risk of a false alarm is going to be seen as minimal, while the risk of a false negative is catastrophic. False alarm makes the news for now because it's novel, we all go "What the hell, guys?" and life goes on. Nobody wants to end up sitting in front of a prosecutor, the media, etc explaining why they chose not to pull the alarm, when the AI _clearly_ identified the gun, and instead chose to let all those kids die.
- jermaustin1 10mo ago
  One more instance of offloading blame to a computer system: "It's not my fault the cops shot the kid, the system said it was a gun." The only way this gets fixed is if there are consequences at every level for false positives.
  - WillAdams 10mo ago
    One approach for this is that the person who makes the call needs to be on-site and in the front of the situation --- similarly, a judge signing off on a No-Knock Warrant --- the judge needs to be at least be present, and should be required to walk through the building/home/apartment after the warrant is served. If it's not important/severe enough for a judge to do this, then I would argue that there's no need for the "no knock" aspect.
  - brk 10mo ago
    That sounds good on paper, but is really impossible to implement in any practical way. In this case, the kid was holding the clarinet like a weapon, and though we have not seen the actual video, the descriptions of it make it sound like overall resolution was poor. The alternative to the false positive here, is to not report anything that you cannot be 110% certain of, which means that you're likely to miss some true positives. Overall this situation mostly reads like everything worked as intended, and the press turned it into more than it needed to be. School shooting are a real thing, there is plenty of evidence of that. Weapons detection has become a necessary component of a school safety strategy. For many reasons, it is not practical to have personnel at the school, or within the district, act as the first-pass reviewer of AI detections of weapons.
    - Zigurd 10mo ago
      Don't be defeatist. The situation under consideration here is probably monitored by security cameras and body cams end to end. Everyone not following correct protocol did so on camera. Punishing willful ignorance and incompetence is certainly possible.
  - dpoloncsak 10mo ago
    >The only way this gets fixed is if there are consequences at every level for false positives. Do we really want consequences for false positives? If a kid is smoking a cigarette in the bathroom and the smoke detector goes off, the school should evacuate. The Smoke Alarm went off. No principal is going to sign off on the assumption that "Timmy is smoking, it's not a real fire". The principal shouldn't be punished for responding to the alarm. Timmy...probably should get reprimanded, but that feels off-metaphor. In the example we are given, Timmy did nothing wrong. Having a clarinet is not contraband, and he should not be punished. The admin who called a lockdown did nothing wrong, as they were responding to the system in the way they were trained to use it. This is all in the name of safety, where things are done in 'an abundance of caution'. >"It's not my fault the cops shot the kid, the system said it was a gun." No, its the cop's fault. The cop hasn't been trained to use the AI security system, and is instead given their own SOP for assessing threats.
- LexGray 10mo ago
  Make the failure billable for failures in the contract as it is a significant downtime. For every false alarm you need to pay the salaries that were wasted and the snacks and therapists for the kids. Likewise for every missed gun hazard pay for teachers and therapists for kids. If they aren’t confident enough to back a service that has such a mental impact on failure they should not be selling it.
htek 10mo ago
This is just conjecture, but I suspect there will be as much review of photos, application of good investigative work and overall professionalism as is conducted during anonymous, virtually untraceable Swatting incidents that terrify the victims, if not get them killed.
j-conn 10mo ago
There was. From the article: ''' ZeroEyes said that trained employees review alerts before they are sent and that its software can make a lifesaving difference in averting mass shootings by alerting law enforcement to weapons on campus within seconds. At Lawton Chiles, the student flagged by ZeroEyes was holding his musical instrument like a rifle, co-founder Sam Alaimo told The Washington Post. “We don’t think we made an error, nor does the school,” Alaimo said. “That was better to dispatch [police] than not dispatch.” '''
brk 10mo ago
This appears to be a ZeroEyes customer, which means there was a human in the loop. One of ZeroEyes' main selling points is their use of 24/7 staffed SOCs to review anything the AI software flags before sending an actual alert to the site/customer.
Zigurd 10mo ago
Nearly all police responses in the US are armed responses. Americans don't realize how dangerous that is. They recognize swatting as a crime, but don't realize that a Karen who lies to police in order to get them to bully someone they don't like is very dangerous. Comparable to aggressive driving and road rage. It needs to be prosecuted more often. By more often I mean more than zero.

by rmunn 10mo ago

nucleardog 10mo ago

Not sure how much that would help overall. Unless it's completely clear that it's not a gun, the reviewer is essentially always going to pull the alarm. The risk of a false alarm is going to be seen as minimal, while the risk of a false negative is catastrophic. False alarm makes the news for now because it's novel, we all go "What the hell, guys?" and life goes on. Nobody wants to end up sitting in front of a prosecutor, the media, etc explaining why they chose not to pull the alarm, when the AI _clearly_ identified the gun, and instead chose to let all those kids die.

jermaustin1 10mo ago
One more instance of offloading blame to a computer system: "It's not my fault the cops shot the kid, the system said it was a gun." The only way this gets fixed is if there are consequences at every level for false positives.
- WillAdams 10mo ago
  One approach for this is that the person who makes the call needs to be on-site and in the front of the situation --- similarly, a judge signing off on a No-Knock Warrant --- the judge needs to be at least be present, and should be required to walk through the building/home/apartment after the warrant is served. If it's not important/severe enough for a judge to do this, then I would argue that there's no need for the "no knock" aspect.
- brk 10mo ago
  That sounds good on paper, but is really impossible to implement in any practical way. In this case, the kid was holding the clarinet like a weapon, and though we have not seen the actual video, the descriptions of it make it sound like overall resolution was poor. The alternative to the false positive here, is to not report anything that you cannot be 110% certain of, which means that you're likely to miss some true positives. Overall this situation mostly reads like everything worked as intended, and the press turned it into more than it needed to be. School shooting are a real thing, there is plenty of evidence of that. Weapons detection has become a necessary component of a school safety strategy. For many reasons, it is not practical to have personnel at the school, or within the district, act as the first-pass reviewer of AI detections of weapons.
  - Zigurd 10mo ago
    Don't be defeatist. The situation under consideration here is probably monitored by security cameras and body cams end to end. Everyone not following correct protocol did so on camera. Punishing willful ignorance and incompetence is certainly possible.
- dpoloncsak 10mo ago
  >The only way this gets fixed is if there are consequences at every level for false positives. Do we really want consequences for false positives? If a kid is smoking a cigarette in the bathroom and the smoke detector goes off, the school should evacuate. The Smoke Alarm went off. No principal is going to sign off on the assumption that "Timmy is smoking, it's not a real fire". The principal shouldn't be punished for responding to the alarm. Timmy...probably should get reprimanded, but that feels off-metaphor. In the example we are given, Timmy did nothing wrong. Having a clarinet is not contraband, and he should not be punished. The admin who called a lockdown did nothing wrong, as they were responding to the system in the way they were trained to use it. This is all in the name of safety, where things are done in 'an abundance of caution'. >"It's not my fault the cops shot the kid, the system said it was a gun." No, its the cop's fault. The cop hasn't been trained to use the AI security system, and is instead given their own SOP for assessing threats.
LexGray 10mo ago
Make the failure billable for failures in the contract as it is a significant downtime. For every false alarm you need to pay the salaries that were wasted and the snacks and therapists for the kids. Likewise for every missed gun hazard pay for teachers and therapists for kids. If they aren’t confident enough to back a service that has such a mental impact on failure they should not be selling it.

htek 10mo ago

This is just conjecture, but I suspect there will be as much review of photos, application of good investigative work and overall professionalism as is conducted during anonymous, virtually untraceable Swatting incidents that terrify the victims, if not get them killed.

j-conn 10mo ago

There was. From the article: ''' ZeroEyes said that trained employees review alerts before they are sent and that its software can make a lifesaving difference in averting mass shootings by alerting law enforcement to weapons on campus within seconds. At Lawton Chiles, the student flagged by ZeroEyes was holding his musical instrument like a rifle, co-founder Sam Alaimo told The Washington Post. “We don’t think we made an error, nor does the school,” Alaimo said. “That was better to dispatch [police] than not dispatch.” '''

brk 10mo ago

This appears to be a ZeroEyes customer, which means there was a human in the loop. One of ZeroEyes' main selling points is their use of 24/7 staffed SOCs to review anything the AI software flags before sending an actual alert to the site/customer.

Zigurd 10mo ago

Nearly all police responses in the US are armed responses. Americans don't realize how dangerous that is. They recognize swatting as a crime, but don't realize that a Karen who lies to police in order to get them to bully someone they don't like is very dangerous. Comparable to aggressive driving and road rage. It needs to be prosecuted more often. By more often I mean more than zero.