17 ms·

Armed police swarm student after AI mistakes bag of Doritos for a weapon

• kirykl 1y ago
  Wouldn’t have thought AI assessment of security image is enough for probable cause
• nickdothutton 1y ago
  "Omnilert Gun Detect delivers instant gun detection, near-zero false positives".
  • dgacmu 11mo ago
    If it's taking images every 30 seconds, it's getting 86400 x 30 = 2.5 million images per day per camera. So when it causes enormous, unnecessary trauma to one student per week, the company can rightfully claim it has less than a 1 in 10 million false positive rate. (* see also "how to lie with statistics").
• tencentshill 1y ago
  "rapid human verification." at gunpoint. The Torment Nexus has nothing on these AI startups.
  • palmotea 1y ago
    Why did they waste time verifying? The police should have eliminated the threat before any harm could be done. Seconds count when you're keeping people safe.
    • anigbrowl 11mo ago
      I get that you're being sarcastic and find the police response appalling, but the sad reality of Poe's Law is that there are a lot of people who would unironically say this and would have cheered if the cops had shot this kid, either because they hate black people or because they get off on violence and police shootings are a social sanctioned way to indulge that taste.
    • vee-kay 11mo ago
      We all know the cops will go for the easy prey: * Even hundreds of cops in full body armor and armed with automatic guns will not dare to engage a single "lone wolf" shooter doing a killing spree in a school; the heartless cowards may even prevent the parents from going inside to rescue their kids: Uvalde school shooting incident * Cop on a ego trip, will shoot down a clearly harmless kid calmly eating a burger in his own car (not a stolen car): Erik Cantu incident * Cops are not there to serve the society, they are not there to ensure safety and peace for the neighborhood, they are merely armed militia to protect the rich and powerful elites: https://www.alternet.org/2022/06/supreme-court-cops-protect-individuals https://www.alternet.org/2022/06/supreme-court-cops-protect-...
      • akimbostrawman 11mo ago
        All of your examples are well known not because its normal and accepted but because they are exceptions. For every one bad example there are thousand good ones, that's humans for you. Doesn't mean they are perfect or shouldn't criticised but claiming that's all they are doing isn't reasonable either. If you look at actual per capita statistics you will easily see this.
        • vee-kay 11mo ago
          The statistics are indeed quite telling.. and chilling! https://en.wikipedia.org/wiki/Lists_of_killings_by_law_enforcement_officers_in_the_United_States https://en.wikipedia.org/wiki/Lists_of_killings_by_law_enfor... In the United States, law enforcement officers shoot and kill more than 1,100 civilians each year, with a significant number of these incidents involving unarmed individuals, particularly among Black Americans who are disproportionately affected. The FBI has begun collecting data on these use-of-force incidents to provide better insights into the circumstances surrounding police shootings. Police killed more than 1,300 people in the U.S. last year, an estimated 0.3% increase in police killings per million people. The increase makes 2024 the deadliest year for police violence by a slim margin since Mapping Police Violence began tracking civilian deaths more than a decade ago. https://www.usatoday.com/story/news/nation/2025/02/26/police-killings-2024-data/80281722007/ https://www.usatoday.com/story/news/nation/2025/02/26/police... There is no national database that documents police killings in the U.S., and the report comes days after the Justice Department removed a database tracking misconduct by federal law enforcement. Researchers spent thousands of hours analyzing more than 100,000 media reports to compile the Mapping Police Violence database. https://en.wikipedia.org/wiki/List_of_mass_shootings_in_the_United_States https://en.wikipedia.org/wiki/List_of_mass_shootings_in_the_... In 2025, the U.S. has experienced significant gun violence, with 11,197 shooting deaths reported through September 30, along with 20,425 nonfatal injuries. The year has seen a total of 341 mass shootings, resulting in 331 fatalities and 1,499 injuries. Shootings have happened in all 50 states, at all times of day, and in locations as varied as schools, gas stations, gyms, Walmarts, and homes. Some involved handguns, others rifles or shotguns. 10.3 million guns have been sold across the U.S. in 2025 through September 30. https://www.thetrace.org/2025/10/shooting-gun-violence-data-america-q3/ https://www.thetrace.org/2025/10/shooting-gun-violence-data-... Mass shootings in the United States are incidents where one or more individuals use firearms to kill or injure multiple people, typically in public settings. The frequency and definitions of these events can vary, but they have been a significant concern in recent years, with the U.S. experiencing more mass shootings than any other country. GVA has recorded 325 mass shootings in the U.S. this year through three quarters. Those have resulted in 309 deaths and 1,490 injuries. Mass shootings in the last quarter included the high-profile shooting at a New York skyscraper, as well as the shooting of 29 people, 26 of them children, at a church in Minneapolis. Two children, aged 8 and 10, were killed in that incident.
  • drak0n1c 1y ago
    The dispatch relayer and responding officers should at least have ready access to a screen where they can see a video/image of the raw footage that triggered the AI alert. If it is a false alarm, they will better see it and react accordingly, and if it is a real threat they will better understand the initial context and who may have been involved.
    • ggreer 11mo ago
      According to a news article[1], a human did review the video/image and flagged it as a false positive. It was the principal who told the school cop, who then called other cops: > The Department of School Safety and Security quickly reviewed and canceled the initial alert after confirming there was no weapon. I contacted our school resource officer (SRO) and reported the matter to him, and he contacted the local precinct for additional support. Police officers responded to the school, searched the individual and quickly confirmed that they were not in possession of any weapons. What's unclear to me is the information flow. If the Department of School Safety and Security recognized it as a false positive, why did the principal alert the school resource officer? And what sort of telephone game happened to cause local police to believe the student was likely armed? 1. https://www.wbaltv.com/article/student-handcuffed-ai-system-mistook-bag-chips-weapon/69114601 https://www.wbaltv.com/article/student-handcuffed-ai-system-...
      • wat10000 11mo ago
        Sounds like a "better safe than sorry" approach. If you ignore the alert on the basis that it's a false positive, then it turns out it really was a gun and the person shoots somebody, you're going to get sued into the ground, fired, name plastered all over the media, etc. On the other hand, if you call in the cops and there wasn't a gun, you're fine.
        • spankibalt 11mo ago
          > "On the other hand, if you call in the cops and there wasn't a gun, you're fine." Yeah, cause cops have never shot somebody unarmed. And you can bet your ass that the possible follow-up lawsuit to such a debacle's got "your" name on it.
  • Etheryte 11mo ago
    Next up, a captcha that verifies you're not a robot by swatting you and checking at gunpoint.
• duxup 1y ago
  >Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” It's ok everyone, you're safer now that police are pointing a gun at you, because of a bag of chips ... just to be safe. /s Absolutely ridiculous. We're living "computer said you did it, prove otherwise, at gunpoint".
  • cranberryturkey 1y ago
    gustapo
  • walkabout 1y ago
    We got our cyberpunk future, except none of it's cool and everything's extremely stupid.
    • duxup 1y ago
      They could at least thrown in some good music and cute girls with colored hair to make us feel better :(
      • irilesscent 1y ago
        You get grok Lady for the latter.
      • forgetfulness 11mo ago
        I’ve got great news for you: there are more girls with colored hair than ever before, and we got the Synthwave revival, just try to find the right crowd and put on Timecop1983 in your headphones Cyberpunk always sucked for all but the corrupt corporate, political and military elites, so it’s all tracking
        • surgical_fire 11mo ago
          > there are more girls with colored hair than ever before The ones I see don't tend to lean cute.
• rolph 1y ago
  Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” prioritize your own safety by not attending any location fitted with such a system, or deemed to be such a dangerous environment that such a system is desired. the AI "swatted" someone.
  • etothet 1y ago
    The corporate version of "It's a feature, not a bug."
  • bilbo0s 1y ago
    Calling it today. This company is going to get innocent kids killed. How many packs of Twizzlers, or Doritos, or Snickers bars are out there in our schools? First time it happens, there will be an explosion of protests. Especially now that the public knows that the system isn't working but the authorities kept using it anyway. This is a really bad idea right now. The technology is just not there yet.
    • mothballed 11mo ago
      And then there's plenty of bullies who might put a sticker of a picture of a gun on someone's back, knowing it will set off the image recognition. It's only a matter of time until they figure that out.
      • withinboredom 11mo ago
        When I was a kid, we made rubber-band guns all the time. I’m sure that would set it off too.
      • xp84 11mo ago
        That's a great and terrifying idea. When that inevitably happens, you'll then have a couple of 13-year-olds: one dead, and one shell-shocked kid in disbelief that a stupid prank idea he cooked up in 60 seconds is now claimed as the root cause why someone was killed. That one may be charged with a crime or sued, though the district who installed this idiotic thing is really to blame.
• ben_w 1y ago
  Memories of Jean Charles de Menezes come to mind: https://en.wikipedia.org/wiki/Killing_of_Jean_Charles_de_Menezes https://en.wikipedia.org/wiki/Killing_of_Jean_Charles_de_Men...
  • Gibbon1 1y ago
    That was my first thought as well. A worry is police officers make mistakes which leads to anywhere from hapless people getting terrorized, harmed or killed. The bad thing about AI is it'll allow police to escape responsibility. Perhaps also where a human will realize it made a mistake they can admit it and everything is okay. But if AI says you had a gun, it won't walk that back. AI said he had a gun. But when we checked, he didn't have it anymore. In the Menezes case the cops were playing a game of telephone that ended up with him being shot in the head.
  • dredmorbius 1y ago
    And Amadou Diallo: <https://en.wikipedia.org/wiki/Killing_of_Amadou_Diallo https://en.wikipedia.org/wiki/Killing_of_Amadou_Diallo>
• proee 1y ago
  Walking through TSA scanners, I always get that unnerving feeling I will get pulled aside. 50% of the time they flag my cargo pants because of the zipper pockets - There is nothing in them but the scanner doesn't like them. Now we get the privilege of walking by AI security cameras placed in random locations, hoping they don't flag us. There's a ton of money to be made with this kind of global frisking, so lots of pressure to roll out more and more systems. How does this not spiral out of control?
  • walkabout 1y ago
    I already adjust my clothing choices when flying to account for TSA's security theater make-work bullshit. Wonder how long before I'm doing that when preparing to go to other public places. (I suppose if I attended pro sports games or large concerts, I'd be doing it for those, too)
    • hinkley 11mo ago
      I was getting pulled out of line in the 90’s for having long hair. I don’t dress in shitty clothes or fancy ones, I didn’t look funny, just the hair, which got regular compliments from women. I started looking at people trying to decide who looked juicy to the security folks and getting in line behind them. They can’t harass two people in rapid succession. Or at least not back then. The one I felt most guilty about, much later, was a filipino woman with a Philippine passport. Traveling alone. Flying to Asia (super suspect!). I don’t know why I thought they would tag her, but they did. I don’t fly well and more stress just escalates things, so anything that makes my day tiny bit less shitty and isn’t rude I’m going to do. But probably her day would have been better for not getting searched than mine was.
  • dheera 1y ago
    The TSA scanners also trigger easily on crotch sweat.
    • hsbauauvhabzb 1y ago
      I enjoy a good grope, so I’ll keep that in mind the next time I’m heading into the us.
• leptons 1y ago
  This is only the beginning of AI-hallucinated policing. Not a good start, and I don't think it's going to end well for citizens.
  • 4ndrewl 1y ago
    "end well for citizens." That ship has long sailed buddy.
    • throwaway173738 1y ago
      Yeah ask all those citizens getting “detained” by ICE how it worked out for them.
• phkahler 1y ago
  >> Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” No. If you're investigating someone and have existing reason to believe they are armed then this kind of false positive might be prioritizing safety. But in a general surveillance of a public place, IMHO you need to prioritize accuracy since false positives are very bad. This kid was one itchy trigger-pull away from death over nothing - that's not erring on the side of safety. You don't have to catch every criminal by putting everyone under a microscope, you should be catching the blatantly obvious ones at scale though.
  • blueflow 1y ago
    The perceived threat of government forces assaulting and potentially killing me for reasons i have no control over, this is the kind of stuff that terminates the social contract. I'd want a new state that protects me from such stuff.
• Havoc 1y ago
  >the system “functioned as intended,” Behold - a real life example of a "Not a hotdog" system, except this one is gun / not-a-gun. Except the fictional one from the series was more accurate...
• idontwantthis 1y ago
  If these AI video based gun detectors are not a massive fraud I will eat one. How on Earth does a person walk with a concealed gun? What does a woman in a skirt with one taped to her thigh walk like? What does a man in a bulky sweatshirt with a pistol on his back walk like? What does a teenager in wide legged cargo jeans with two pistols and a extra magazines walk like?
  • VTimofeenko 1y ago
    The brochure linked from TFA has a screenshot of a combination of segmentation and object recognition models which are fairly standard in NVRs. Quick skim of the vendor website seems to confirm this[1] and states a claim that they are not analyzing the gait. [1]: https://www.omnilert.com/blog/what-is-visual-gun-detection-technology-training-and-impact https://www.omnilert.com/blog/what-is-visual-gun-detection-t...
  • walkabout 1y ago
    The whole idea even accepting the core premise is OK to begin with needs to have a similar analysis applied to it that medical tests do: will there be enough false positives, with enough harm caused by them, that this is actually worse than doing nothing? Compared with likelihood of improving an outcome and how bad a failure to intervene is on average, of course. Given that there's no relevant screening step here and it's just being applied to everyone who happens to be at a place it's truly incredible that such an analysis would shake out in favor of this tech. The false positive rate would have to be vanishingly tiny, and it's simply not plausible that's true. And that would have to be coupled with a pretty low false negative rate, or you'd need an even lower false positive rate to make up for how little good it's doing even when it's not false-positiving. So I'm sure that analysis was either deliberately never performed, or was and then was ignored and not publicized. So, yes, it's a fraud. (There's also the fact that as soon as these are known to be present, they'll have little or no effect on the very worst events involving firearms at schools—shooters would just avoid any scheme that involved loitering around with a firearm where the cameras can see them, and count on starting things very soon after arriving—like, once you factor in second order effects, too, there's just no hope for these standing up to real scrutiny)
  • 11mo ago
• gdulli 1y ago
  The only way we could have foreseen this was immediately.
• AuthAuth 1y ago
  What is happening in the world. There should be some liability for this but nothing will happen.
  • coldpie 1y ago
    [flagged]
  • SoftTalker 1y ago
    Not sure nothing will happen. Some trial lawyers would love to sue a city, a school system, and an AI surveillance company over "trauma, anxiety, and mental pain and suffering" caused by the incident. There will probably be a settlement that nobody ever hears about.
    • wat10000 11mo ago
      A settlement paid by the taxpayers with no impact at all on anyone actually responsible.
  • cool_man_bob 1y ago
    [flagged]
    • jmcgough 1y ago
      The people building these things are good friends with the bullies and scammers now.
  • tamimio 1y ago
    Law enforcement officers, judicial officials, social workers, and similar generally maintain qualified immunity from liability in the course of their work. This case for example in which judges and social workers allegedly failed to properly assess a mother's fitness for child custody despite repeated indicators suggesting otherwise. The child was ultimately placed in the mother's care, and later was killed in an execution style (not due to negligence). https://www.youtube.com/watch?v=wzybp0G1hFE https://www.youtube.com/watch?v=wzybp0G1hFE
• j45 1y ago
  Sad for the student. Imagine the head scratching that's going on with execs who are surprised things might work when a probabilistic software is being used for deterministic purposes without realizing there's a gap between it kind of by nature.
  • doublerabbit 1y ago
    > Imagine the head scratching that's going on with execs I can't. The execs won't care and probably in their sadist ways, cheer.
    • j45 11mo ago
      Fair. Only a matter of time until it's big enough that it can't be avoided.
  • kelnos 11mo ago
    I'm sure there will be no head scratching. They already know that this can happen, and don't care, because they know that if someone gets killed because of it, they won't be held responsible. And may not even lose any customers.
• 6stringmerc 1y ago
  Feed the same system an image of an Asian kid and it will think the bag of chips is a calculator /s Or am I kidding? AI is only as good as its training and humans are...not bastions of integrity...
  • AndrewKemendo 1y ago
    Using humans for training gurantees bad outcomes because humans cannot demonstrate sociality at the same scale as antisociality.
  • kelnos 11mo ago
    I think it's almost guaranteed that this model has race-related biases, so no, I don't think you're kidding at all. I think it's entirely likely that an Asian (or white) kid of the same build, wearing the same clothes, with a crumpled-up bag of Doritos in his pocket, would not get flagged as having a gun.
• mentalgear 1y ago
  Ah, the coming age of Palantir's all seeing platform; and Peter Thiel becoming the shadow Emperor. Too bad non-deterministic ml systems are prone to errors that risk lives when applied wrongly to crucial parts of society. But in an authoritarian state, those will be hidden away anyway, so there's nothing to see here: move along folks. Yes, surveillance and authoritarianism go hand in hand, ask China. It's important to protest these methods and push lawmakers to act against them; now, before it's too late.
  • jason-phillips 1y ago
    [flagged]
    • mentalgear 1y ago
      Pre-emptive compliance out of fear - then my boy, the war is already lost.
      • polotics 1y ago
        "competition is for losers"... right? ;-)
      • jason-phillips 1y ago
        [flagged]
        • Forgeties79 1y ago
          > it just doesn’t impact me It didn’t impact these people before the Dorito incident either.
        • hypeatei 1y ago
          So you just live a reactionary life? Nothing matters until it affects you personally? Should we get rid of free speech if jason-phillips doesn't have anything to say?
• macintux 1y ago
  I think the most amazing part is that the school doubled down on the mistake by parroting the corporate line. I expect a school to be smart enough to say “Yes, this is a terrible situation, and we’re taking a closer look at the risks involved here.”
  • JKCalhoun 1y ago
    Lawyer's advice?
    • macintux 1y ago
      I would think "no comment" would be safer/smarter than "yeah, your kids are at risk of being shot by police by attending our school, deal with it".
      • JKCalhoun 11mo ago
        Good point.
  • lisbbb 11mo ago
    Except they never say that.
    • macintux 11mo ago
      > Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” > Baltimore County Public Schools echoed the company’s statement in a letter to parents, offering counseling services to students impacted by the incident. (Emphasis mine)
• malux85 1y ago
  Poor kid, and what an incompetent police department not to use their own judgement …… But …… Doritos should definitely use this as an advertisement, Doritos - The only weapon of mass deliciousness, or something like that And of course pay the kid, so something positive came come out of the experience for him
  • rkomorn 1y ago
    [flagged]
    • tartoran 1y ago
      [flagged]
      • Dilettante_ 11mo ago
        "Stop resisting...the flavor"
        • rkomorn 11mo ago
          Man. People did not like this bit of thread.
          • sebastiennight 11mo ago
            To be fair, it was in poor taste. And against the general HN policy. I'm proud of my bad pun (in the best French tradition of never letting a bon mot go to waste), but not proud of posting it.
  • sebastiennight 11mo ago
    The snack you'll only SWAT from my cold dead hands
• fritzo 1y ago
  > “They didn’t apologize. They just told me it was protocol. I was expecting at least somebody to talk to me about it.” I wonder how effective an apology and explanation would have been? Just some respect.
  • cool_man_bob 1y ago
    Effective at what? No one is facing any consequences anyway.
    • throwaway173738 1y ago
      Except for the kids who experienced the “rapid human verification” firsthand.
      • mothballed 1y ago
        Not a bad point, but a fake apology is worse than none.
        • eastbound 1y ago
          Maybe an apology from the AI?
    • hn_go_brrrrr 1y ago
      More's the pity. The school district could use some consequences.
  • chasd00 11mo ago
    The school admin has no understanding of the tech and only the dimmest comprehension of what happened. Asking them to do anything besides what the tech company told them to do is asking wayyy too much.
• hsbauauvhabzb 1y ago
  I would be certainly curious to test ethnicity with this system. Will white students with a bag of Doritos be flagged, or only if they’re black?
  • 12_throw_away 11mo ago
    Exactly. I wonder if this a purpose-built image-recognition system, or is it a lowest-possible effort generic image model trained on the internet? Classifying a Black high school student holding Doritos as an imminent shooting threat certainly suggests the latter.
• more_corn 1y ago
  Wait… AI hallucinated and the police overreacted to a black kid who actually posed no threat? I thought those two things were impossible?
• jmcgough 1y ago
  The guidance counselor does not have the training or time to "fix" the trauma you just gave this kid and his friends. Insane to put minors through this.
• froobius 1y ago
  Stuff like this feels like some company has managed to monetize an open source object detection model like YOLO [1], creating something that could be cobbled together relatively easily, and then sold it as advance AI capabilities. (You'd hope they've have at least fine-tuned it / have a good training dataset.) We've got a model out there now that we've just seen has put someone's life at risk... Does anyone apart from that company actually know how accurate it is? What it's been trained on? Its false positive rate? If we are going to start rolling out stuff like this, should it not be mandatory for stats / figures to be published? For us to know more about the model, and what it was trained on? [1] https://arxiv.org/abs/1506.02640 https://arxiv.org/abs/1506.02640
  • dfxm12 1y ago
    [flagged]
  • EdwardDiego 11mo ago
    And it feels like they missed the "human in the loop" bit. One day this company is likely to find itself on the end of a wrongful death lawsuit.
    • an0malous 11mo ago
      They’ll likely still be profitable after accounting for those. This is why sociopaths are so successful at business
• deleted 1y ago
  [deleted]
• vezycash 1y ago
  With high level of hallucination, cops need to tranquilizers more. If the student had reached for his bag just before the cops arrived, BLM 2.0 would have started.
• throw7 1y ago
  If false positives are known to happen, then you design a system where the image is vetted before telling the cops the perpetrator is armed. The company is basically swatting, but I'm sure they'll never be held liable.
  • lisbbb 11mo ago
    Actually, if a system has too many false positives or false negatives, it's basically useless. There will eventually be doubts amongst the operators of it and the whole thing will implode, which is the best possible outcome. We already went through this years ago with all those terrorism databases and we (humanity) have learned nothing--any database will have a percentage of erroneous data, it is impossible to eliminate erroneous data completely. Therefore, any database used to identify <fill in the blank> will have erroneous conclusions. It's been observed over and over again and governments can't help themselves that "this time it will be different because <fill in the blank> e.g. AI.
• shaky-carrousel 1y ago
  He could have been easily murdered. It's not the first time by a far margin that a bunch of overzealous cops murder a kid. I would never ever in my life set foot in a place that sends me armed cops so easily. That school is extremely dangerous.
  • ggreer 11mo ago
    I think the reason the school bought this silly software is because it's a dangerous school, and they're grasping at straws to try and fix the problem. The day after this false positive, a student was robbed.[1] Last month, a softball coach was charged with rape and possession of child pornography.[2] Last summer, one student was stabbed while getting off the bus.[3] Last year, there were two incidents where classmates stabbed each other.[4][5] 1. https://www.nottinghammd.com/2025/10/22/student-robbed-outside-kenwood-high-school-one-suspect-apprehended/ https://www.nottinghammd.com/2025/10/22/student-robbed-outsi... 2. https://www.si.com/high-school/maryland/baltimore-county-high-school-softball-coach-charged-with-rape-01k4qker1xpc https://www.si.com/high-school/maryland/baltimore-county-hig... 3. https://www.wbaltv.com/article/knife-assault-rossville-juvenile-in-custody/65501227 https://www.wbaltv.com/article/knife-assault-rossville-juven... 4. https://www.wbal.com/stabbing-incident-near-kenwood-high-school-under-investigation https://www.wbal.com/stabbing-incident-near-kenwood-high-sch... 5. https://www.cbsnews.com/baltimore/news/teen-injured-after-reported-disturbance-at-kenwood-high-school-in-baltimore-county/ https://www.cbsnews.com/baltimore/news/teen-injured-after-re...
    • evanelias 11mo ago
      That certainly sounds bad, but it's all relative; keep in mind this school is in Baltimore County, which is distinct from the City of Baltimore and has a much different crime profile. This school is in the exact same town as Eastern Tech, literally the top high school in Maryland.
      • boneitis 11mo ago
        Hi, I'm not following the point being made. I skimmed through all the articles linked in GP and finding them pretty relevant to whatever decision might have been made to utilize the AI system (not at all to comment on how badly the bad tip was acted on). Hailing from and still living in N. California, you could tell me that this school is located in Beverly Hills or Melrose Place, and it would still strike me as a piece of trivia. If anything, it'd just be ironic?
• tartoran 1y ago
  "“We understand how upsetting this was for the individual that was searched as well as the other students who witnessed the incident,” the principal wrote. “Our counselors will provide direct support to the students who were involved.”" Make them pay money for false positives instead of direct support and counselling. This technology is not ready for production, it should be in a lab not in public buildings such as schools.
  • akoboldfrying 11mo ago
    > Make them pay money for false positives instead of direct support and counselling. Agreed. > This technology is not ready for production No one wants stuff like this to happen, but nearly all technologies have risks. I don't consider that a single false positive outweighs all of its benefits; it would depend on the rates of false and true positives, and the (subjective) value of each (both high in this case, though I'd say preventing 1 shooting is unequivocally of more value than preventing 1 innocent person being unnecessarily held at gunpoint).
    • Zigurd 11mo ago
      In the US cops kill more people than terrorists. As long as you quantifying values take that into account.
      • akoboldfrying 11mo ago
        I get that people are uncomfortable with explicit quantification of stuff like this, but removing the explicitness doesn't remove the quantification, it just makes it implicit. If, say, we allow people to drive cars even though car accidents kill n people each year, then we are implicitly quantifying that the value of the extra productivity society gets by being able to get places quickly in a car is worth the deaths of those people. In your example, if terrorists were the only people killing people in the US, and police (a) were the only means of stopping them, and (b) did not benefit society in any other way, the equation would be simple: get rid of the police. There wouldn't need to be any value judgements, because everything cancels out. But in practice it's not that easy, since the vast majority of killings in the US occur at the hands of people who are neither police nor terrorists, and police play a role in reducing those killings too.
• neilv 1y ago
  > Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” It just created a situation in which a bunch of people with guns were told that some teen had a gun. That's a very unsafe situation that the system created, out of nothing. And some teen may be traumatized. Again, unsafe. Incidentally, the article's quotes make this teen sound more adult than anyone who sold or purchased this technology product.
  • bilbo0s 11mo ago
    >And some teen may be traumatized. Um. That's not really the danger here. The danger is that it's as clear as day that in the future someone is gonna be killed. That's not just a possibility. It's a certainty given the way the system is set up. This tech is not supposed to be used in this fashion. It's not ready.
    • wat10000 11mo ago
      I fully agree, but we also really need to get to a place where drawing the attention of police isn't an axiomatically life-threatening situation.
      • Zigurd 11mo ago
        If the US wasn't psychotic, not all police would have to be armed, and not every police response would be an armed response.
        • wlesieutre 11mo ago
          Even if not all police were armed, the response to "AI said someone has a gun" would always be the armed police
          • georgefrowny 11mo ago
            Why would it not be "human reviews the image that the AI said was a gun"?
• jawns 11mo ago
  I expect that John Bryan -- who produces content as The Civil Rights Lawyer https://thecivilrightslawyer.com https://thecivilrightslawyer.com -- will have something to say about it. He frequently criticizes the legality of police holding people at gunpoint based on flimsy information, because the law considers it a use of force, which needs to be justified. For instance, he's done several videos about "high risk" vehicle stops, where a car is mistakenly flagged as stolen, and the occupants are forced to the ground at gunpoint. My take is that if both the AI automated system and a "human in the loop" police officer looked at the picture and reasonably believed that the object in the image was a gun, then the stop might have been justified. But if the automated system just sent the officers out without having them review the image beforehand, that's much less reasonable justification.
  • cyanydeez 11mo ago
    Someday there'll be a lawyer in court telling us how strong the AI evidence was because companies are spending billions of dollars on it.
    • mothballed 11mo ago
      Or they'll tell us police have started shooting because an acorn falls, so they shouldn't be expected to be held to higher standards and are possibly an improvement.
    • bluGill 11mo ago
      And there needs to be an opposing lawyer ready to tear that argument to pieces.
    • Terr_ 11mo ago
      You mean in the same fallacious sense of "you can tell cigarettes are good because so many people buy them"?
      • simulator5g 11mo ago
        That sort of rhetoric works very well unfortunately.
• whycome 11mo ago
  Can someone write the novel “Computer says die”
  • deleted 11mo ago
    [deleted]
• mchannon 11mo ago
  In 1987, Paul Verhoeven predicted exactly this in the original Robocop. ED-209 mistakenly viewed a young man as armed, blows him away in the corporate boardroom. The article even included an homage to: “Dick, I’m very disappointed in you.” “It’s just a small glitch.”
• 1970-01-01 11mo ago
  Very ripe for a lawsuit. I would expect lawyers to be calling daily.
• nullbyte808 11mo ago
  I would get my GED at that point. Screw that school.
• satisfice 11mo ago
  T0 be fair most commercials for Doritos, Skittles, Mentos, etc., if occurring in real life, would result in a strong police response just after they cut away.
• einrealist 11mo ago
  Let's hope that, thanks to AI, the young man will now have a healthier diet! /s
• gnarlouse 11mo ago
  And so begins the ending of the "unfinished fable of the sparrows"
• programjames 11mo ago
  The model seems pretty shitty. Does it only look on a frame-by-frame basis? Literally one second of video context and it would never make that mistake.
• BeetleB 11mo ago
  The "AI mistake" part is a red herring. The real question is: Would this have happened in an upper/middle class school. The student has dark skin. And is attending a school in a crime ridden neighborhood. Were it a white student in a low crime neighborhood, would they have approached him with guns drawn? The AI failure is masking the real problem - bad police behavior.
• ratelimitsteve 11mo ago
  the best part of the technocracy is that they're not actually all that good at anything. the second best part is that when their mistakes end in someone dead there will be some way that they're not responsible.
• neverkn0wsb357 11mo ago
  It’s unsurprising, since this kind of classification is only as good as the training data. And police do this kind of stuff all the time (or in the very least you hear about it a lot if you grew up in a major city). So if you’re gonna automate broken systems, you’re going to see a lot more of the same. I’m not sure what the answer is but I definitely feel that “security” system like this that are purchased and rolled out need to be highly regulated and be coupled with extreme accountability and consequences for false positives.
• adam12 11mo ago
  This is what we get instead of reasonable gun control laws.
  • 15155 11mo ago
    You're free to (attempt to) amend the Second Amendment, but the Supreme Court of the United States has already affirmed and reaffirmed that individual ownership of firearms in common use is a right. What do you propose that is "reasonable" given the frameworks established by Heller, McDonald, Caetano, and Bruen? I can 3D print or mill basically every item you can imagine prohibiting at home: what exactly do laws do in this case?
    • kelnos 11mo ago
      > the Supreme Court of the United States has already affirmed and reaffirmed that individual ownership of firearms in common use is a right. The current interpretation of 2A is actually a fairly recent invention; in the past, it's been interpreted much more narrowly. And if SCOTUS can overturn Roe v. Wade's precedent, they can do the same with their interpretation of 2A. They won't of course, at least not until some of its members age out and get -- hopefully -- replaced with people who aren't idiots. But I'd be fine if 2A was amended away. Let the states make whatever gun laws they want, and we can see whether blue or red states end up with lower levels of gun violence as a result.
      • 15155 11mo ago
        Roe v. Wade was bad jurisprudence according to RBG herself (who voted unanimously in Caetano.) Great example! > The current interpretation of 2A is actually a fairly recent invention; What was Miller, over a century ago? Is that "recent?" > hopefully -- replaced with people who aren't idiots. See you in 30 years! > But I'd be fine if 2A was amended away. Good luck! I'll keep milling and printing in the meantime!
• kayge 11mo ago
  I wonder if the AI correctly identified it as a bag of Doritos, but was also trained on the commercial[0] where the bag appears to beat up a human (his fault for holding on too tight) and then it destroys an alien spacecraft. [0] https://www.youtube.com/watch?v=sIAnQwiCpRc https://www.youtube.com/watch?v=sIAnQwiCpRc
• crazygringo 11mo ago
  It sounds like the police mistook it as well: > “They showed me the picture, said that looks like a gun, I said, ‘no, it’s chips.’” So AI did the initial detection, but police looked at it and agreed. We don't see the image, but it probably did look like a gun because of a weird shadow or something. Fundamentally, this isn't really any different from a person seeing someone with what looks like a gun and calling the cops, only it turns out the person didn't see it clearly. The main issue is just that with increased numbers of images, there will be an increase in false positives. Can this be fixed by including multiple images, e.g. from motion of the object, so police (and the AI) can better eliminate false positives before traumatizing some poor teen?
  • axus 11mo ago
    Computer says it looks like a gun. https://en.wikipedia.org/wiki/Computer_says_no https://en.wikipedia.org/wiki/Computer_says_no
  • Mawr 11mo ago
    Picture? Images? But those are just frames of footage the cameras have captured! Why would one purposefully use less information to make a decision rather than more? Just put the full footage in front of an unbiased third party for a multi-stage verification first. The problem space isn't "is that weird shadow in the picture a gun or not?" it's "does the kid in the video have a gun?". It's not hard to figure out the difference between a bag of chips and a gun based on body language. Presumably the kid ate chips out of the bag? Using certain motions that one makes when doing that? Presumably the kids around him all saw the object in his hands and somehow did not react as if it was a gun? Jeez.
    • crazygringo 11mo ago
      Those are a lot of "presumablies". Maybe you're right. Or maybe it was mostly obscured so you really couldn't tell. How do you know it was open and he was eating? How do you know there were other kids around and he wasn't solo? Why do you think the body language would be so different? Nobody is claiming he was using a gun or threatening anyone with it. If you're just carrying something in your hand, I don't know how you could tell what the object is or isn't from body language.
• jmyeet 11mo ago
  There are two basic ways AI can be used: 1. To enhance human productivity; or 2. To replace humans. Companies, particularly in the US, very much want to go with (2) and part of the reason they can is because there are zero consequences for incidents like this. A couple ofexamples spring to mind: 1. the UK Royal Mail scandal where a bad system accused postmasters of theft, some of whom committed suicide over the allegations. Those allegations were later proven false and it was the system's fault. IMHO the people who signed off and deployed this should be charged with negligent homicide; and 2. The Hertz case where people who had returned cars were erroneously flagged as car thieves and report was made to police. This created hell for people who would often end up with warrants they had no idea about and would be detained on random traffic stops over a car that was never stolen. Now these aren't AI but just like the Doritos case here, the principle is the same: companies are trying to replace people with computers. In all cases, a human should be responsible for reviewing any such complaint. In the Hertz case, a human should check to see if the car is actually stolen. In the Royal Mail situation, the system needs to show its work. Deployment should be against the existing accounting system and discrepancies between the two need to be investigated for bugs until the system is proven correct. Particularly in the early stages, a forensic accountant (if necessary) should verify that funds were actually stolen before filing a criminal complaint. And if "false positive" criminal complaints are filed, the people who allowed that to happen, if negligent (and we all know they are), should themslves be criminally charged. We are way too tolerant of black box systems that can result in significant harm or even death to people. Show your work. And make a human put their name and reputation to any output of such systems.
• zkmon 11mo ago
  At least there is a check done by humans in a human way. What if this human check is removed in future, as AI decisions would be deemed no longer requiring a human inspection?
  • Ylpertnodi 11mo ago
    Isn't that what happened here?
• blindriver 11mo ago
  How is this not slander? I would absolutely sue the fuck out of this system where it puts people's lives in danger.
  • kelnos 11mo ago
    > How is this not slander? Because that's not what slander is.
• anothernewdude 11mo ago
  America does American things.
• aussieguy1234 11mo ago
  Inflicting trauma on a harmless human in the name of the "safety of others" is never ok. The victim here was not unharmed, but is likely to end up with PTSD and all the mental health issues that come with it. I hope they sue the police department over this.
• johnnyApplePRNG 11mo ago
  Who knew eating Doritos could make you a millionaire? I hope this kid gets what he deserves. What a tragedy. I'm sure racial profiling on behalf of the AI and the police had absolutely nothing to do with it.
• tomxor 11mo ago
  Armed and dangerous until proven chips.
• jMyles 11mo ago
  Everything around us: political tumult and weaponization of the justice system, ICE and other capricious projections of federal authority, the failure of drug prohibition, and on and on and on, points to a very simple solution: Abolish SWAT teams. Do away with the idea that the state employees can be permitted to be more armed than anyone else. Blaming the so-called 'swatter' (whether it's a human or AI) is really not getting at the root of the problem.
• beloch 11mo ago
  Systems like this need to report confidence in their assertions. e.g. Not "this student has a gun" but "this model says the student has a gun with a probability of 60%". If an AI can't quantify it's degree of confidence, it shouldn't be used for this sort of thing.
  • xp84 11mo ago
    Even better, share the frame(s) that the guess was drawn from with a human for verification before triggering ANYTHING. How much trouble could that possibly be? How many "guns" is this thing detecting in a day across all sites? I doubt more than a couple or we'd have heard about tons of incidents, false positives or not. I wanna see the frames too.
    • anigbrowl 11mo ago
      I don't find that especially good as a sole remedy, because lots of people are stupid. If they see a green outline box overlaid on a video image with the label 'gun', many many people will just respond to the label instead of looking at the underlying image and trying to make a decision. Probability and validation history need to be built into the product so that there are audit logs that can be pored over and challenged. Bad human decision-making, which is rampant, is always smoothed over with justifications like 'I was concerned for everyone's safety', and usually treated in isolation rather than assessed longitudinally.
  • tecoholic 11mo ago
    Doesn’t work. Competition will make them report higher accuracy to make the product look better.
• tiagod 11mo ago
  The regular types of school shootings weren't enough, so they invented AI-powered police school shootings to the mix.
  • doublerabbit 11mo ago
    Up next: gun kitted drones patrolling the school playground.
• anal_reactor 11mo ago
  The core of the issue is that many Americans do carry weapons which means that whatever the security system, it needs to keep in mind that the suspect might be armed and about to start shooting. This makes the police biased towards escalation because the only way against a shooter is to shoot first. This problem doesn't exist in Europe or Japan because guns aren't that ubiquitous, which means that the police have the time to think before they act, which makes them less likely to escalate and start shooting. Obviously, for Americans, the only solution is to get rid of the gun culture, but this will never happen, so suck it up that AI gets you swatted.
  • Ylpertnodi 11mo ago
    You had me up until the last sentence... >Obviously, for Americans, the only solution is to get rid of the gun culture, but this will never happen, so suck it up that AI gets you swatted. ...and you are correct.
  • deleted 11mo ago
    [deleted]
• ignormies 11mo ago
  > Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” This exact scenario is discussed in [1]. The "human in the loop" failed, but we're supposed to blame the human, not the AI (or the way it was implemented). The humans serve as "moral crumple zones". """ The emphasis on human oversight as a protective mechanism allows governments and vendors to have it both ways: they can promote an algorithm by proclaiming how its capabilities exceed those of humans, while simultaneously defending the algorithm and those responsible for it from scrutiny by pointing to the security (supposedly) provided by human oversight. """ [1]: https://pluralistic.net/2024/10/30/a-neck-in-a-noose/ https://pluralistic.net/2024/10/30/a-neck-in-a-noose/
  • kelnos 11mo ago
    The article doesn't confirm that there was definitely a human in the loop, but it sorta suggests that police got a chance to manually verify the photo before going out to harass this poor kid. I suspect, though, that the AI flagging that image heavily influenced the cop doing the manual review. Without the AI, I'd expect that a cop manually watching a surveillance feed would have found nothing out of the ordinary, and this wouldn't have happened. So I agree that it's weird to just blame the human in the loop here. Certainly they share blame, but the fact of an AI model flagging this sort of thing (and doing an objectively terrible job of it) in the first place should take most of the blame here.
• perplex3d 11mo ago
  An alert by one of these AI tools, which from what I understand have a terrible track record, should not be reasonable suspicion or probable cause to swarm a teenager with guns drawn. I wish more people in local communities would understand how much harm this type of surveillance and response causes. Our communities should not be using these tools.
• teeray 11mo ago
  > “They showed me the picture, said that looks like a gun, I said, ‘no, it’s chips.’” "Sorry, that's Nacho gun"
• rglover 11mo ago
  This may be mean, but we should really be careful about just handing AI over to technically illiterate people. They're far more likely to blind trust the LLM/AI output than someone who may be more experienced and take a beat. AI in an agentic-state society (what we have in America at least) is an absolute ticking time bomb. Honestly, this is what AI safety teams should be concentrated on: making sure people who think the computer is infallible understand that, no, it isn't, and you shouldn't just assume what it tells you is correct.
  • hollow-moe 11mo ago
    We already handed over the Internet to technically illetrate people long time ago.
  • hanspeter 11mo ago
    It's basically a failure of setting up the proper response playbook. Instead of: 1. AI detects gun on surveillance 2. Dispatch armed police to location It should be: 1. AI detects gun on surveillance 2. Human reviews the pictures and verifies the threat 3. Dispatch armed police to location I think the latter version is likely what already took place in this incident, and it was actually a human that also mistook a bag of Doritos for a gun. But that version of the story is not as interesting, I guess.
• sans_souse 11mo ago
  I thought on first glance the source was from doritos.com That would have been bold
• SanjayMehta 11mo ago
  Robocop. Edit: And racism. Just watched the video.
• ncr100 11mo ago
  Sincere, and snarky summary: "Omnilert" .. "You Have 10 Seconds To Comply" -now targeting Black children! Q: What was the name of the Google AI Ethicist who was fired by Google for raising the concern that AI overwhelmingly negatively framed non-white humans as threats .. Timnit Gebru https://en.wikipedia.org/wiki/Timnit_Gebru#Exit_from_Google https://en.wikipedia.org/wiki/Timnit_Gebru#Exit_from_Google We, as technologists, ARE NOT DOING BETTER. We must do better, and we are not on the "DOING BETTER" trajectory. We talk about these "incidents" with breathless, "Wwwwellll if we just train our AI better ..." and the tragedies keep rolling. Q2: Which of you has had a half dozen Squad Cars with Armed Police roll up on you, and treat you like you were a School Shooter? Not me, and I may reasonably assume it's because I am white, however I do eat Doritos.
• bobbyprograms 11mo ago
  All right they’ve gotta have a plain clothes bro go up there make sure the kid is chill. You know the difference between a murder and not can be as little as somebody being nice
• Lucian6 11mo ago
  [dead]
• hshdhdhehd 11mo ago
  The solution is easy. Gun control. We dont feel the need to have AI surveillance on people to detect guns in ROTW.
• adxl 11mo ago
  Hallucinate much?
• G_o_D 11mo ago
  its not gun detection that ai is racist just like its white creators
• slipperybeluga 11mo ago
  [dead]
• uda 11mo ago
  When people wonder how can AI mistake a bag of snacks as a weapon, simply answer "42" It is about the question, the answer will become very clear once you understand what was the question presented to the inference model, and of course what data and context was fed
• balls187 11mo ago
  > “false positive” but claimed the system “functioned as intended,” Fuck you.
• kelnos 11mo ago
  I don't have kids yet, but I may someday. I went to public school myself, and would prefer to send any kid of mine to public school as well. (I'm not hard against private schools, but I'd prefer my kid gets to make friends from all walks of life, not just people who have parents who can afford private school.) But I really wouldn't want to send my kid to a school that surveils students all the time, and uses garbage software like this that directly puts kids into dangerous situations. I feel like with a private school, I'd have more choice and ability to influence that sort of thing.
• prmoustache 11mo ago
  We blame AI here but what's up with law enforcment that comes with loaded guns in hand and send someone to the ground and cuff him before actually doing any check? That is the real issue. Police force anywhere else in the world that know how to behave would have approched the student, have had a small chat with him, found out all he had in hands was a bag of doritos, maybe would have asked politely to see the content of his bag, explaining the search has been triggered by an autodetection system that may lead to occasional errors and wished him a good day.
  • lloda 11mo ago
    Or they'd have looked at whatever images the system had used for its decision and see it was a false positive without having to send anyone over.
  • balls187 11mo ago
    >That is the real issue. No. Trusting AI is clearly the issue. If there was a 9-1-1 call to the police that there was an active shooter at your kids school, how would you want the police to show up?
• uvaursi 11mo ago
  I can understand the outrage in this thread but literally none of what you are all calling for will be done. No one from justice or law reads HN to see what should be done. I wish folks here would keep a cooler head rather than posting lengthy rants and vents that call for punishing school staff. Really unprofessional and immature from a community that prides itself, to fall constantly into a cycle of vitriol. Can someone outline a more pragmatic, if not likely, course of what happens next after this? Is it swept under the rug and we move on?
• thescriptkiddie 11mo ago
  we need personal liability for the owners of companies that make things like this
• chvid 11mo ago
  How likely is it that the AI system would have classified the bag of Doritos as a weapon had the person carrying it been white instead of black?
• scotty79 11mo ago
  I was unduly surprised and disappointed when I saw the photo of the kid and he turned out to be black. I would love to believe that this had no impact on how the whole thing played out, but I don't.
• metalman 11mo ago
  AI is a false (political) wish, it can and never work, it is the desperation of an over extended power structure to hold on and permanently consolodate controll of all of the worlds population, and nothing else. the proofs are there. philosophers mulled this over long ago and made clear statements as to why ai cant work though not that for a second do I misdunderstand that it is "all in" for ai, and we all get to go for the 100 trillion dollar ride to hell. can we have truely awsome automation for manufacturing and mundane beurocratic tasks?, fuck ya we can! but anything that requires understanding, is forever out of reach, which unfortunatly is also lacking in the people pushing, this thing, now
• rs186 11mo ago
  * AI mistakenly reports the student as a threat * the student was black Is that really a coincidence? It's just a matter of time before this or something worse happens.
• ninalanyon 11mo ago
  Durely a human being should review the evidence before going off half cocked.
• twoquestions 11mo ago
  Before I clicked the article, I said to myself "The victim's gotta be Black", and lo and behold. AI has inherited police's (shitty, racist, and dangerous) idea that any Black person is a dangerous monster for whom anything is a weapon.
• pickleglitch 11mo ago
  Sounds like this high school is doing a great job preparing students for the real world, where they can be swarmed by jackbooted thugs at any moment for any reason.
• dpk84 11mo ago
  Was this really an AI gun detection system, or just a machine that goes off randomly?
• lunias 11mo ago
  "I am invoking my 4th and 5th amendment rights afforded to me by the Constitution of the United States of America. I have no further comment until I have consulted with and am in the presence of my legal council." Then, just sit back and enjoy as the lawsuit unfolds.
• jharrison11 11mo ago
  Looks like per their website it did function as intended... It surfaces potential threats for the school to look at and make a human decision. The principal decided to send the police after the school safety team dismissed it as part of the correct process. I mean fire alarms go off for lots of things that are not fire alarms... This was an alert meant to be validated by a human that messed up. Its pretty clearly documented how it works here: https://www.omnilert.com/solutions/gun-detection-system https://www.omnilert.com/solutions/gun-detection-system https://www.omnilert.com/solutions/ai-gun-detection https://www.omnilert.com/solutions/ai-gun-detection https://www.omnilert.com/solutions/professional-monitoring https://www.omnilert.com/solutions/professional-monitoring
• lawiejtrlj 11mo ago
  This is Brazil-level insanity, but half the people on this forum hope to make money off it so it's fine
• iamleppert 11mo ago
  If I was that kid I'd be suing the school, the AI company, the police, anyone and everyone who had to be subjected to the mistake.
• burnt-resistor 11mo ago
  It wasn't sour cream and onion and didn't contain cash, so it's super sus. But really this is typical of cop overreaction with escalation and ego rather than calm, legal, and reasonable investigation. Karens may SWAT people they don't like, but it's police officers who must use reasonableness and restraint to defend the vestiges of their impartiality and community confidence based on asking questions and gathering evidence in a legal and appropriate manner rather than rushing to conclusions. Case in point: The NYC rough false arrest of a father in front of his kid to retrieve his mis-delivered package where the egomaniacal bully cop aggressively lectures a guy for his own mistake to cover his own ego while blaming the victim: https://youtu.be/LXd-4HueHYE https://youtu.be/LXd-4HueHYE