8 ms·
> Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and a
by neilv 1y ago
> Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.”
It just created a situation in which a bunch of people with guns were told that some teen had a gun. That's a very unsafe situation that the system created, out of nothing.
And some teen may be traumatized. Again, unsafe.
Incidentally, the article's quotes make this teen sound more adult than anyone who sold or purchased this technology product.
- bilbo0s 1y ago>And some teen may be traumatized. Um. That's not really the danger here. The danger is that it's as clear as day that in the future someone is gonna be killed. That's not just a possibility. It's a certainty given the way the system is set up. This tech is not supposed to be used in this fashion. It's not ready.
- wat10000 1y agoI fully agree, but we also really need to get to a place where drawing the attention of police isn't an axiomatically life-threatening situation.
- Zigurd 1y agoIf the US wasn't psychotic, not all police would have to be armed, and not every police response would be an armed response.
- wlesieutre 1y agoEven if not all police were armed, the response to "AI said someone has a gun" would always be the armed police
- georgefrowny 1y agoWhy would it not be "human reviews the image that the AI said was a gun"?
- krapp 1y agoThe entire selling point of AI is to not have humans in the loop.
- array_key_first 1y agoEven despite the massive protests in the past few years, we're moving further in that direction.
- akoboldfrying 1y ago> The danger is that it's as clear as day that in the future someone is gonna be killed. This argument can be made about almost every technology, including fire, electricity, and even the base "technology" of just letting a person watch a camera feed. I'm not downplaying the risks. I'm saying that we should remember that almost everything has risks and benefits, and as a society we decide for or against using/doing them based mostly on the ratio between those two things. So we need some data on the rates of false vs. true detections here. (A value judgment is also required.)
- GuinansEyebrows 1y ago> This argument can be made about almost every technology, including fire, electricity, and even the base "technology" of just letting a person watch a camera feed. huh, i can think of at least one recent example of a popular figure using this kind of argument to extreme self-detriment.
- akoboldfrying 1y agoIs HN really this ready to dive into obvious logical fallacies? My original comment, currently sitting at -4, has so far attracted one guilt-by-association plus implied-threat combo, and no other replies. To remind readers: My horrifying proposal was to measure both the risks and the benefits of things. If anyone genuinely thinks measuring the risks and benefits of things is a bad idea, or that it is in general a good idea but not in this specific case, please come forward.
- GuinansEyebrows 1y agosorry for being glib; it was low hanging fruit. my actual point should have been more clearly stated: measuring risk/benefit is really complicated because there's almost never a direct comparison to be made when balancing profit, operational excellence and safety.
- Mawr 1y ago> Is HN really this ready to dive into obvious logical fallacies? No, which is why your comment was downvoted - the following is a fallacy: > This argument can be made about almost every technology, That's the continuum fallacy.
- deleted 1y ago[deleted]
- neilv 1y agoDid you want to emphasize or clarify the first danger I mentioned? My read of the "Um" and the quoting, was that you thought I missed that first danger, and so were disagreeing in a dismissive way. When actually we're largely in agreement about the first danger. But people trying to follow a flood of online dialogue might miss that. I mentioned the second danger because it's also significant. Many people don't understand how safety works, and will think "nobody got shot, so the system must have worked, nothing to be concerned about". But it's harder for even those people to dismiss the situation entirely, when the second danger is pointed out.
- Aeolun 1y agoI’d argue the second danger is worse, because shooting might be incidental (and up to human judgement) but being traumatized is guaranteed and likely to be much more frequent.
- krapp 1y agoAmericans are killed by police all the time, and by other Americans. We've already decided as a society that we don't care enough to take the problem seriously. Gun violence, both public and from the state, is accepted as unavoidable and defended as a necessary price to pay to live in a free society[0]. Having a computer call the shots wouldn't actually make much of a difference. Hell, it wouldn't even move the needle on racial bias much because LLMs have already shown themselves to be prejudiced against minorities due to the stereotypes in their training data. [0]Even though no other free society has to pay that price but whatever.
- coryrc 1y agoFar more deaths by automobile than homicides by guns.
- deleted 1y ago[deleted]
- oh_my_goodness 1y agoIn the US, guns and automobiles kill roughly the same number of people each year.
- coryrc 1y agoThat's why I said homicides.
- oh_my_goodness 1y agoI was responding to your original comment.
- ivell 1y agoGuns are actually easier to control and significantly reduce ability to target multiple people at once. There are a lot of countries successfully controlling guns. To the argument that then only criminals have guns - in India at least, criminals have very limited access to guns. They have to resort to unreliable handmade guns which are difficult to procure. Usually criminals use knives and swords due to that.
- omnipresent12 1y agohttps://www2.ljworld.com/news/schools/2025/aug/07/lawrence-school-district-sued-in-federal-court-for-use-of-ai-powered-surveillance-system-students-claim-gaggle-results-in-illegal-searches/ https://www2.ljworld.com/news/schools/2025/aug/07/lawrence-s... Another false positive by one of these leading content filters schools use - the kid said something stupid in a group chat and an AI reported it to the school, and the school contacted the police. The kid was arrested, stripped searched, and held for 24 hours without access to their parents or counsel. They ultimately had to spend time in probation, a full mental health evaluation, and go to an alternative school for a period of time. They are suing Gaggle, who claims they never intended their system to be used that way. These kinds of false positives are incredibly common. I interviewed at one of their competitors (Lightspeed), and they actually provide a paid service where they have humans review all the alerts before being forwarded to the school or authorities. This is a paid addon, though.
- reaperducer 1y agoThe kid was arrested, stripped searched, and held for 24 hours without access to their parents or counsel. They ultimately had to spend time in probation, a full mental health evaluation, and go to an alternative school for a period of time. All he wanted was a Pepsi. Just one Pepsi. And they wouldn't give it to him.
- michaelt 1y ago> They are suing Gaggle, who claims they never intended their system to be used that way. Yeah, there's a shop near me that sells bongs "intended" for use with tobacco only.
- avidiax 1y agohttps://archive.is/DYPBL https://archive.is/DYPBL > Gaggle’s CEO, Jeff Patterson, said in an interview that the school system did not use Gaggle the way it is intended. The purpose is to find early warning signs and intervene before problems escalate to law enforcement, he said. “I wish that was treated as a teachable moment, not a law enforcement moment,” said Patterson. It's entirely predictable that schools will call law enforcement for many of these detections. You can't sell to schools that have "zero tolerance" policies and pretend that your product won't trigger those policies.
- random3 1y agoIt’s actually “AI swarmed” since no human reasoning, only execution, was exerted - basically have an AI directing resources.
- actionfromafar 1y agoReverse Centaur. MANNA.
- NackerHughes 1y ago3-in-1. Lack.
- trhway 1y agodelegating decision to AI, excluding human from the "human in the loop" is kind of unexpected as a first step, as in general it was expected that exclusion will start from the other end. Sideway i wonder how is that going to happen on the battlefield. for this civilian use case the next step - AR googles worn by police with that AI will be projecting onto the googles where that teenager has his gun (kind of Black Mirror style), and the next step after that is obviously excluding the humans even from the execution step.
- goopypoop 1y agowhen attacked by bees am I hive swarmed?
- janalsncm 1y agoIn any system, there are false positives and false negatives. In some situations (like a high recall disease detection) false negatives are much worse than false positives, because the cost of a false positive is a more rigorous screening. But in this case both are bad. If it was a false negative students might need therapy for a more tragic reason. Aside from improving the quality of the detection model, we should try to reduce the “cost” of both failure modes as much as possible. Putting a human in the loop or having secondary checks are ways to do that.
- nkrisc 1y agoIn this case false positives are far, far worse than false negatives. A false negative in this system does not mean a tragedy will occur, because there are many other preventative measures in place. And never mind the fact that this country refuses to even address the primary cause of gun violence in the first place: the ubiquity of guns in our society. So systems like this is what we end up with when we ignore to address the problem of guns and choose to deal the downstream effects of that instead.
- Lammy 1y ago> the primary cause of gun violence in the first place: the ubiquity of guns in our society I would have gone with “a normalized sense of hopelessness and indignity which causes people to feel like violence is the only way they can have any agency in life” considering “gun” is the adjective and “violence” is the actual thing you're talking about.
- pixelready 1y agoBoth are true. The underlying oppressive, lonely, pro-bullying culture creates the tension. The proliferation of high lethality weapons makes it more likely that tension will eventually release in the form of a mass tragedy. Improvement in either area would be a net positive for society. Improvement in both areas is ideal but solving proliferation seems a lot more straightforward than fixing the generally miserable society problem.
- b00ty4breakfast 1y ago>...its purpose is to “prioritize safety and awareness through rapid human verification. Oh look, a corporation refusing to take responsibility for literally anything. How passe.
- chillingeffect 1y agoThe invention of the corporation is virtually to eliminate responsibility/culpability from any individual. Human car crash? Human punishment. Corporate-owned car crash? A fine which reduces salaries some negligible percent.
- perplex3d 1y agoYes, corporations have all of the rights of a person, abilities beyond a person, yet few of the responsibilities of a person.
- parineum 1y agoThey actually don't have all the rights of a person and they do have those same responsibilities. If this company was a sole proprietorship, the only recourse this kid would have is to sue the owner, up to bankruptcy. Since it's a corporation, his recourse is to sue the company, up to bankruptcy. As for corporations having rights, I can explain it further if necessary but the key understanding is that the singular of "corporations are people" is "a corporation is people" not "a corporation is a person".
- wkat4242 1y agoYou can't put a corporation in prison. But a person you can. This is one of the big problems. The people making the decisions at corporations are shielded from personal consequences by the corporation. A corporation can be shut down but it rarely happens. Even when Boeing knowingly caused the deaths of hundreds (especially the second crash was entirely preventable if they would have been honest after the first one), all they got were some fines. Those just end up being charged back to their customers, a big one being the government who fined them in the first place.
- DrewADesign 1y agoEngineer: hey I made this cool thing that can help people in public safety roles process information and make decisions more efficiently! It gives false positives but you save more time than it takes less time to weed through them. Someone nearby: well what if they use it to replace human thinking instead of augment it? Engineer: well they would be ridiculous. Nobody would ever think that’s a good idea. Marketing Team: it seems like this lands best when positioning it as a decision-making tool. Let’s get some metrics on how much faster it is at making decisions than people are. Sales Rep: ok, Captain, let’s dive into our flagship product, DecisionMaker Pro, the totally automated security monitoring agent… ::6 months later—some kid is being held at gunpoint over snacks.::
- JimTheMan 1y agoRefer to the post office scandal in Britain and the robodebt debacle in Australia. The authorities are just itching to have their brains replaced with by dumb computer logic, without regard for community safety and wellbeing.
- jononor 1y agoLack of Accountability as-a-Service! Very attractive proposition to negligent and self-serving organizations. The people in charge don't even have to pay for it themselves, can just funnel the organization money to the vendor. Encouraging widespread adoption helps normalizes the practice. If anyone objects, shut them down as not thinking-of-the-children and something-must-be-done (and every other option is surely too complicated/expensive).
- DrewADesign 1y agoAnd the black box sentencing recommendation systems some US states bought into like a decade ago.
- casey2 1y agoNice fantasy, but the reality is that the "people in public safety roles" love using flimsy pretenses to harass and abuse vulnerable populations. I wish it was just overeager sales and marketing, but you're view of humanity is way too naive especially as masked thugs are disappearing people in the street as we type.