29 ms·

School AI surveillance can lead to false alarms, arrests

• cramcgrab 1y ago
  And with the recent cell phone ban you have no proof, no video or pictures.
  • tyleo 1y ago
    Did you even read the article? For the example at the top, all the stuff these kids did was well documented. I don’t understand how having a phone helps here except to add even more distraction in school and another avenue for surveillance. The kids used a school communication program to say something racist. Schools should monitor school communication platforms. The only thing I disagree with is the level of punishment (sending a kid to jail for a night).
    • internalfx 1y ago
      AND strip searching them, AND not allowing communication with their parents...
      • tyleo 1y ago
        I don’t understand your point. It seems like you are saying, “this punishment is so severe we shouldn’t be correcting the behavior at all.”
    • 7thaccount 1y ago
      Yes. I think some are pointing out that this crazy level of surveillance is at Orwellian levels. Why are kids even messaging each other on these platforms? I know books and paper is old school, but it does the job well. If they have to use some kind of internal system, then a racist joke could've led to a detention and parent/teacher talk. It didn't need to auto-flag law enforcement and traumatize the kid for life. Involuntary commitment at school sounds as unAmerican as it gets.
      • 7thaccount 1y ago
        Edit: the threat of violence is obviously not acceptable in an age of horrifying events like school shootings. I think that goes without saying, but wanted to add to my above comment. As I unfortunately didn't make it super clear. However, there should be a way to address the risk and prevent acts of terrorism without turning into a police state. A more reasonable response might have been to have the parents come by for a long meeting with the principal, school counselor, and school resource officer to talk about the severity of making such statements - even if she did make it sarcastically as tone/intent is difficult to judge. Then suspend the student for a day or so and have the school resource officer periodically check to ensure she didn't bring any weapons. To play devil's advocate though, I can't imagine the stress from parents who have children of Mexican descent in such a situation.
  • soulofmischief 1y ago
    When I was in high school, the police illegally body searched my entire school as a racist flex after tackling, beating and tasing a student over a pack of cigarettes one foot outside the door of my math class. A student responded by filming them and handing it over to a local news station, who ran the story and footage. To retaliate, the next day administrators had metal detector wands waiting for us right off the busses, took every single cellphone they found and locked them up at the school board office for the rest of the year. That school was absolute hell, a battleground between students and teachers. I am not exaggerating at all when I say that being spotted outside your classroom was an immediate expulsion, with not even enough time between classes to pee or use your locker (two minutes, we had to run). As part of the escalation, the fire alarm began being pulled at least twice a day. Any student who had even a moment alone with one would pull it immediately. Absolute chaos and a direct result of power collapse due to a racist, authoritarian school board that only knew how to wield institutional violence. For this reason, my kid will always have a phone in order to protect themselves from administrative abuse. I will fight for that tooth and nail. I had over 40 write-ups in just elementary school for refusing entertain abuse from authoritarian staff, and I'll be a failure if my kid doesn't walk the same path.
    • 1718627440 1y ago
      Wow, US schools sound like prisons.
      • soulofmischief 1y ago
        This was probably in the top ten worst high schools in the country. It was the worst high school in my state, which I attended a semester after attending the best public high school in my state, so I really got to see the wide gamut of education available in my state. And my state floats between the 1st-3rd worst ranked in the US for education, so it was truly a bottom-of-the-barrel experience. Each time you'd walk into class, it was a dice roll if another piece of equipment had been stolen. We stopped buying and using projectors because any new ones got stolen within days. Sometimes I'd walk into class and every single piece of furniture had been completely turned upside down. Students would play on their phones, yell, throw things at the teachers, and the teachers would just ignore them. It was also an extremely racially charged situation which had a big impact. I think there were less than ten white people in the school, the rest of the school was black, and a handful of hispanic folk. The white people were almost all racist, and so apart from my neighbor, I only hung around black people. As the only white person there who hung out with non-white people, my nickname across the entire high school somehow became Tarzan, in part due to my long hair. When I'd walk down the halls each day, people would beat their chests and make ape noises and give me dap. I'll never forget being at an award ceremony and walking up for a few awards, each time the entire student body would start stomping in the bleachers and making jungle noises and shouting my nickname. The visibly perplexed administration had no idea what was going on. Because my credits would not cleanly transfer to their curriculum, I had two free periods each day and used this time to help out the administration and teaching staff, running errands, organizing, doing paperwork, etc. This allowed me to interface often with the school board, who was... entirely comprised of white, racist, old men. Men who fundamentally refused to understand the predicament facing the student body, and who were too eager to escalate instead of finding a way to bridge communication between the student body and faculty. Instead of empowering students, they saw them all as future bodies for the prison and service industry, and it was obvious. Students learned nothing, but would get straight A's. They would of course absolutely fail state and national standardized tests. They were shoved through the meat grinder with nothing to show for it.
• internalfx 1y ago
  It's 1984 in Lawrence, KS...
• kayodelycaon 1y ago
  Strip searching a 13-year-old girl and locking her in a jail cell for a joke? Wow. And they did all this without telling her parents.
  • 7thaccount 1y ago
    Absolutely horrifying
  • bryanrasmussen 1y ago
    I remember when in high school in Utah we went on a field trip to a halfway house, for some reason - it was during the time when scared straight was real popular so of course every kid should be scared straight whether they have juvenile offenses or not. It was some sort of law class I think. Anyway they took us to look at the inmates in the halfway house who were behind bars and then they could come out to the bars and as their part of the whole scared straight exercise they would of course yell stuff at us, which was mainly about how they wanted to have sex with the approximately 16 year old girls on the trip. Gosh, Utah sure is a morally upstanding place.
    • DonHopkins 1y ago
      [flagged]
    • yard2010 1y ago
      This is nefarious.
  • sofixa 1y ago
    A joke which could, out of context, be interpreted to be a threat of violence: > on Thursday we kill all the Mexico’s Especially considering the frequency of violence in American schools, can't really blame the school for jumping to conclusions.
    • deleted 1y ago
      [deleted]
    • 1y ago
• malwrar 1y ago
  This will condition children to think this sort of surveillance is normal, and when they’re adults the ones who think it kept them safe from mass shootings will try and advocate using our existing mass-surveillance powers to proactively monitor everyone like this. Please, we need to stop terrifying children with this lazy oppression, this is not worth the damage to society we’re causing by conditioning kids this way.
  • password321 1y ago
    Half the point of school has always been to condition them to behave and think a certain way for adulthood.
  • astura 1y ago
    This sort of surveillance is normal, though. Most platforms are monitored for threats of violence. Are you thinking the outcome would have been much different if she threatened to kill Mexicans on Teams at work instead?
    • malwrar 1y ago
      The child in OP sounds to me like she thought she was making a bad-taste joke in a private forum, and was shocked when it promptly led to an entanglement with an unfeeling system who was looking over her shoulder. I’ve never threatened to kill anyone in my work DMs, but I’ve definitely written stuff that I wouldn’t post in public threads. I think we all to some degree use these “private” systems this way until it burns us, only then do we adjust. Privacy is much more a feeling than a technical reality. In that sense, it isn’t “normal”, it’s just “something that’s happening in theory but eh maybe it only affects scary people or whatever idk”. I feel like this tolerance we’re developing for outside forces invading “private” spaces, nominally for these loose justifications of harm reduction, will be what _actually does_ make it normal. Once it’s truly normal, and people think it’s what keeps them safe from mass shootings or whatever, it will be too late to get rid of it. I think fear and normalcy will motivate its spread to places beyond school chat platforms and Snapchat.
      • deleted 1y ago
• tyleo 1y ago
  I don’t like this article. It mixes two things together: 1. Schools running and monitoring their own communication platforms. This seems fine. 2. The US government monitoring private communication platforms like Snapchat and arresting kids on school grounds. This seems bad. The way it’s written though mixes these together and makes it look like schools are monitoring the private communication of students. I think it’s fine for schools to monitor there own platforms but weird for the government to monitor all platforms haphazardly.
  • abtinf 1y ago
    You have no problem with the opening example in the article?
    • tyleo 1y ago
      I have a problem with the degree of punishment but not that the student was punished. A student said, “on Thursday we kill all the Mexico’s,” on a schools private communication platform. The school should correct that behavior. Unfortunately they involved law enforcement. Thats where I see the problem. A better solution would be detention and informing the parents.
      • kayodelycaon 1y ago
        When I was a kid, detention over a stupid joke would have been an extreme overreaction and would result in the teacher being pulled into the principal’s office, not the kid.
        • JumpCrisscross 1y ago
          > detention over a stupid joke When you were a kid gun laws were stricter.
          • giantg2 1y ago
            Not exactly true. You don't know how old the person is nor their jurisdiction. Some states have fewer laws, others have more. At the federal level, there are more laws and rules than there were in the 60s or 70s (overall more than at any prior time). And of course enforcement varies. I remember many people coming to school with guns in their cars during hunting season even though it's not legal.
• IshKebab 1y ago
  Even ignoring the whole idea being awful, their response to someone who they think might want to enact a school shooting is to strip search and imprison them. What effect do they think that is going to have? Not the sharpest bullets in the barrel...
• Atreiden 1y ago
  This is utterly dystopian. We say some stupid things as kids, because they're just words and we're missing greater context at that age. Immediately and automatically engaging law enforcement, and even the FBI, is horrific. Kids have always had greatly restricted freedoms in schools, but transcending the classroom and monitoring their digital lives is just training them to accept the surveillance state.
  • worldsayshi 1y ago
    The fact that this technology exists might make decision makers feel compelled to always to add as much surveillance as possible and acting on it as diligently as possible. Because it's their responsibility to create safety. And the most short term solution is to always enact more control over everything.
    • doctorwho42 1y ago
      Adding more and more safety and control is always a self-defeating policy. It always ends with creating uncontrollable societal discord. Same thing with unfettered capitalism, the systems only work if we continue to support said systems. When the rules break down, so do the desires of the collective to maintain said systems.
      • worldsayshi 1y ago
        It seems that a deeper cause of this is lack of trust in long term solutions and the ability to come up with a plan that beat the short term knee jerk solutions - which could make the problems worse. If we can somehow win back trust in our collective ability to democratically solve problems... that should solve the problem. I think that involves some creative solutions to collective decision making.
        • doctorwho42 1y ago
          I agree, though I find that society has been throughly indoctrinated in thinking only about profit/economic motives and impacts. That is going to be the first major hurdle, teaching enough people that economics aren't the only metric for running a society, community, and life.
• giantg2 1y ago
  The problem are the lazy idiots using the software. This is the reason we need capable humans in the loop for anything important. Here we have a bad joke. The system flags it. The school sends it to the police. The police detain and interrogate the kid. Everyone is treating the determination of a complex automated system as their own determination. We also have every actor treating this as a credible threat. For this to be credible, you have to have the means to accomplish it. They gave a timeline. You know you have time to investigate before making an arrest. Problem is, nobody cares.
  • yard2010 1y ago
    If we have to rely on competent humans, we are doomed.
    • giantg2 1y ago
      Any system will have failures. The goal is to eliminate some of those failures through multiple checks. The automated system can help by bringing up possible issues that couldn't be monitored by humans. But then it should be humans reviewing these alerts for false positives. It should be much like using a security scanning tool. Findings will come up and you need someone to disposition them. Some will need no actions false positives or issues under your risk tolerance, but then others need appropriate responses. It seemes in the cases like the example, you have a intern running the scan tool and turning every finding into a high vulnerability because they don't know any better than to blindly trust the tool.
    • 1718627440 1y ago
      We always rely on humans being competent. If we forget that and treat everything as being correct then we are doomed.
• clwncr 1y ago
  School officials make the dubious claim that "the technology has detected dozens of imminent threats of suicide or violence." Oh, really? Do they have data that shows a significant reduction in violence since surveillance started, or is this just reframing false positives (that can result in arrest, eight weeks of house arrest, a psychological evaluation and 20 days at an alternative school) as a net benefit. My money is on the latter.
  • kotaKat 1y ago
    Guaran-fucking-teed the bullied kids are still getting bullied just as bad and the system is still failing those kids while the school can put an outward appearance to the rest of the community that nothing is wrong and everything is perfectly fine inside the walls. The victims will still be victims, they’ll just be punished by the system even harder for being a victim.
  • paradox460 1y ago
    I'm hoping the school installs tiger repellent rocks soon. You know that every school that installed the rocks saw no tiger attacks after the fact?
• ludicrousdispla 1y ago
  Hopefully none of the students type German into the chat.
• SeanDav 1y ago
  What the article seems to have glossed over is this: >>> A court ordered eight weeks of house arrest, a psychological evaluation and 20 days at an alternative school for the girl. If one assumes that the court did take into consideration context and age, it appears to largely validate the follow up decision once flagged. (I don't agree with lack of parental contact, to be clear.)
  • kayodelycaon 1y ago
    I’ve been at small courts. I wouldn’t assume the court took any of that into consideration. If the police were willing to make an example out of someone, the judge they work with is likely to do that as well.
• scotty79 1y ago
  > 13-year-old girl > When a friend asked what she was planning for Thursday, she wrote: “on Thursday we kill all the Mexico’s.” > Taken to jail, the teen was interrogated and strip-searched, and her parents weren’t allowed to talk to her until the next day, > She didn’t know why her parents weren’t there. > A court ordered eight weeks of house arrest, a psychological evaluation and 20 days at an alternative school for the girl. Wow, 2025 is wild. Police and court should have psychological evaluation instead and maybe some time off without pay to cool off. Protecting children, no matter how many of them they have to traumatize and incarcerate.
  • DonHopkins 1y ago
    [dead]
• evaXhill 1y ago
  Not that surprising that things have gotten to this point, just a few days ago schools in Florida were testing a new drone defense system against shootings. Between see-through backpacks, armed teachers, metal detectors, and other things you’d think it would be more easier to severely restrict firearm access to under 21 year olds and make the parents criminally liable if they are found to have facilitated access in any way in the wake of a shooting. But yeah i guess dipping into online conversations and immediately notifying both school officials and law enforcement is a good solution (/j)
• seagnson 1y ago
  This article really makes you think about how far is too far with school surveillance. It's scary to imagine kids getting in trouble just for being kids. The balance between safety and freedom is tricky, but arresting students seems extreme. Thanks for shedding light on this important issue.
• zahlman 1y ago
  > Among them is Tennessee, which passed a 2023 zero-tolerance law requiring any threat of mass violence against a school to be reported immediately to law enforcement. > ... > Gaggle’s CEO, Jeff Patterson, said in an interview that the school system did not use Gaggle the way it is intended. The purpose is to find early warning signs and intervene before problems escalate to law enforcement, he said. > “I wish that was treated as a teachable moment, not a law enforcement moment,” said Patterson. Of course they didn't do things as the Gaggle CEO describes. They were legally compelled not to. I love a good complaint about corporate meddling as much as anyone, but sometimes it really is the government to blame.
  • pavel_lishin 1y ago
    > The purpose is to find early warning signs and intervene before problems escalate to law enforcement, he said. The purpose of a system is what it does.
    • zahlman 1y ago
      > The purpose of a system is what it does. If you take this view, then a fair assessment recognizes that "the system" in this case consists of more than just the Gaggle software.
      • potato3732842 1y ago
        Of course. The school, the police, the local and state governments that made them what they are are all culpable here.
    • senderista 1y ago
      This faux-profound cliche contributes absolutely no real insight to any discussion.
      • tremon 1y ago
        On the contrary, it short-circuits the usual knee-jerk apologetic responses that seek to defend a system's bad effects with (unrealized) good intentions. Killing thought-terminating cliches early allows for a deeper discussion.
• puppycodes 1y ago
  God this horrifying, i'm really glad i did not grow up in this type of environment. Why people involve police in this is beyond me. Also how is that even a legal arrest? What crazy judge would write an arrest warrant for that? IANAL but sounds like an illegal warrantless arrest.
• paradox460 1y ago
  An important, if unfortunate, lesson to learn: you are being spied on, by people who demand your trust and give you none in return Back when I was in school the web filtering got so draconian those of us in the know started keeping Tor browser on a thumb drive, and carrying it to whatever computer we were using. Some went as far as to reboot from a Linux image on said thumb drive. That was nearly 20 years ago, can't imagine it's any better now