These days, AI surveillance systems are everywhere. They’re watching our roads, tracking our purchases at the grocery store, and even monitoring school bathrooms.
Yet those systems come with a harmful catch — they’re a little too good at flagging bad behavior, leading to a staggering number of arrests and involuntary detentions.
New reporting by the Milwaukee Independent found that schools deploying surveillance systems to watch for online chatter referencing violence have logged an astonishing number of positive hits. Many of them are obviously not actionable threats, though they can lead to involuntary detention all the same.
In one school district in Lawrence, Kansas, for example, an online safety monitoring tool called Gaggle flagged over 1,200 online incidents over a 10-month period. Yet two thirds of those alerts were nonissues, the Independent found.
While many false positives can be waved away, schools in states with zero-tolerance reporting laws like Tennessee or Florida are required to inform police officials about cases where students hint at violence in any form.
In Florida’s Polk County School District — where Gaggle flagged some 500 students over four years — this meant that at least 72 kids so far have been arrested or hospitalized involuntarily under the state’s Baker Act.
“A really high number of children who experience involuntary examination remember it as a really traumatic and damaging experience — not something that helps them with their mental health care,” Sam Boyd, an attorney with the Southern Poverty Law Center, told the Independent.
Many of those students don’t even realize their emails, online chats, and Google searches on school equipment are being monitored, let alone that they’ve done anything wrong. Tennessee mother Lesley Mathis, for example, told the Independent her eighth-grade daughter was arrested, interrogated, strip-searched, and held in jail for a night, over some teasing online.
Specifically, the student’s friends had heckled her about her “Mexican” complexion, even though she has a different ancestry. “On Thursday we kill all the Mexico’s [sic],” the eighth-grader quipped back.
Those comments are obviously very nasty — but it was clearly a bit of eighth-grade immaturity boiling over, not an actionable threat.
... continue reading