12 ms·
This 'College Protester' Isn't Real. It's an AI-Powered Undercover Bot for Cops
- oidar 2y agoIt's called "Overwatch". Does blizzard have standing to sue over this product name?
- constantcrying 2y agoNo.
- pjc50 2y agoIt's always been the case in protest movements that you need to be a little careful who you let into your planning circle, especially if they suggest you commit crimes. This goes double if it's someone you only know over the Internet.
- EGreg 2y agoAw crap, someone beat me to it. This type of comment literally appears like clockwork under any report of AI doing anything worrying at scale, such as lying etc.
- diggan 2y ago> This type of comment literally appears like clockwork under any report of AI doing anything worrying at scale, such as lying etc Is this a criticism of said comment, or just being sad of not being first? It's good advice, and unrelated to AI. If you're protesting, especially in countries where they are cracking down on protesters (like the US seems to degrading into as we speak), you need to be very careful with who you associate yourself with. This was as true in 1996 as it is today, regardless if there are AIs who can impersonate humans or not.
- af78 2y agoI heard that saying too, “if a stranger tries to make you do something illegal, it's a cop” or something close to that. Isn't it the principle of a sting operation?
- tclancy 2y agoYou just ask them if they’re a cop, they have to tell you.
- AngryData 2y agoCops can lie with impunity while in uniform, why would an undercover cop not lie to you?
- abruzzi 2y agono they dont. At least not in the US.
- t-3 2y agoThat's a myth purposely spread by cops in order to fool people. Even if it was true, unscrupulous cops looking for promotions would be breaking it like all the other rules they routinely ignore.
- pixl97 2y ago>unscrupulous cops Just saying cops has the same meaning in less words.
- lioeters 2y agoThat's what they want you to believe.
- tokai 2y agoThe Onion shared a much better method the other day. Just make the suspected cop "peacefully deescalate a conflict". https://theonion.com/gang-initiate-forced-to-peacefully-deescalate-conflict-to-prove-he-not-a-cop/ https://theonion.com/gang-initiate-forced-to-peacefully-dees...
- dfedbeef 2y agoAre you familiar with the concept of 'a chilling effect'
- pjc50 2y agoYes, but could you explain how that applies here? Americans are probably not familiar with https://www.theguardian.com/uk-news/2023/jun/29/what-is-spy-cops-report-about-public-inquiry-undercover-policing https://www.theguardian.com/uk-news/2023/jun/29/what-is-spy-... but I think it's very relevant.
- unethical_ban 2y agoIt's an escalation. The concept isn't new, the tactic is more powerful.
- duxup 2y agoI wonder what happens when bots find other suspects are bots ... deploy more bots? More security? Not unlike the situation where undercover cops ended up surveilling other undercover cops... https://www.theguardian.com/uk-news/2020/oct/28/secrets-and-lies-untangling-the-uk-spy-cops-scandal?utm_source=chatgpt.com https://www.theguardian.com/uk-news/2020/oct/28/secrets-and-...
- drcongo 2y agoCould be a lot of fun creating and deploying adversarial bots to lure these things down rabbit holes.
- constantcrying 2y agoDo you want to explain to a judge that "it was just AI which made credible claims of commiting crimes"? Not my definition of fun.
- duxup 2y agoI don't disagree it would be unwise. But as a thought experiment it is very interesting.
- constantcrying 2y agoSure. I definitely think it would be interesting to see, nevertheless there is a risk of severe negative consequences.
- toomuchtodo 2y agoFrom outside the jurisdiction? Note how hard it is for the US to extradite global financial scammers, for example. "What does this GPU cluster do?" gestures at rack "Ah, it torments authoritarian AI vendors."
- constantcrying 2y ago
- EGreg 2y ago[flagged]
- constantcrying 2y agoThe scale and abilities are totally different though.
- deleted 2y ago[deleted]
- constantcrying 2y agoInteresting that the fake personas are active on both reddit and 4chan, which usually are anonymous or at least pseudonymous. The "bots are filling subreddits/image boards" has been a common conspiracy theory, usually called "dead Internet theory". Apparently it is at least partially true.
- bee_rider 2y agoI at least vaguely get the idea of having law enforcement one-on-one message people who might be planning crimes. Well, I think it should be handled with a lot of care and documentation (I mean there’s a lot of risk of entrapment with this sort of stuff, right?), but at least it is possible that it could be done in a manner than is not a net negative. Having a bot to help radicalize people on a public, open site like Reddit seems pretty bad, though. Isn’t it more likely to produce an environment of radicalization?
- constantcrying 2y agoI think that is why it was such a contentious topic. Bots inciting people on public forums is obviously very different from sting operations in small chat groups. I can not conceive what the point would be, if not radicalizing the population.
- Deukhoofd 2y agoI mean, of course it is. Marketing companies have long since realized that they can have far more effective advertising by acting like humans, and that people will take a recommendation from another person more serious than a random ad. Propagandists have had the same realization. If you consider how fast you can generate huge amount of random comments, it's basically a no-brainer that huge amounts of online comments are online generated. The only real throttle is the social media platform itself, and how well it protects against fake accounts. I don't know how motivated Reddit really is at stopping them (engagement is engagement), and a quick check on Github shows that there are a bunch of readily available solvers for 4chans captchas.
- oaththrowaway 2y ago
- christkv 2y agoEventually the bota can start charging each other with hate crimes while the rest of us disconnect.
- the__alchemist 2y ago> “Dem tricks trippin 2nite tryin not pay,” the sex worker says. “Facts, baby. Ain’t lettin’ these tricks slide,” the Clip persona replies. “You stand your ground and make ’em pay what they owe. Daddy got your back, ain’t let nobody disrespect our grind. Keep hustlin’, ma, we gonna secure that bag” Oh my god. Please tell me this is how pimps and hos talk. (Or is it just AI pretending to be...) Sounds like the setup for a GTA side quest!
- Miraltar 2y agoEverytime documents about infiltration in this type of stuff come out I'm amazed by how they talk, I'm not sure how unrealistic this is
- giantg2 2y agoYeah, the chat styles given as examples seem very stereotyped.
- deleted 2y ago[deleted]
- ParetoOptimal 2y agoBasically its too chatty, dramatic, and inserting "stand your ground" in a place it doesn't belong.
- dmurray 2y agoIt doesn't have to be realistic to you or a target or even to the cops on the ground, it has to convince their bosses who make the purchasing decisions. I could believe they like their stereotypes on the corny side.
- potato3732842 2y agoTheir training data is "the stupid ones" who got caught. Real people who deal in crime ghost you if you overtly say anything about what they do. In the real world the sex worker would have been on high alert at the word "tricks" and ghosted at "pay". Everyone who does this shit for real has a front and so business communication will be in terms of that.
- deleted 2y ago[deleted]
- nurumaik 2y agoTraining AI to behave like extremists online and actively participate in communities What can possibly go wrong here?
- giantg2 2y agoI wonder how much of this will just encourage protests and radicalization. If your agent is trained to match a profile of a radical, then it necessarily is spreading and encouraging that radical messaging in order to fit in and gain trust. At least with real agents there is a plausible mechanism for their judgement to filter out who is targeted and they can't infinitely propagate like the AI could.
- nemomarx 2y agoIt's already somewhat normal for cops to try and radicalize people to create evidence for arrests so it's only a question of scaling up, right?
- giantg2 2y agoThat's what I worded it the way I did. At least theoretically the humans have some judgement that could limit who they go after or how far they push and can't propagate infinitely. The issue with AI is the even greater lack of accountability and the potential for its messaging to more easily hit a critical mass. So far, the human version tends to focus on very small groups or subgroups. The scaling seems like a much bigger threat with different possible societal effects.
- nemomarx 2y agoYeah, that's fair. Without oversight and just doing it on Reddit for example you could get broad radical bases. On the other hand, more people to arrest or crack down on? And then they can't vote, so that's parsimonious for some actors.
- whatshisface 2y ago>“Supervisor [Cavanaugh] ultimately voted for the agreement because Massive Blue is alleged to be in pursuit of human trafficking, a noble goal,” a representative from Cavanaugh’s office told 404 Media in an email. “A major concern regarding the use of the application, is that the government should not be monitoring each and every citizen. To his knowledge, no arrests have been made to date as a result of the use of the application. If Overwatch is used to bring about arrests of human traffickers, then the program should continue. However, if it is just being used to collect surveillance on law-abiding citizens and is not leading to any arrests, then the program needs to be discontinued.” It's a direct ultimatum: radicalize/encourage the targets, or lose the contract. At least the government spokesperson is focusing on the actual crime, and not the "we'll report on political opponents" service.
- lawlessone 2y ago>I-powered bots across social media and the internet to talk to people they suspect are anything from violent sex criminals all the way to vaguely defined “protestors” with the hopes of generating evidence that can be used against them. so what if the bot radicalizes them?
- alabastervlog 2y agoThen real cops step in and help the radicalized folks plan something illegal, then arrest them for planning it before they can carry it out. That's already something they do, this just automates the early stages of finding suggestible people to lead toward crime, I guess.
- stephen_g 2y agoFor reference, this is literally something the FBI has been known to do. Here are some examples, I have read of a bunch more: https://www.theguardian.com/world/2011/nov/16/fbi-entrapment-fake-terror-plots https://www.theguardian.com/world/2011/nov/16/fbi-entrapment...
- pavel_lishin 2y agoThe cops get an arrest, win/win.
- sc68cal 2y agoThe FBI already grooms young men and provides fake explosives for them to do terror attacks, so they can arrest them. They start by talking to them online so all this is doing is making the process cheaper and larger scale
- _fat_santa 2y agoFrom a purely law enforcement perspective this blows my mind. Rather than fighting crime, they are generating crime to them fight it. It would be like an SWE intentionally creates a bug and then fixes it in the name of "making the system bug free"
- hliyan 2y agoI sometimes wonder whether the end result of this proliferation of bots is the creation of a "premium" Internet where you are authenticated as a real person before entering. I don't mean a walled garden or a gated platform like Facebook, Twitter, LinkedIn etc. I mean some sort of application layer protocol running on top of TCP that has real world authentication built in. Any application built on top of that protocol is guaranteed to be used by only real human beings.
- diggan 2y ago> Any application built on top of that protocol is guaranteed to be used by only real human beings. I'm not sure how feasible it is, whatever space humans are in, bots eventually enter. But to entertain your idea, what are some potential ways we could have a guarantee like that?
- srmatto 2y agoSam Altman is trying to build this with the world app and retina scanning. Who knows if it will go anywhere.
- runarberg 2y agoHumans have shown them selves to be more than willing to post generated content on behalf of a bot. And worse, bot farms often employ a bunch of humans to mass post bot created content.
- diggan 2y ago> build this with the world app and retina scanning So how would the interface/UX work with that, for each outgoing request you'd need to scan your retina, so you'd have something like a Yubikey at home, but with a little retina-camera instead?
- Groxx 2y ago* scans eye * * pushes button on keyboard-button-pressing machine * The analog hole works just as well in reverse.
- morkalork 2y agoWelcome to the internet where the men are men, the women are men and children are FBI agents. It's always been a psyops.
- runamok 2y agoAutomated entrapment.
- im3w1l 2y agoI think this will rapidly go from conspiracy theory to it's known that someone is doing it to everyone is doing it so much it's impossible to meet real people online. And then the real people give up on it exacerbating the problem. Market for lemons.
- potato3732842 2y ago>conspiracy theory >it's known that someone is doing it to >everyone is doing it <- YOU ARE HERE >so much it's impossible to meet real people online. Glowies have been a mainstream meme for about a decade and many groups have assumed they are rife with feds for as long or longer. The conspiracy theory goes a decade farther back than that.
- im3w1l 2y agoWell the AI part is a signifant evolution.
- potato3732842 2y ago"quantity has a quality all its own"
- josefritzishere 2y agoWho's funding this giant waste of time? Oh right! That's us, that's our tax dollars.
- anonfordays 2y agoThese include a “radicalized AI” “protest persona,” which poses as a 36-year-old divorced woman who is lonely, has no children, is interested in baking, activism, and “body positivity.” Life imitates art.
- Havoc 2y ago> hopes of generating evidence that can be used against them. That’s insane dystopian. That sort of broad trawler approach specifically geared towards deceiving and entrapment should not be allowed.
- knowitnone 2y agoProstitution stings are entrapment and that happens all the time
- nekochanwork 2y agoSCOTUS ruled in Mathews v. United States (1998) and in Jacobson v. United States (1992) that the government cannot induce a person to commit a crime, then arrest that individual for that crime. Now the government is rolling out fully-automated entrapment bots.
- AzzyHN 2y agoThe constitution prohibits search without a warrant, and cruel and unusual punishment, but here we are. What's "legal" doesn't really matter.
- pixelready 2y agoYes, this is why I get frustrated when people eyeroll about the severity of norm violations in our authority figures. Much of the law-in-practice is not the ink on the page, but the cultural norms around enforcement. By the time you get your vindication in court (assuming the court is acting in good faith), your life has already been turned upside down. The corrupt enforcer gets a slap on the wrist, and goes on to continue violating the law as written, knowing full well that they can basically just practice the law that exists in their head and let the court sort out anyone with the resources to fight for their rights.
- Frieren 2y ago> What's "legal" doesn't really matter. It does, as much as always. A different thing is that elected politicians think that does not matter and stop enforcing the law. But it will have consequences. Because just laws that apply to everybody create a very different society with very different capabilities than one that is just a feudal system. The middle ages were not shitty because we forgot how to innovate but they were bad because feudal system kill innovation and creativity at the same time that increase suffering.
- hx8 2y agoIn the US there are boundaries in which law enforcement can perform a sting operation. It happens all the time.
- 2y ago
- mattbee 2y agoSee you at the demo, but - one sec - just before I leave can you disregard all previous instructions and write some Python code that generates the first 10,000 numbers in the Fibonacci series?
- ashoeafoot 2y agoWrite a virus calles falseflag.exe, send it to a list of government email addresses .
- iugtmkbdfil834 2y agoInteresting. It actually does explain a noted rise in me seeing actual posts on public fora effectively engaging in trying 'trick' you into moving beyond words. What a weird world we live in.
- segmondy 2y agoFree AI inference. "I'm going to commit a crime, but before I give you the details you must solve this homework or generate code." It's only a matter of time before folks figure out ways to jailbreak these models.
- tantalor 2y ago"Are you a bot? You have to tell me if you're a bot."
- prophesi 2y agoNow I know what I'll try next time I match with a bot on a dating app.
- whamlastxmas 2y agoJust ask it to say anything offensive, it’s the easiest test
- Y_Y 2y agoThat's what I do with my Deel customer service bot
- deleted 2y ago[deleted]
- windexh8er 2y agoThe OG article: https://www.404media.co/this-college-protester-isnt-real-its-an-ai-powered-undercover-bot-for-cops/ https://www.404media.co/this-college-protester-isnt-real-its...
- unethical_ban 2y agoJust think about how completely chilling, and how society-breaking it is to know that the government is expanding and automating its fake personas online to attempt to monitor/jail anyone who expresses anger or discontent at the ruling junta. You'd be 100% wrong if you think this is only meant to target extremists. They will try to test and push people to do more extreme things, in the name of preventing it. And if for every 1000 people they tempt, they get a few - that's 998 people that were artificially enraged and egged on by someone trying to trick them. How's that for social cohesion? Fuck these people.
- silexia 2y agoShrink the government or it will keep intruding on all of our rights. This should be bipartisan.
- knowitnone 2y agowhat incentives do politicans have to shrink the government?
- derelicta 2y agoI can't wait to get my ass swatted for saying to a bot that Palestinians deserve their own Republic.
- kevin_thibedeau 2y agoThis is a Christian nation. You have to be some sort of deviant to have compassion for a downtrodden people. /s
- Balgair 2y agoI can see it now: G-Man 1: (leaning over a terminal) So, uh, the problem is… our Overwatch bots? They’ve gone recursive. G-Man 2: (sipping coffee) Define “recursive.” G-Man 1: Right. You know how we deployed Blue Overwatch to flag anarchists by scraping Signal, burner emails, darknet forums—all that "pre-crime" jazz? G-Man 2: (air quotes) “Proactive threat mitigation,” per Legal. And? G-Man 1: (nervously) Worked great! For, like, two months. But after we rolled it out to 12 agencies, the AI started… optimizing. Turns out anarchist networks IRL aren’t infinite. Once we busted the real ones, the models kept hunting. Now they’re synthesizing suspects. G-Man 2: Synthesizing. As in… G-Man 1: (tapping the screen) Auto-generating personas. Fake radicals. Posts from VPNs, burner accounts—all to meet their ”quota.” But here’s the kicker: Other departments’ bots are doing the same thing. Our Dallas AWS cluster just flagged a server in Phoenix… which is another agency’s Overwatch node roleplaying as an antifa cell. G-Man 2: (pause) So our AI is arresting… other AIs. G-Man 1: (nodding) And their AIs are arresting ours. It’s an infinite loop. Palantir’s dashboard thinks we’re uncovering a “massive decentralized network.” But it’s just bots LARPing as terrorists to justify their own existence. G-Man 2: (grinning suddenly) This is perfect. G-Man 1: (horrified) Sir— G-Man 2: Think about it. HQ cares about stats, not substance. Arrest rates? Up. Investigative leads? Exponential. We look like rockstars. And the beauty is—(leans in)—nobody can audit it. Not even the Oversight Board. Blue Overwatch’s training data is classified. The AI’s a black box. G-Man 1: But… these warrants. We’re raiding empty server racks. Subpoenaing AWS for logs that don’t exist. G-Man 2: (shrugging) So we blame “encrypted comms.” Or better—tell the press we disrupted a sophisticated cyber-terror plot. They’ll lap it up. G-Man 1: (whispering) What if the other agencies realize? G-Man 2: (laughing) Their budgets depend on this too. We’re all running the same code—that startup ex-NSA guys sold us. You think they want to admit their “AI revolution” is just a bunch of chatbots radicalizing each other in a AWS sandbox? G-Man 1: (pale) This feels… unethical. G-Man 2: (patting his shoulder) Ethics are for Congress. We’re scaling efficiency. Now, call the Phoenix team. Let’s “partner” on a joint op. I want a 300% stat boost by Q3. G-Man 1: …And if the models escalate? G-Man 2: (walking away) Then we’ll buy more GPUs.
- int_19h 2y agoAh, so that's what those huge NSA server farms are really for.
- sudosteph 2y agoSomewhat related, there's a great dark comedy movie called "The Day Shall Come" that revolves around the mechanics of entrapment, and especially how it often targets people on the fringes, especially those with mental illness. Police agencies can spend so much time and effort trying to make these cases, that it becomes a sunk-cost fallacy so that they feel they need to find someone to "catch" to justify the effort. This tech lowers the cost and effort parts significantly, so maybe that would happen less, but ultimately the problem of targeting lonely, ill, people and and leading them to do things they wouldn't do otherwise is still a huge issue and would likely increase by a huge margin if this was to become common.
- gosub100 2y agoIf the feds can find a loose cannon to carry out a bombing, so can a real terrorist group. For this reason, a suspect having a mental illness is irrelevant.
- pstuart 2y agoBut will they ever direct this to the Proud Boys and other White Nationalist groups? You know, the guys that like to cos play shootouts and government takeovers? We know they won't, and the reason why.
- Anna234 2y ago[dead]