6 ms·

Disrupting deceptive uses of AI by covert influence operations

98 points by Jimmc414 2y ago

catchnear4321 2y ago
> Activity by a commercial company in Israel called STOIC, because technically we disrupted the activity, not the company. hmm. is this legal cya because a name was named? naming and shaming is a curious choice. taking care to specify what was disrupted is… well, it draws the eye. all others were called operations or groups. unlikely the groups were disrupted even if their operations were. no difference in pattern. just in categorization. company vs cell. what is the extra care for, exactly?
- giraffe_lady 2y ago
  Probably all of them are state affiliated but only this one affiliated with a state the US considers an ally?
  - thiagoharry 2y ago
    This is what I thought reading the article. To be fair, before reading the article I was expecting to find zero disrupted activities related with the west and its allies, but the fact that they disrupted that AI usage by Israel was a good surprise. However, I really doubt that US does not do the same, spreading propaganda in Internet, based on the fact that they employ and finance several radios, and newspapers around the globe with this purpose (ex: Radio Free Asia). Giving how the US government and tech companies are close, with backdoors planted in CISCO routers, and information collected from social media, I find hard to believe that OpenAI or any other US-based company is trusted to report and disrupt activities like these when performed by US government, or perhaps some closer allies.
    - giraffe_lady 2y ago
      Yes agreed with all of that. I'm sure this was handled... carefully.
- grooly 2y ago
  Meta also reported disrupting activity from the same company: https://www.reuters.com/technology/meta-identifies-networks-pushing-deceptive-content-likely-generated-by-ai-2024-05-29/ https://www.reuters.com/technology/meta-identifies-networks-...
  - kbigdelysh 2y ago
    Based on the Reuters article, The Israeli company STOIC put many ai-generated comments on social media posed from Jewish, African Americans and concerned citizens praising Israel's handling of the war in Gaza.
sturza 2y ago
The operations were: • A previously unreported operation from Russia, which we dubbed "Bad Grammar", operating mainly on Telegram and targeting Ukraine, Moldova, the Baltic States and the United States; • A persistent Russian threat actor posting content about Ukraine across the internet, known as "Doppelganger"; • A persistent Chinese threat actor posting content across the internet to praise China and criticize its critics, known as "Spamouflage"; • A persistent Iranian threat actor posting web content that supported Iran and criticized Israel and the US, known as the International Union of Virtual Media (IUVM); • A commercial company in Israel called STOIC, generating content about the Gaza conflict, and to a lesser extent the Histadrut trade unions organization in Israel and the Indian elections. We have nicknamed this operation "Zero Zeno" for the founder of the stoic school of philosophy, and to reflect the low levels of engagement that its various campaigns attracted.
- newzisforsukas 2y ago
  No US based operations targeting foreign or domestic entities? I guess they already have that in house?
  - red-iron-pine 2y ago
    we've been talking about GPT-fueled propaganda campaigns on HN since before COVID... so very likely in-house. Likely in-house for many of the other actors above; this may just be low-hanging fruit, or operations designed to hit specific demographics in specific countries.
  - fzzzy 2y ago
    Surely the US has already had this in house for at least a decade.
- deleted 2y ago
  [deleted]
mjburgess 2y ago
With OpenAI's breathless "fears" about AI safety, their anthropomorphising a chatbot interface, and so on.. one should be aware here that OpenAI isn't against deception, it just wishes to control its manner. Note also, that in the west corporate propaganda is permissible, whilst state propaganda is orwellian. It's no mystery then that much state propaganda appears under a corporate guise. If OpenAI were against corporate propaganda, much of its business model would vanish overnight.
- nottorp 2y ago
  [flagged]
- exe34 2y ago
  what matters is what the oligarchs control. in the east, it's the state. in the west, the corporations. democracy is just the ointment to help with the pain, for now. eventually it won't be needed.
- deleted 2y ago
  [deleted]
talldayo 2y ago
At this rate I'm convinced ChatGPT is the red mercury of digital threat campaigns. "Mr. President, Mr. President, come quick! We've reports that Chechen Separatists have began to delegate portions of their work to ChatGPT." "Excellent, then we have nothing to worry about."
vintermann 2y ago
> A previously unreported operation from Russia, which we dubbed Bad Grammar Good! Great that they don't give criminals cool names ... > Activity by a commercial company in Israel called STOIC, because technically we disrupted the activity, not the company. We nicknamed this operation Zero Zeno, for the founder of the stoic school of philosophy Never mind.
- deleted 2y ago
  [deleted]
mamonster 2y ago
Russian bots are targeting 9GAG? What year is it?
- deleted 2y ago
  [deleted]
- red-iron-pine 2y ago
  they're still pretty active on the *chan's, if I believe threat intelligence reporting. no surprise they're in 9GAG. also slashdot, hackernews, and various reddits. it call all be automated, and failing that, plenty of folks willing to do the "mechanical turk" approach and shitpost for pay.
  - mamonster 2y ago
    They are for sure on 4chan(AFAIK there is a pretty obvious propaganda thread that is updated daily), but I thought 9Gag was dead? Do people still get memes there?
    - jazzyjackson 2y ago
      maybe its just the dev environment, lacking good antibot mechanisms they can test new delpoyments without showing their hand to reddit etc
dekhn 2y ago
I have very little confidence that OpenAI has the technical expertise or company culture required to deal with APT (advanced persistent threats).
- ActionHank 2y ago
  Also, the article doesn't state it outright, but the implication is that they only stopped bad actors after they had already been using their services, with no indication of how long they were using them for.
- baxtr 2y ago
  I recommend anyone interested in the subject to study Meta’s Adversial Threat Reports. They started their counter operations a while back and regularly draft extensive reports [1]. Reading through those reports it becomes obvious how much effort this requires. I’m not sure how much effort OpenAI can put into this at all. [1] https://transparency.meta.com/de-de/metasecurity/threat-reporting/ https://transparency.meta.com/de-de/metasecurity/threat-repo...
  - lolinder 2y ago
    This is a slight tangent, but I would love to see a sentiment analysis graph of HN perceptions of Meta/Facebook over time for the past two years or so. Subjectively, my sense is that releasing Llama 2/3 gained them a lot of good will in this community and attitudes have generally become much more positive towards them as OpenAI's star has fallen. In terms of developer goodwill, they may be the real winners of this whole gold rush.
    - baxtr 2y ago
      I had a similar thought recently. They really do a good job on many fronts right now. Additionally the whole TikTok debate will help them a lot too.
    - ben_jones 2y ago
      Facebook, I mean Meta, has benefitted from the enshittification of the tech industry. Next to TikTok they’re better moralistically. Next to Sam Altman, I mean OpenAI, they’re better moralistically. Next to other companies doing layoffs they’re better moralistically (biggest exit packages?)
renegade-otter 2y ago
I argue here that sentient AI destroying the world is effectively fiction. Beware of the people, state actors, and those who get ultra rich on the hype and then flood our politics with money: https://renegadeotter.com/2024/04/22/artificial-intelligence-skynet-is-not-coming-to-kill-you.html https://renegadeotter.com/2024/04/22/artificial-intelligence...
cheald 2y ago
Given that any of these bad actors can spin up local inference infrastructure to run their operations pretty trivially, what practical effect does this have beyond just "hey, look, we're doing something"?
- thegrim33 2y ago
  Exactly .. every state actor will already have their own local infrastructure for all this, there's no way they're actually dependent on using OpenAI. Most likely this is just a disinfo campaign - "hey look guys, we caught some bad covert actors, so there's totally no nation states making massive use of LLMs to manipulate online discourse, look we caught them, everything is secure, don't worry about it". The funny part is, if it is all a disinfo campaign, then the very resources behind the campaign will downvote and attack this post and try to silence/discredit such voices. Or, maybe I'm just completely wrong and I'll get downvoted for having dumb beliefs. There's actually very little way to tell the difference between the two scenarios. In addition, does the act of me pointing out that malicious actors would want to downvote this now make it less likely for them to actually do so, as it would show their hand? You can recurse down these rabbit holes forever. Fun stuff.
  - ImPostingOnHN 2y ago
    Your comment violates the site rules (commenting on downvotes) and cheapens the conversation with the same unfalsifiable conspiracy victim rhetoric, so you should expect to see downvotes for that alone, even if none of the rest of the comment existed.
- ben_w 2y ago
  Even if you're not skilled enough to stop the FSB from misusing your services, it's worth putting in the effort to stop the next Jim Jones from doing so. And learning to stop a Jim Jones, even knowing that you need to, may help you stop you from being named by the UN as complicit in a genocide, the way the UN explicitly named Facebook as having played a "determining role" in the Rohingya genocide: https://web.archive.org/web/20220217165239/https://www.reuters.com/article/us-myanmar-rohingya-facebook-idUKKCN1GO2PN https://web.archive.org/web/20220217165239/https://www.reute... Of course, if you want to prevent even the most capable of intelligence agencies from spinning up their own local LLMs at all, then at a minimum you can't ever publish the model weights themselves — a solution which itself is the cause of around 50% of the criticisms people around here make of OpenAI.
nerdjon 2y ago
Something about them caring about this bothers me a bit. Like, yeah ok it is a good thing that they are trying to curtail this abuse. However, it kinda feels like we are just delaying the inevitable? With open source models like LLAMA, detecting and doing anything about this will become impossible in a year or 2 (in reality its already possible). It just feels very, "look at us doing this good thing but ignore what we unleashed on the world" so please pat us on the back here. Maybe I am just pessimistic, but I am not about to give OpenAI any praise here for this when they are the reason we are in the current situation.
- UberFly 2y ago
  My feelings exactly. As usual it feels like lawyers are saying "just do this even though it's BS so in a future court room you can bring it up". You're not pessimistic, just realistic.
- ADeerAppeared 2y ago
  > It just feels very, "look at us doing this good thing but ignore what we unleashed on the world" so please pat us on the back here. It is that. These AI firms have been very vocal about their "ends justify the means" attitude. (Which is doubly damning considering how little these firms care about any purported AI safety for their AGI dreams. Months of "AI is going to kill us all" while they couldn't even be arsed to stop using CSAM-contaminated training data, nevermind using quality training data to ensure alignment) And "Foreign state actors generated some propaganda texts" is very small as AI-problems in the here-and-now go.
- Lockal 2y ago
  You are saying as if automatic detection of misinformation and brigades was ever possible before LLMs, or it is possible now. Only now, when people started to massively notice things like https://news.ycombinator.com/item?id=40444872 https://news.ycombinator.com/item?id=40444872 OpenAI publicly executed few accounts. By the way, they didn't even say exactly how many. Hundreds new will be created in 3.. 2.. 1..
S0y 2y ago
>We’ve terminated accounts linked to covert influence operations; >no significant audience increase due to our services. I can't help but find this extremely funny. you mean to tell me that massive operations trying to use your service didn't actually gain anything from using your service? OpenAI wasting time and resources shutting them down and the bad actors wasting time and resources using chatGPT no benefit. It just reads like a massive waste of time for everyone involved.
- quinncom 2y ago
  Hilarious, I love this take. I’d guess that “covert ops” using ChatGPT are amateurs, by definition (because it’s clearly not covert). Surely, the serious operators would use offline local LLMs.
- mensetmanusman 2y ago
  You’re thinking too much like an engineer. News like this teaches people that they are clearly ahead in capabilities far beyond any government or other corporation. Anytime they can get good publicity like this is good for their business.
- dialup_sounds 2y ago
  I get the sense that they would be using ChatGPT to save the time and resources required to produce content, rather than expecting it to produce better results.
  - yencabulator 2y ago
    Save time -> can do more -> can gain larger audience.
barryrandall 2y ago
How effective were OpenAI's safety/security measures? With these confirmed bad actors, what percentage of their usage attempts were disrupted by automatic safety measures? Of the blocked attempts, is there any evidence that the influence actor successfully circumvented those protections?
- asne11 2y ago
  > we repeatedly observed cases where our models refused to generate the text or images that the actors asked for So the protections were bypassed at least some of the time. Many of these are demonstrably trivial to bypass.
mensetmanusman 2y ago
YouTube is already completely full of well-read bots supporting Russian attacks on Ukraine.
- llamaimperative 2y ago
  "Couldn't get any worse!" - Things people say right before it gets worse
ganoushoreilly 2y ago
While likely true, it's also helpful for pushing narratives requiring legislation that OPENAI just happens to be grandfathered into. I suspect more of these claims as the government forms a stronger opinion / rules around AI proliferation.
hex4def6 2y ago
"Our Usage Policies prohibit the use of our services to deceive or mislead others. This includes deceiving people about the authorship or source of content generated using our models, by engaging in activities such as: ... Creating automated systems that falsely present as being a real person, or don’t disclose to people that they are interacting with AI (unless it’s obvious from the context)." Suspiciously absent from the list of takedowns is anything on Reddit. Given the amount of obviously-chatGPT puppets on there, I'm somewhat 'surprised'. Also a good reminder that if you're using ChatGPT for anything, they can and will examine the traffic and contents of your messages / replies. I wonder if in the future they may offer a "service" to content providers. Say you're a reddit / forum / etc. You get supplied an API that you can run queries through that match any content that may have been generated through their service. You can then tag those accounts as bots. Of course, such an API will have costs associated with it...
- hermitdev 2y ago
  > I wonder if in the future they may offer a "service" to content providers. Say you're a reddit / forum / etc. You get supplied an API that you can run queries through that match any content that may have been generated through their service. You can then tag those accounts as bots. Of course, such an API will have costs associated with it... I'm not a lawyer, but this sounds like racketeering to me. Basically extorting "protection" money from sites.
TillE 2y ago
> “You can generate the content, but if you don't have the distribution systems to land it in front of people in a way that seems credible, then you're going to struggle getting it across,” The clearest example of a foreign influence operation I can remember was during the annexation of Crimea, when there was a flood of pro-Russian comments on Reddit. Thanks to the voting system and the centralized discussion threads, it's a platform that's relatively easy to manipulate. It's much harder to gain visibility on social media platforms where you choose which accounts to follow.
- timschmidt 2y ago
  > [reddit is] a platform that's relatively easy to manipulate. For all sides, apparently. https://www.youtube.com/watch?v=V7GtYaruTys https://www.youtube.com/watch?v=V7GtYaruTys
082349872349872 2y ago
We were hearing a little while back that it had become exceedingly difficult to concoct diplomatic cover for intelligence agents, in a world full of socials. Does the availability of AI mean that (as long as you own, or at least can retcon, several socials?) spinning up entire innocuous provenance networks for fresh identities is once again a push-button solved problem?
- rurp 2y ago
  It might help against casual checks, but a large web of invented social accounts will be uncoverable by any serious actor. If there's no corroborating trail outside of social media for any of those accounts it will be very possible to sniff out the fakery.
- tivert 2y ago
  > We were hearing a little while back that it had become exceedingly difficult to concoct diplomatic cover for intelligence agents, in a world full of socials. > Does the availability of AI mean that (as long as you own, or at least can retcon, several socials?) spinning up entire innocuous provenance networks for fresh identities is once again a push-button solved problem? AI doesn't defeat biometrics. I'd bet the problem with social media for intelligence agents is it allows facial recognition to uncover their true identity. If the search uncovers a real profile and a fake AI generated one with fake AI generated friends, it's still fishy to an adversary. Also if you're an agent and have gone to a country under one name, you can never go back under a different name without being trivially caught.
- djhn 2y ago
  There’s plenty of people in Europe in the 30-50 age bracket that aren’t on any social media. Many people from extremely wealthy families or careers in finance, consulting, medicine, law, law enforcement, military and the judicial branch came of age before ubiquitous social media and had already picked a career path where being discoverable is more of a liability than a benefit. No Linkedin, no facebook, no twitter, no instagram. Maybe a faceless fake name lurking account. Hard to say whether there are as many people in younger age groups who are like that.
nullc 2y ago
I was recently a defendant in a a set of lawsuits by a scammer seeking hundreds of billions of dollars in damages, using the courts as a weapon to shut down people who were in the way of his confidence game. The cumulative costs incurred by his opponents ended up well over ten million £‎. After a five week trial the court found against him in every respect, concluding his evidence was forged, that his actions constituted a fraud upon the court, etc. Relevant to this thread is that partly through the process he began to use ChatGPT. He appeared to use it in four ways: To create forgeries that he didn't have the technical skills to create (at least not on a short timeframe) with e.g. ChatGPT walking him through the technical steps, to concoct excuses and explanations, and to flood out his opponents with hundreds of pages of ChatGPT authored testimony that expended our resources just reading the material. He also used it to suggest legal strategy, which didn't cause us any problems and probably worked in our favor as ChatGPT's suggestions on that front appear to have been uniformly idiotic. Though he concealed his earlier usage, claiming to have lost access to his prior account, all in all his ChatGPT activity from around September 2023 to January 2024 amounted to some 22 million lines of text. We were extremely fortunate that his usage of ChatGPT began after his litigation began and that he made numerous errors that made his usage unambiguous. Had he been using it all along and less obviously our cost of defending the claim may have been increased dramatically-- even at his current usage had we been funding our defense entirely out of our own pockets his abuse of chatgpt may well have been the straw that bankrupted us before we could secure our victory. I think the us of LLMs as a tool in crime is interesting. For many negative uses of LLMs like spamming you can counter that people could just hire humans for the task so the LLM is more a question of cost or degree. But when committing a crime anyone you hire is a party that could extort you or whistleblow. We suspect our con made as many errors as he did with his forgeries in part due to his limited ability to enlist third parties to vet them. The potential for productive, honest, usage of the technology in this case appeared pretty limited relative to the potential for abuse. Some of the abuse could be addressed through improvements in standard practices like filing length limits, but its less clear to me how abuse like using it as a confidential forgery aid can be addressed. I certainly don't think it can be solved with safety approaches, since the interaction never needs to tell the LLM the true purpose. The same doesn't work for a human forgery aid because their access to outside information and potential for being questioned would make it much harder to avoid the risk that they discover that they're being used as a tool for fraud.
beretguy 2y ago
All this AI stuff is a godsend to the oppressive regimes. Makes their jobs a lot easier, effective, more believable, etc. > “These operations may be using new technology, but they're still struggling with the old problem of how to get people to fall for it,” said Ben Nimmo Even if true it’s just a matter of time before people will have absolutely no idea what truth really is.
- wruza 2y ago
  Makes their jobs a lot easier, effective, more believable, etc. Does it? Troll factories with authentic humans exist for at least a decade. They cost dirt cheap compared to a regime’s budget. There’s no problem with creating a network of bs, the problem is how to make it relevant. people will have absolutely no idea what truth really is We already have no idea. The only things that are true are fundamental human motivations. You can’t miss too far starting from there. Once you get that point of view, everything becomes so clear, despite missing details.
Lockal 2y ago
"Influence operations": Oh no! We were removed! What should we do? Oh, wait, there is "Sign up" button
dralley 2y ago
There was some interesting discussion about this on reddit: > In Ukraine this is a very known problem, to the point when there are jokes about those bots , cause they often post texts translated to Ukrainian from Google translated russian text. For example a year ago or so there was a meme "I can't stand this hellish flour" , which was created cause Google translator translated "hellish torment" from russian to "hellish flour" in Ukrainian , cause 'flour" in Ukrainian written exactly the same as "torment" and 'flour' in russian, which isn't the case in Ukrainian language (well , kinda the case , but Google translator translated word that has two meanings in the word that means only flour) https://www.reddit.com/r/europe/comments/1d4j1oo/openai_has_busted_two_russian_propaganda/ https://www.reddit.com/r/europe/comments/1d4j1oo/openai_has_...
Qem 2y ago
Israeli too. Curious they left it out of the headline.
- dralley 2y ago
  Headlines can only be so long. Other outlets (e.g. The Guardian) left the Chinese out of the headline, and nearly all of them (and yourself) left out the Iranians.

98 points by Jimmc414 2y ago

> Activity by a commercial company in Israel called STOIC, because technically we disrupted the activity, not the company. hmm. is this legal cya because a name was named? naming and shaming is a curious choice. taking care to specify what was disrupted is… well, it draws the eye. all others were called operations or groups. unlikely the groups were disrupted even if their operations were. no difference in pattern. just in categorization. company vs cell. what is the extra care for, exactly?

giraffe_lady 2y ago
Probably all of them are state affiliated but only this one affiliated with a state the US considers an ally?
- thiagoharry 2y ago
  This is what I thought reading the article. To be fair, before reading the article I was expecting to find zero disrupted activities related with the west and its allies, but the fact that they disrupted that AI usage by Israel was a good surprise. However, I really doubt that US does not do the same, spreading propaganda in Internet, based on the fact that they employ and finance several radios, and newspapers around the globe with this purpose (ex: Radio Free Asia). Giving how the US government and tech companies are close, with backdoors planted in CISCO routers, and information collected from social media, I find hard to believe that OpenAI or any other US-based company is trusted to report and disrupt activities like these when performed by US government, or perhaps some closer allies.
  - giraffe_lady 2y ago
    Yes agreed with all of that. I'm sure this was handled... carefully.
grooly 2y ago
Meta also reported disrupting activity from the same company: https://www.reuters.com/technology/meta-identifies-networks-pushing-deceptive-content-likely-generated-by-ai-2024-05-29/ https://www.reuters.com/technology/meta-identifies-networks-...
- kbigdelysh 2y ago
  Based on the Reuters article, The Israeli company STOIC put many ai-generated comments on social media posed from Jewish, African Americans and concerned citizens praising Israel's handling of the war in Gaza.

sturza 2y ago

The operations were: • A previously unreported operation from Russia, which we dubbed "Bad Grammar", operating mainly on Telegram and targeting Ukraine, Moldova, the Baltic States and the United States; • A persistent Russian threat actor posting content about Ukraine across the internet, known as "Doppelganger"; • A persistent Chinese threat actor posting content across the internet to praise China and criticize its critics, known as "Spamouflage"; • A persistent Iranian threat actor posting web content that supported Iran and criticized Israel and the US, known as the International Union of Virtual Media (IUVM); • A commercial company in Israel called STOIC, generating content about the Gaza conflict, and to a lesser extent the Histadrut trade unions organization in Israel and the Indian elections. We have nicknamed this operation "Zero Zeno" for the founder of the stoic school of philosophy, and to reflect the low levels of engagement that its various campaigns attracted.

newzisforsukas 2y ago
No US based operations targeting foreign or domestic entities? I guess they already have that in house?
- red-iron-pine 2y ago
  we've been talking about GPT-fueled propaganda campaigns on HN since before COVID... so very likely in-house. Likely in-house for many of the other actors above; this may just be low-hanging fruit, or operations designed to hit specific demographics in specific countries.
- fzzzy 2y ago
  Surely the US has already had this in house for at least a decade.
deleted 2y ago
[deleted]

mjburgess 2y ago

With OpenAI's breathless "fears" about AI safety, their anthropomorphising a chatbot interface, and so on.. one should be aware here that OpenAI isn't against deception, it just wishes to control its manner. Note also, that in the west corporate propaganda is permissible, whilst state propaganda is orwellian. It's no mystery then that much state propaganda appears under a corporate guise. If OpenAI were against corporate propaganda, much of its business model would vanish overnight.

nottorp 2y ago
[flagged]
exe34 2y ago
what matters is what the oligarchs control. in the east, it's the state. in the west, the corporations. democracy is just the ointment to help with the pain, for now. eventually it won't be needed.
deleted 2y ago
[deleted]

talldayo 2y ago

At this rate I'm convinced ChatGPT is the red mercury of digital threat campaigns. "Mr. President, Mr. President, come quick! We've reports that Chechen Separatists have began to delegate portions of their work to ChatGPT." "Excellent, then we have nothing to worry about."

vintermann 2y ago

> A previously unreported operation from Russia, which we dubbed Bad Grammar Good! Great that they don't give criminals cool names ... > Activity by a commercial company in Israel called STOIC, because technically we disrupted the activity, not the company. We nicknamed this operation Zero Zeno, for the founder of the stoic school of philosophy Never mind.

deleted 2y ago
[deleted]

mamonster 2y ago

Russian bots are targeting 9GAG? What year is it?

deleted 2y ago
[deleted]
red-iron-pine 2y ago
they're still pretty active on the *chan's, if I believe threat intelligence reporting. no surprise they're in 9GAG. also slashdot, hackernews, and various reddits. it call all be automated, and failing that, plenty of folks willing to do the "mechanical turk" approach and shitpost for pay.
- mamonster 2y ago
  They are for sure on 4chan(AFAIK there is a pretty obvious propaganda thread that is updated daily), but I thought 9Gag was dead? Do people still get memes there?
  - jazzyjackson 2y ago
    maybe its just the dev environment, lacking good antibot mechanisms they can test new delpoyments without showing their hand to reddit etc

dekhn 2y ago

I have very little confidence that OpenAI has the technical expertise or company culture required to deal with APT (advanced persistent threats).

ActionHank 2y ago
Also, the article doesn't state it outright, but the implication is that they only stopped bad actors after they had already been using their services, with no indication of how long they were using them for.
baxtr 2y ago
I recommend anyone interested in the subject to study Meta’s Adversial Threat Reports. They started their counter operations a while back and regularly draft extensive reports [1]. Reading through those reports it becomes obvious how much effort this requires. I’m not sure how much effort OpenAI can put into this at all. [1] https://transparency.meta.com/de-de/metasecurity/threat-reporting/ https://transparency.meta.com/de-de/metasecurity/threat-repo...
- lolinder 2y ago
  This is a slight tangent, but I would love to see a sentiment analysis graph of HN perceptions of Meta/Facebook over time for the past two years or so. Subjectively, my sense is that releasing Llama 2/3 gained them a lot of good will in this community and attitudes have generally become much more positive towards them as OpenAI's star has fallen. In terms of developer goodwill, they may be the real winners of this whole gold rush.
  - baxtr 2y ago
    I had a similar thought recently. They really do a good job on many fronts right now. Additionally the whole TikTok debate will help them a lot too.
  - ben_jones 2y ago
    Facebook, I mean Meta, has benefitted from the enshittification of the tech industry. Next to TikTok they’re better moralistically. Next to Sam Altman, I mean OpenAI, they’re better moralistically. Next to other companies doing layoffs they’re better moralistically (biggest exit packages?)

renegade-otter 2y ago

I argue here that sentient AI destroying the world is effectively fiction. Beware of the people, state actors, and those who get ultra rich on the hype and then flood our politics with money: https://renegadeotter.com/2024/04/22/artificial-intelligence-skynet-is-not-coming-to-kill-you.html https://renegadeotter.com/2024/04/22/artificial-intelligence...

cheald 2y ago

Given that any of these bad actors can spin up local inference infrastructure to run their operations pretty trivially, what practical effect does this have beyond just "hey, look, we're doing something"?

thegrim33 2y ago
Exactly .. every state actor will already have their own local infrastructure for all this, there's no way they're actually dependent on using OpenAI. Most likely this is just a disinfo campaign - "hey look guys, we caught some bad covert actors, so there's totally no nation states making massive use of LLMs to manipulate online discourse, look we caught them, everything is secure, don't worry about it". The funny part is, if it is all a disinfo campaign, then the very resources behind the campaign will downvote and attack this post and try to silence/discredit such voices. Or, maybe I'm just completely wrong and I'll get downvoted for having dumb beliefs. There's actually very little way to tell the difference between the two scenarios. In addition, does the act of me pointing out that malicious actors would want to downvote this now make it less likely for them to actually do so, as it would show their hand? You can recurse down these rabbit holes forever. Fun stuff.
- ImPostingOnHN 2y ago
  Your comment violates the site rules (commenting on downvotes) and cheapens the conversation with the same unfalsifiable conspiracy victim rhetoric, so you should expect to see downvotes for that alone, even if none of the rest of the comment existed.
ben_w 2y ago
Even if you're not skilled enough to stop the FSB from misusing your services, it's worth putting in the effort to stop the next Jim Jones from doing so. And learning to stop a Jim Jones, even knowing that you need to, may help you stop you from being named by the UN as complicit in a genocide, the way the UN explicitly named Facebook as having played a "determining role" in the Rohingya genocide: https://web.archive.org/web/20220217165239/https://www.reuters.com/article/us-myanmar-rohingya-facebook-idUKKCN1GO2PN https://web.archive.org/web/20220217165239/https://www.reute... Of course, if you want to prevent even the most capable of intelligence agencies from spinning up their own local LLMs at all, then at a minimum you can't ever publish the model weights themselves — a solution which itself is the cause of around 50% of the criticisms people around here make of OpenAI.

nerdjon 2y ago

Something about them caring about this bothers me a bit. Like, yeah ok it is a good thing that they are trying to curtail this abuse. However, it kinda feels like we are just delaying the inevitable? With open source models like LLAMA, detecting and doing anything about this will become impossible in a year or 2 (in reality its already possible). It just feels very, "look at us doing this good thing but ignore what we unleashed on the world" so please pat us on the back here. Maybe I am just pessimistic, but I am not about to give OpenAI any praise here for this when they are the reason we are in the current situation.

UberFly 2y ago
My feelings exactly. As usual it feels like lawyers are saying "just do this even though it's BS so in a future court room you can bring it up". You're not pessimistic, just realistic.
ADeerAppeared 2y ago
> It just feels very, "look at us doing this good thing but ignore what we unleashed on the world" so please pat us on the back here. It is that. These AI firms have been very vocal about their "ends justify the means" attitude. (Which is doubly damning considering how little these firms care about any purported AI safety for their AGI dreams. Months of "AI is going to kill us all" while they couldn't even be arsed to stop using CSAM-contaminated training data, nevermind using quality training data to ensure alignment) And "Foreign state actors generated some propaganda texts" is very small as AI-problems in the here-and-now go.
Lockal 2y ago
You are saying as if automatic detection of misinformation and brigades was ever possible before LLMs, or it is possible now. Only now, when people started to massively notice things like https://news.ycombinator.com/item?id=40444872 https://news.ycombinator.com/item?id=40444872 OpenAI publicly executed few accounts. By the way, they didn't even say exactly how many. Hundreds new will be created in 3.. 2.. 1..

S0y 2y ago

>We’ve terminated accounts linked to covert influence operations; >no significant audience increase due to our services. I can't help but find this extremely funny. you mean to tell me that massive operations trying to use your service didn't actually gain anything from using your service? OpenAI wasting time and resources shutting them down and the bad actors wasting time and resources using chatGPT no benefit. It just reads like a massive waste of time for everyone involved.

quinncom 2y ago
Hilarious, I love this take. I’d guess that “covert ops” using ChatGPT are amateurs, by definition (because it’s clearly not covert). Surely, the serious operators would use offline local LLMs.
mensetmanusman 2y ago
You’re thinking too much like an engineer. News like this teaches people that they are clearly ahead in capabilities far beyond any government or other corporation. Anytime they can get good publicity like this is good for their business.
dialup_sounds 2y ago
I get the sense that they would be using ChatGPT to save the time and resources required to produce content, rather than expecting it to produce better results.
- yencabulator 2y ago
  Save time -> can do more -> can gain larger audience.

barryrandall 2y ago

How effective were OpenAI's safety/security measures? With these confirmed bad actors, what percentage of their usage attempts were disrupted by automatic safety measures? Of the blocked attempts, is there any evidence that the influence actor successfully circumvented those protections?

asne11 2y ago
> we repeatedly observed cases where our models refused to generate the text or images that the actors asked for So the protections were bypassed at least some of the time. Many of these are demonstrably trivial to bypass.

mensetmanusman 2y ago

YouTube is already completely full of well-read bots supporting Russian attacks on Ukraine.

llamaimperative 2y ago
"Couldn't get any worse!" - Things people say right before it gets worse

ganoushoreilly 2y ago

While likely true, it's also helpful for pushing narratives requiring legislation that OPENAI just happens to be grandfathered into. I suspect more of these claims as the government forms a stronger opinion / rules around AI proliferation.

hex4def6 2y ago

"Our Usage Policies prohibit the use of our services to deceive or mislead others. This includes deceiving people about the authorship or source of content generated using our models, by engaging in activities such as: ... Creating automated systems that falsely present as being a real person, or don’t disclose to people that they are interacting with AI (unless it’s obvious from the context)." Suspiciously absent from the list of takedowns is anything on Reddit. Given the amount of obviously-chatGPT puppets on there, I'm somewhat 'surprised'. Also a good reminder that if you're using ChatGPT for anything, they can and will examine the traffic and contents of your messages / replies. I wonder if in the future they may offer a "service" to content providers. Say you're a reddit / forum / etc. You get supplied an API that you can run queries through that match any content that may have been generated through their service. You can then tag those accounts as bots. Of course, such an API will have costs associated with it...

hermitdev 2y ago
> I wonder if in the future they may offer a "service" to content providers. Say you're a reddit / forum / etc. You get supplied an API that you can run queries through that match any content that may have been generated through their service. You can then tag those accounts as bots. Of course, such an API will have costs associated with it... I'm not a lawyer, but this sounds like racketeering to me. Basically extorting "protection" money from sites.

TillE 2y ago

> “You can generate the content, but if you don't have the distribution systems to land it in front of people in a way that seems credible, then you're going to struggle getting it across,” The clearest example of a foreign influence operation I can remember was during the annexation of Crimea, when there was a flood of pro-Russian comments on Reddit. Thanks to the voting system and the centralized discussion threads, it's a platform that's relatively easy to manipulate. It's much harder to gain visibility on social media platforms where you choose which accounts to follow.

timschmidt 2y ago
> [reddit is] a platform that's relatively easy to manipulate. For all sides, apparently. https://www.youtube.com/watch?v=V7GtYaruTys https://www.youtube.com/watch?v=V7GtYaruTys

082349872349872 2y ago

We were hearing a little while back that it had become exceedingly difficult to concoct diplomatic cover for intelligence agents, in a world full of socials. Does the availability of AI mean that (as long as you own, or at least can retcon, several socials?) spinning up entire innocuous provenance networks for fresh identities is once again a push-button solved problem?

rurp 2y ago
It might help against casual checks, but a large web of invented social accounts will be uncoverable by any serious actor. If there's no corroborating trail outside of social media for any of those accounts it will be very possible to sniff out the fakery.
tivert 2y ago
> We were hearing a little while back that it had become exceedingly difficult to concoct diplomatic cover for intelligence agents, in a world full of socials. > Does the availability of AI mean that (as long as you own, or at least can retcon, several socials?) spinning up entire innocuous provenance networks for fresh identities is once again a push-button solved problem? AI doesn't defeat biometrics. I'd bet the problem with social media for intelligence agents is it allows facial recognition to uncover their true identity. If the search uncovers a real profile and a fake AI generated one with fake AI generated friends, it's still fishy to an adversary. Also if you're an agent and have gone to a country under one name, you can never go back under a different name without being trivially caught.
djhn 2y ago
There’s plenty of people in Europe in the 30-50 age bracket that aren’t on any social media. Many people from extremely wealthy families or careers in finance, consulting, medicine, law, law enforcement, military and the judicial branch came of age before ubiquitous social media and had already picked a career path where being discoverable is more of a liability than a benefit. No Linkedin, no facebook, no twitter, no instagram. Maybe a faceless fake name lurking account. Hard to say whether there are as many people in younger age groups who are like that.

nullc 2y ago

I was recently a defendant in a a set of lawsuits by a scammer seeking hundreds of billions of dollars in damages, using the courts as a weapon to shut down people who were in the way of his confidence game. The cumulative costs incurred by his opponents ended up well over ten million £‎. After a five week trial the court found against him in every respect, concluding his evidence was forged, that his actions constituted a fraud upon the court, etc. Relevant to this thread is that partly through the process he began to use ChatGPT. He appeared to use it in four ways: To create forgeries that he didn't have the technical skills to create (at least not on a short timeframe) with e.g. ChatGPT walking him through the technical steps, to concoct excuses and explanations, and to flood out his opponents with hundreds of pages of ChatGPT authored testimony that expended our resources just reading the material. He also used it to suggest legal strategy, which didn't cause us any problems and probably worked in our favor as ChatGPT's suggestions on that front appear to have been uniformly idiotic. Though he concealed his earlier usage, claiming to have lost access to his prior account, all in all his ChatGPT activity from around September 2023 to January 2024 amounted to some 22 million lines of text. We were extremely fortunate that his usage of ChatGPT began after his litigation began and that he made numerous errors that made his usage unambiguous. Had he been using it all along and less obviously our cost of defending the claim may have been increased dramatically-- even at his current usage had we been funding our defense entirely out of our own pockets his abuse of chatgpt may well have been the straw that bankrupted us before we could secure our victory. I think the us of LLMs as a tool in crime is interesting. For many negative uses of LLMs like spamming you can counter that people could just hire humans for the task so the LLM is more a question of cost or degree. But when committing a crime anyone you hire is a party that could extort you or whistleblow. We suspect our con made as many errors as he did with his forgeries in part due to his limited ability to enlist third parties to vet them. The potential for productive, honest, usage of the technology in this case appeared pretty limited relative to the potential for abuse. Some of the abuse could be addressed through improvements in standard practices like filing length limits, but its less clear to me how abuse like using it as a confidential forgery aid can be addressed. I certainly don't think it can be solved with safety approaches, since the interaction never needs to tell the LLM the true purpose. The same doesn't work for a human forgery aid because their access to outside information and potential for being questioned would make it much harder to avoid the risk that they discover that they're being used as a tool for fraud.

beretguy 2y ago

All this AI stuff is a godsend to the oppressive regimes. Makes their jobs a lot easier, effective, more believable, etc. > “These operations may be using new technology, but they're still struggling with the old problem of how to get people to fall for it,” said Ben Nimmo Even if true it’s just a matter of time before people will have absolutely no idea what truth really is.

wruza 2y ago
Makes their jobs a lot easier, effective, more believable, etc. Does it? Troll factories with authentic humans exist for at least a decade. They cost dirt cheap compared to a regime’s budget. There’s no problem with creating a network of bs, the problem is how to make it relevant. people will have absolutely no idea what truth really is We already have no idea. The only things that are true are fundamental human motivations. You can’t miss too far starting from there. Once you get that point of view, everything becomes so clear, despite missing details.

Lockal 2y ago

"Influence operations": Oh no! We were removed! What should we do? Oh, wait, there is "Sign up" button

dralley 2y ago

There was some interesting discussion about this on reddit: > In Ukraine this is a very known problem, to the point when there are jokes about those bots , cause they often post texts translated to Ukrainian from Google translated russian text. For example a year ago or so there was a meme "I can't stand this hellish flour" , which was created cause Google translator translated "hellish torment" from russian to "hellish flour" in Ukrainian , cause 'flour" in Ukrainian written exactly the same as "torment" and 'flour' in russian, which isn't the case in Ukrainian language (well , kinda the case , but Google translator translated word that has two meanings in the word that means only flour) https://www.reddit.com/r/europe/comments/1d4j1oo/openai_has_busted_two_russian_propaganda/ https://www.reddit.com/r/europe/comments/1d4j1oo/openai_has_...

Qem 2y ago

Israeli too. Curious they left it out of the headline.

dralley 2y ago
Headlines can only be so long. Other outlets (e.g. The Guardian) left the Chinese out of the headline, and nearly all of them (and yourself) left out the Iranians.