5 ms·
Disrupting a covert Iranian influence operation
- darkoob12 2y ago[flagged]
- ImHereToVote 2y agoI wonder if it would be possible to get a list of countries that can have influence operations using ChatGPT and countries that can't.
- janice1999 2y ago> Russia, China, Iran, and Israel https://www.npr.org/2024/05/30/g-s1-1670/openai-influence-operations-china-russia-israel https://www.npr.org/2024/05/30/g-s1-1670/openai-influence-op...
- ein0p 2y agoThis doesn’t pass the sniff test. Each of these countries has domestically developed, strong LLMs. And even if they did not, they’d have absolutely no issue running the larger FOSS ones. You don’t need GPT4 to generate political drivel.
- axus 2y agoSounds like OpenAI marketing is stronger than patriotism for home-grown LLMs.
- ein0p 2y agoMore likely someone is trying to pass a few basement dwellers as major nation state influence operations.
- kgeist 2y agoIndeed, Russia has YandexGPT and Gigachat which are more than enough for generating articles (although they're far worse at reasoning).
- dtquad 2y agoIt takes a lot of GPU to run the larger FOSS LLMs. The OpenAI API is still the cheapest way to get high-quality LLM generations.
- ein0p 2y agoTell that to someone else. I run LLaMA70B in my garage. Works fine, cost is negligible.
- ComplexSystems 2y agoWhat, are nation states trying to interfere in US elections on a real tight budget or something?
- wslh 2y agoThose four results only show the metapropaganda limitations. There are more countries in the world that are less "fancy" and included in the list. Weird that you don't include America though. BTW, your comment reminds me this very basic espionage issue [1]. [1] https://www.reddit.com/r/todayilearned/comments/bdjh8o/til_that_when_china_ordered_a_boeing_767_from_the/ https://www.reddit.com/r/todayilearned/comments/bdjh8o/til_t...
- lancesells 2y agoI don't think that would matter. I live in the US and I would guess there are plenty of people in this country that work for countries that aren't allowed to use it. Same goes for other countries having people who work for the US. I really don't think it would take that many people to run propoganda campaigns with modern tools.
- hobo_in_library 2y agoOnly countries with dedicated PsyOps divisions are allowed to use ChatGPT for influence operations, obviously https://en.wikipedia.org/wiki/Psychological_operations_(United_States) https://en.wikipedia.org/wiki/Psychological_operations_(Unit...
- throwawaycar 2y ago[dead]
- colordrops 2y agoI hope they put the same restrictions on Israel but I doubt it. Multiple core OpenAI team members have expressed pro-Israel comments, some very murderous and ugly, so I doubt their ability to be unbiased here.
- jjmarr 2y agohttps://www.nbcnews.com/tech/security/meta-openai-say-disrupted-israeli-companys-influence-campaign-rcna154774 https://www.nbcnews.com/tech/security/meta-openai-say-disrup... https://openai.com/index/disrupting-deceptive-uses-of-AI-by-covert-influence-operations/ https://openai.com/index/disrupting-deceptive-uses-of-AI-by-... OpenAI has also banned Israeli influence operations. Do you think the above isn't going far enough? It's also strange that you call out all pro-Israel comments as a cause for concern.
- jedimind 2y ago>OpenAI has also banned Israeli influence operations. Do you think the above isn't going far enough? How is that far enough when their head of research is a genocidal zionist?[1] >It's also strange that you call out all pro-Israel comments as a cause for concern. What's so strange about calling pro-Israel comments as a cause for concern when Israel has been on what jewish holocaust scholars like Amos Goldberg & Raz Segal have described as [2]"undoubtedly Genocide"? Quite the opposite actually, the OP is spot on: how can someone like Tal Broda still work at OpenAI as head of research when he has openly spewed genocidal[1] incitement? People in tech always talk about avoiding bias in AI and the proceed to retain such a hateful genocidal individual. There is no way in hell that if the roles were reversed and an arab person had made the statements that Tal Broda has made that he would have ever been allowed to keep his job. They would have fired him immediately. The double standards are so ugly and glaring that they will be studied by future generations. [1] https://x.com/StopArabHate/status/1806450091399745608 https://x.com/StopArabHate/status/1806450091399745608 [2] https://www.jewishvoiceforlabour.org.uk/article/prof-amos-goldberg-yes-it-is-genocide/ https://www.jewishvoiceforlabour.org.uk/article/prof-amos-go...
- 2y ago
- vkou 2y agoWhy stop at countries? Is it possible to get a list of organizations that are running influence, propaganda, and astroturfing operations using ChatGPT, and its ilk?
- trompetenaccoun 2y agoWe could start with OpenAI itself. The raw output of the models is clearly manipulated by the company, leading to politically biased results. Which many of their users naively take as 'truth' because they think it's artificial intelligence.
- akira2501 2y agoWow. What a high value target. [0] I mean. I get that low rent actors will use low rent services to try to generate political garbage. Is there any evidence that this is actually having a measurable or meaningful impact? Who exactly is fooled by these sites? And is it the sites that are the problem or the relative lack of sophistication in American education when it comes to political corruption? [0]: https://niothinker.com/ https://niothinker.com/
- deleted 2y ago[deleted]
- Maken 2y agoMy guess is that these sites are used to generate headlines that will be then shared by bots. Most people will never get pass that headline, the site is there to create the illusion of a real news source for those few who actually click the headline (but will probably never read the entire article because TL;DR). These healines are designed to reinforce certain bias and keep people inside their echo chambers, and that does have an impact.
- akira2501 2y agoWe don't have to guess. We can precisely know the reach of these articles. Aside from this I wonder what value does GPT have here? It seems like it's saving "the covert Iranians" (how covert can you be if an American business points the finger right at you?) on the order of a few dollars an hour. It's hard to take seriously. On any level.
- Maken 2y agoGPT just lets you do this on scale. It lets you create an endless stream of legit sounding new sites and flood the news aggregators until people that only gets information by Twitter/Facebook are consuming mostly your fake media. Then, when you have captured an audience by pandering to their interests, you slowly herd them towards whatever political view/consumer habits you want. It is nothing that haven't already be done in the past, but with LLMs is like going from craftsmanship to mass-production assembly lines.
- katzinsky 2y agoYeah. Knowing my tools are judging my political positions and could self destruct if the authors disagree just makes me love using my computer. This is why I only use local models these days. EDIT: Out of posts for today but I've been pretty happy with Gemma2. The context is short but the performance is very good and it's easy to disable refusal.
- guerrilla 2y agoJust curious, which local models?
- tonetegeatinst 2y agoI know the Facebook model was good. Checkout r/locallamma for guidance, or check the ranking website.
- lambdaba 2y agoThe only thing noteworthy about this is how small-scale this is, and that the perpetrators don't even bother/have the means to set up their own infrastructure.
- chatmasta 2y agoThe noteworthy thing is that OpenAI is reporting it. They’re signaling that they are proactively monitoring and investigating this activity, and that they’re willing to work with federal agencies while self-policing their negative externalities. This is all part of an ongoing conversation with lobbyists about “safe AI,” and it’s ultimately done to show that OpenAI is making an effort to mitigate the risks that regulators claim it creates. But there’s also another signal, which is what they’re not broadcasting: “ChatGPT can be used for propaganda, it works in Persian too, and we’re happy to sell to the DoD.”
- lancesells 2y agoIt's a good thing for them to report this, but for a company that decided against watermarking their output they are kind of complicit. I know they might lose their teens and college demographic if they do it, but if it's truly this world-changing tool that they claim than not watermarking is scraping the serial number off the guns they're selling.* * Maybe a shitty analogy
- mondobe 2y agoHow exactly would they watermark textual content?
- rolph 2y agowith fonts and typos.
- Something1234 2y agoWith unicode characters. Greek question mark anyone instead of semicolon?
- programmarchy 2y agoThe linked PDF (Storm-2035 [1]) from Microsoft is more detailed and interesting than the blog post. However, what's missing from the reports is how they detected those operations and how they tied them to different groups. There's a lot of claims being made without showing all of the supporting evidence. To give them the benefit of the doubt, they likely want to keep their detection methods secret to make circumvention more difficult. And it all sounds totally plausible of course. But at the same time, a degree of skepticism is warranted because Microsoft has a huge incentive to fearmonger about AI so they can lock it down and capture the market. And what better way is there than to use the usual bogeymen to do so. [1] https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/microsoft/final/en-us/microsoft-brand/documents/5bc57431-a7a9-49ad-944d-b93b7d35d0fc.pdf https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcor...
- com2kid 2y agoHaving worked at Microsoft for almost a decade, I remember chatting with their security people plenty after meetings. One interesting thing I learned is that Microsoft (and all the other top tech companies presumably) are under constant Advanced Persistent Threat from state actors. From literal secret agents getting jobs and working undercover for a decade+ to obtain seniority, to physical penetration attempts (some buildings on MS campus used to have armed security, before Cloud server farms were a thing!). Microsoft is one of the few companies that goes toe to toe with world governments every day of the year. And I imagine balancing that next to all the engineers who demand admin access everywhere is a royal pain! Although the best Government VS Business story I heard was during intern orientation at Boeing about French agents breaking into Boeing employee's hotel rooms during a conference in France while the employees were out to dinner, and going through laptops. One of the employees returned earlier than expected, and the men in suits shut the laptop, turned around and walked out of the room w/o saying anything!
- BoingBoomTschak 2y ago> Microsoft is one of the few companies that goes toe to toe with world governments every day of the year. It's also the company which was the first and longest member of PRISM, meaning very deep ties to the less savory parts of the US gov and {five,nine,fourteen} eyes. I know it's a boring advice, but I'd take this kind of declaration with a truckload of salt.
- greatgib 2y agoWhat they don't say on their post but that we can guess and is from interest is that they probably had to spy their user messages to determinate that they used the account for generate content for the influence operation. For sure the purpose is noble, but it is good to remind everyone that everything you type, submit or generate there is not private but could be randomly snooped by strangers!
- TOMDM 2y agoThat or they could store hashes of chunks of output to compare to propaganda in the wild.
- greatgib 2y agoIt would be very costly and nothing say that the bad actor is using the message as-is and not a subset.
- szundi 2y agoNaah, probably they used OpenAI apis to read, summarize and categorize everything
- deleted 2y ago[deleted]
- simonw 2y agoFrom a Google search it looks like this is one of the articles in question: https://teorator.com/index.php/2024/08/12/x-censors-trumps-tweets-a-hidden-war-on-free-speech/ https://teorator.com/index.php/2024/08/12/x-censors-trumps-t... I requested the Internet Archive grab this copy: https://web.archive.org/web/20240816210620/https://teorator.com/index.php/2024/08/12/x-censors-trumps-tweets-a-hidden-war-on-free-speech/ https://web.archive.org/web/20240816210620/https://teorator....
- throw310822 2y agoLooks exceedingly good to be Ai-generated. Also, it appears to be slightly pro-Trump, which doesn't sound like in Iran's interest.
- stickfigure 2y agoIt is clear to me that Russia, Iran, North Korea are playing a zero-sum game. Anything that creates chaos in US politics (or European politics) is in their interest. Donald Trump represents chaos.
- ignoramous 2y ago> Russia, Iran, North Korea You're missing China & Israel: https://www.theguardian.com/technology/article/2024/may/30/openai-disinformation-russia-israel-china-iran https://www.theguardian.com/technology/article/2024/may/30/o... / https://archive.is/y0V3A https://archive.is/y0V3A And, according to OpenAI, they're interfering not just in the US: https://www.thehindu.com/elections/lok-sabha/openai-says-stalled-attempts-by-israel-based-company-to-interfere-in-indian-elections/article68237334.ece https://www.thehindu.com/elections/lok-sabha/openai-says-sta... / https://archive.is/RWn8X https://archive.is/RWn8X
- stickfigure 2y agoIsrael and China are not playing zero-sum games. Israel certainly derives benefits from a powerful United States. They may want to influence US politics, but they have a strong collaborative interest. China is more mixed, but our economies are still intertwined. The CCP might wish to shift balance and reduce the power of the United States, but we still have mutual interests. On the other hand, if the leaders of Russia or Iran or North Korea could push a button that immediately and completely depopulates the United States without attribution, I'm pretty sure they would push it without hesitation.
- joe_the_user 2y agoI don't think people should jump to any "this is the level Iran is at?" conclusions. Many nations employ "patriotic citizens" in an informal and semi-formal fashion along with trained propaganda and "infowar" experts. I know of China and Israel doing this but I'd assume it's everywhere.
- det2x 2y agoIran is definately putting out mass propaganda especially on social media. Israel has one of if not the largest propaganda/bot campaigns on the entire planet, followed by either Russia or China. It's definately working for them so they won't be stopping anytime soon.
- djaouen 2y agoThis is as much an indictment of ChatGPT as it is of the Iranians. According to OpenAI, their product produces output that no one in their right mind would want to read for any purpose.
- echoangle 2y agoHow is that a bad thing for OpenAI though? It depends on the prompting, I wouldn’t count the ability to generate useless/stupid/misleading content when prompted to do this as a negative.
- djaouen 2y agoI think the point is, nobody likes slop. The fact that slop failed to promote the enemy doesn't mean that slop is inherently good.
- echoangle 2y agoBut being able to generate slop doesn’t mean the AI is bad, even AGI could be able to generate slop. I certainly can. So how does it say bad things about OpenAI that their AI can generate slop?
- hobo_in_library 2y agoIt's hard to take an article like this at face value when they provide zero evidence for any of their claims. This is coming soon after Trump decided to accuse Iran of being behind his assassination attempt (done by a white 20 year old) and Israel literally assassinated Hamas's chief negotiator while he was visiting Iran. It seems like the powers that be are desperate for a war with Iran and will continue beating the drum to build consent. Reminds me of the build up to the 2003 Iraq invasion (you know, because "they have WMDs")
- deleted 2y ago[deleted]
- throwawaycar 2y ago[dead]
- usefulcat 2y agoI'd be more interested in an analysis of the likely intention of the campaign. Is it just an attempt to reduce voter turnout? If so, that doesn't seem all that useful by itself.
- dfhvneoieno 2y agoIran wants Democrats, Russia wants Republicans. It's easy to see which one is more successful. Every time you blame something on Russia, people come out of the woods "explaining" why Russia is actually the victim (Russia is ALWAYS the victim, is what I've learned from this people).
- epolanski 2y agoRussia definitely has a non zero amount of people supporting or apologizing it's actions all around the world. It's dangerous to assume all of the content you see is simply some shady operation or fake, having controversial opinions on any topic is nothing new, millions of Americans don't believe in the moon landings. In general I don't think it's a good idea to discuss politics online, and I feel like many talk too much geopolitics but can't even tell the policies or programs of their own city mayor candidates which are way more important for their own lives and directly impacting.
- aguaviva 2y agoIn regard to local issues being way more impactful in one's life: If one is from, or has close contacts in or from the regions affected by geopolitics -- then this is unfortunately very much not the case. As disappointed as I am with local politicians -- they aren't bombing my friends into the ground, or forcing them to move halfway across the planet indefinitely. Or causing these to deal with significant emotional anguish, even if they aren't forced to move, or directly under threat. While a solid contingent on HN regularly either apologies for / minimizes their actions, or just seems to shoot from its hip based on their hunch as to what is happening, without any indication of having done much research or questioning their sources. Or even just thinking logically about the various narratives they're reading. For most of them it seems to be more about ideological abstractions than anything real and concrete, in any case. So that, plus the simple fact that this is a global community is why geopolitics floats to the top, as it were. Whether talking about it online is productive or helpful in any way is another matter altogether. Turns out it's generally not easy to talk about these things in person, even with people one knows rather well (as most of the time they'd prefer to talk about pretty much anything but "the situation"). Being as the "shit" in question is all too real for them.
- PerilousD 2y agoThese jokers seem like the AI version of "script kiddie" hackers, and OpenAI may be engaging in a bit of humble bragging. It doesn't take considerable investments in time or money to run local LLMs, INCLUDING ChatGPT, where your questions, prompts, and results are not sent home to the mothership, so it's a BS article as to (the real) actors who may or may not be doing this. NOW, if OpenAI or Gemini or LLama, etc, showed how they analyzed social media posts and flagged the ones that were AI generated and the analysis as to WHY the article is flagged, then that would be much more useful, actionable by at least some of the readers and would put the accounts spreading the content (particularly the rebroadcast fluffers) in the spotlight.
- Arainach 2y agoIt wouldn't be useful at all, and would only serve to educate malicious actors how to better evade detection. It's like claiming a search engine open sourcing its ranking algorithm would help people be informed instead of making spammers able to perfectly hijack all the results.
- navaed01 2y agoAmen
- commandpaul 2y agoGiven that any self hosted open source model would have worked just as well. I can’t see this good faith post as anything more than forwarding open ais long campaign for regulatory capture.
- dtquad 2y agoSelf-hosting LLMs is expensive at scale. It's cheaper to use VC subsidized model inference like the OpenAI APIs.
- CyberDildonics 2y agoAt "scale"? At what scale?
- littlestymaar 2y agoThere are plenty of VC-subsidized inference provider which uses open source LLM for much cheaper than OpenAI (which isn't really VC-subsidized at this point but Microsoft-subsidized).
- refulgentis 2y agoMy anecdata is most teams I've talked to say its below OpenAI at scale, and vLLM is a beast. It's interesting to hear the opposite, there's lots of cheaper providers, but the "VC dollars" argument can go "turtles all the way down", I suppose. Still, reality seems to differ.
- deleted 2y ago[deleted]
- aaron695 2y ago[dead]
- ComplexSystems 2y agoWhy wouldn't Iran just use Llama or something?
- defamation 2y agothey might be using that too, it's that OpenAI can detect and catch those using their service. Cmon mate.
- swat535 2y agoOf course they are.. This is just a marketing piece by OpenAI
- rvnx 2y agoThe headline should be "OpenAI publicly admits it supported Iran influence operation despite the sanctions"
- dj_gitmo 2y agoThey supported it by banning them? That doesn’t make much sense to me.
- FactKnower69 2y agoMain takeaway is that OpenAI is reading all of your prompts :)
- sweeter 2y agoCNN just did a piece on private Israeli groups doing the exact same thing. It is pretty scary the shear scale of this. Literally any entity or group can spin up a ton of bots and use any AI service locally or otherwise, to attempt to sway public opinion.
- ants_everywhere 2y agoReddit especially is full of LLMs. I would imagine there are ones from every major country plus several privately funded ones.
- theGnuMe 2y agothey are here as well.
- mandmandam 2y agoCan they be drawn out, I wonder, when the context is too thin for them to catch... GMOs are dangerous for reasons beyond immediate health implications. Israel has been determined in the world's highest court to be plausibly committing genocide. The tech industry is complicit in war crimes right now. Media consolidation is a massive and immediate threat, as are climate change and AIPAC. ... Or is it like what the park ranger said about the overlap between people and bears?
- suby 2y agoYeah, all of the major subreddits are gamed, especially subs like r/politics. Even going back to the 2016 election it was clear that much of the activity on there was not genuine.
- shusaku 2y agoIt’s been well known for over a decade that Reddit is incredibly easy to game. So while LLMs may seem useful, a well run political campaign can easily afford to astroturf the site while paying good money to their staff.
- deleted 2y ago[deleted]
- pessimizer 2y agoNothing like 900 million articles about secret Iranian plots from government entangled dystopian megacorps and the pundits who love them, soon after the US starts moving troops into the Middle East to defend the progress of an ongoing genocide. I suppose this is a great opportunity for the people whose entire income comes from the fact that the US overpays insiders to supply its own military; if Trump wins, he gets to play tough and have the papers start his term praising him to the skies for starting a war with Iran, he'll also be able to blame everything on the last administration and work closely with the people who replace Netanyahu. If Harris wins, she's made no promises and has no beliefs, and will be aided by the press in blaming Gaza on Iran. OpenAI itself is surely running larger covert influence operations in order to affect US legislation and elections. > Similar to the covert influence operations we reported in May, this operation does not appear to have achieved meaningful audience engagement. The majority of social media posts that we identified received few or no likes, shares, or comments. We similarly did not find indications of the web articles being shared across social media. Sounds like Russian Facebook ads.
- TiredOfLife 2y agoUS is going to protect Hamas?
- irthomasthomas 2y agospeaking of influence operations, the strawberry dude and 'lily' just did one of those twitter voice group chat things where everyone tried to guess if lily was an ai or not. there just happened to be a worldcoin rep in the room...
- navaed01 2y agoCan you expand?
- irthomasthomas 2y agoStrawberry is alleged to be a codename for some secret openai technology. A twitter account/s that keeps spamming strawberries and making cryptic prophesies was shared by openai and sama. Lots of openai staff and connected people also spammed twitter with strawberries, making it look like an official campaign, or like something was really about to be announced. The same account co-hosted a chat with someone called lily who some claimed was an ai. worldcoin is a private company owned by sama that verifies identity by scanning the eyeball. He's been rolling out the technology in places like South Africa, where they also recruited people to train openai models by RLHF etc. It seems it has since been confirmed a scam by some of the participants. This chap looks to have more details on it https://twitter.com/BenjaminDEKR/status/1824676550379192324 https://twitter.com/BenjaminDEKR/status/1824676550379192324
- kjellsbells 2y agoOne area that OpenAI did not comment on, was the likelihood that the AI-generated content here was itself used in the training data for a later model. Let's say the OpenAI engineers are working on ChatGPT 5 and spent last month scraping Teorater and X/Twitter, where this material ended up. How does OpenAI know that the new model is not poisoned? This isn't just OpenAI's problem of course. Anyone training on the open Internet now has this problem.
- deleted 2y ago[deleted]
- Arrath 2y ago> This isn't just OpenAI's problem of course. Anyone training on the open Internet now has this problem. 'Low-background Steel' has been a nerd trivia tidbit for a while now, will we have to consider a 'Pre' and 'Post' AI internet era for training uses?
- nolroz 2y agoMaybe one of the uses of the generated content detection method that OpenAI supposedly has is they can filter at least some of it out?
- ethbr1 2y agoGiven recent advancements, buying Twitter solely for its use as a data source seems prescient, given only the first party has additional metadata to identify bot replies. ... if the actual purchase hasn't been such a cluster$&@$ so as to thoroughly disabuse the notion there was some master plan.
- navaed01 2y agoIts great that openAI is using this infraction as an opportunity to posture about how open they are and how they are a company that can tame evil applications of AI, while totally missing and not addressing the broader concern, what if this was run on a local instance?? How are we stopping / spotting and squatting that?! Side note. This is some pretty terrible propaganda. The post about Kamala, immigrants and climate change barely makes any sense. X just hosted trump for a live stream, who is being affected by a headline that reads ‘X censors trump tweet”.
- bangaladore 2y ago"...We ran these images through our DALL·E 3 classifier, which identified them as not being generated by our services..." I'd be shocked if Twitter weren't stripping the metadata they are checking. It is apparently C2PA per [1] [1] https://openai.com/index/understanding-the-source-of-what-we-see-and-hear-online/ https://openai.com/index/understanding-the-source-of-what-we...
- throwawaycar 2y ago[dead]
- langsoul-com 2y agoLLM powered disinformation machines is terrifying. The barrier to entry and sustained cost is so low. But the social impact is significant. I'm reminded of a fake story about kid kidnapping in India, that caused a mob to burn the two people that were targeted alive... They were completely innocent, the mob attacked them based on fake news. Now, that can happen enmasse.
- mediumsmart 2y agoinfluencing the politics of the United States? Is there an alternative choice I have not heard of? I thought it was samesame in blue or red as it always has been.
- JoeyBananas 2y ago[flagged]
- deleted 2y ago[deleted]