7 ms·

Why AI companies want you to be afraid of them

290 points by rolph 6mo ago

scratchyone 6mo ago
Honestly we should have learned this claim from AI companies was purely fear-mongering back when GPT-2 was "too dangerous to release".
- detectivestory 6mo ago
  That's true but in reality I think people are far more afraid of AI in terms of how it is being used in warfare and policing. Automatic target detection and deployment of drones, or even how it might simply make their role at work redundant etc
  - scratchyone 6mo ago
    100% agreed. That's part of the issue imo, these companies pretend their new models are "too dangerous" to seem like they care about the world, yet they have no qualms deploying existing models in warfare or bragging about impending mass-unemployment.
  - notrealyme123 6mo ago
    I am to be honest not sure what I am more scared. AI shaping warfare Vs. Using AI to justify outrageous warfare
    - scratchyone 6mo ago
      Don't worry, these companies will make sure we get to experience both nightmare futures.
    - yieldcrv 6mo ago
      that’s not a war crime, that’s boundary setting, and honestly, that’s rare would you like me to list the applicable sections of the Geneva convention?
    - MSFT_Edging 6mo ago
      We sadly don't need AI to justify outrageous warfare. You just need to remember when the US invaded Iraq over WMDs, including a full investigation into the WMDs that never found any. We then invaded anyway, to the detriment of everyone except defense contractors.
  - palmotea 6mo ago
    > That's true but in reality I think people are far more afraid of AI in terms of how it is being used in warfare and policing. Automatic target detection and deployment of drones, or even how it might simply make their role at work redundant etc I think the last one should be first on the list: regular people are afraid AI will negatively affect their economic security (i.e. knowledge and service workers will get the rust-belt factory worker treatment). And the potential of giving knowledge and service workers the rust-belt factory worker treatment is exactly what makes Wall Street excited about AI and has the AI company leaders salivating about the profit they can make. Warfare, policing, bio-engineered viruses are theoretical and far down the list.
    - detectivestory 6mo ago
      Its already being used in warfare though.
      - palmotea 6mo ago
        > Its already being used in warfare though. What I mean is theoretical to the common person. They don't have killbot drones hunting them down, and are unlikely to have that experience anytime soon. But most people have jobs, most people would be hard-hit if they lost theirs, lots of people lose theirs, and our elites are just itching to make that happen. I'm certainly most worried about AI: my employer started an ongoing silent layoff campaign about the same time they started enforcing AI usage. I don't think those are unconnected.
    - wongarsu 6mo ago
      Not to mention that "automatic target detection" was primarily enabled by the ~2016-2020 AI hype/boom around image recognition, not the 2022-current hype/boom around LLMs
  - mossTechnician 6mo ago
    To me, the more interesting divergence in discussion is on its capabilities. AI industry insiders (including "safety" groups like ControlAI) talk about the dangers only in terms of its power: "Scheming", job loss, breaking containment, the New Cold War with China. Critics outside the industry talk in terms of its lack of power: Inaccuracy, erroneous translation of user intent, failure to deliver on its promises and investment, environmental cost from the former, and ultimately the danger of people in power (e.g. law enforcement, military officials) treating its output as valid and unbiased, or simply laundering their wishes through it.
  - chasd00 6mo ago
    AI has been used in defense for a while now, a modern tomahawk cruise missile and its associated targeting systems is a good example. I think most people fear AI taking their job and only source of income.
    - sublinear 6mo ago
      Linear regression?
  - sublinear 6mo ago
    These were all already very valid concerns long before this era of "AI" or computational power. The broader public is just now barely beginning to understand because all they have to do is ask a chatbot. AI does not enable new capabilities, but it does aggregate an idea into a rough sketch and do it quickly on-demand. None of this really means it will play out that way. The devil is in the details. What it does mean is much more nuanced attention on the politics and money because that's where the power always was.
    - detectivestory 6mo ago
      AI does enable new capabilities when it comes to constant mass surveillance, and automated weaponry.
      - sublinear 6mo ago
        No it doesn't. We already have all of that right now and have had it for decades. The big investment into Project Stargate is all about managing risk. The government contractor and security clearance situation is out of control. As well, every human mistake is costly and time consuming to address. If you instead blame it on AI, you can skip the court proceedings and postmortems. The other part of this is likely an attempt to surface information with summaries and shorten the chain of command. This is just a power grab and a dangerous dismissal of necessary implementation detail. It's a tantrum being thrown by ignorant people at the top being displaced. We live in an ever-complicated world that demands more experienced leadership than we have available. AI is their hail mary pass. LLMs are being abused as a political battering ram. They are not the technological breakthrough advertised. The AI label is borderline absurd. AGI even moreso. NLP is an accessibility tool at best.
- maplethorpe 6mo ago
  It seems like they were correct, to me.
  - iugtmkbdfil834 6mo ago
    I was going to say.. I think people in general have this weird understanding of the word dangerous. Just because something is not movie level dramatic and/or does not generate over the top violence does not automatically make it less dangerous. In a sense, just the fact that is benign on the surface and allowed to embed in our day to day life is what makes the upcoming rug pull so painful. And I am saying this as a person who actually likes this tech.
  - elar_verole 6mo ago
    Yes, I love how everyone uses this argument, when what they were saying was among the lines of "GPT-2 would make it too easy to generate spam, deepfakes, content to manipulate opinion..." (not the actual quote but something like that). Turns out it was completely correct if you look at the state of the internet right now. Obviously, they still overhype and oversell this end of humanity stuff, but this argument regurgitated ad-nauseam is not THAT great of an example when you think about it.
- mofeien 6mo ago
  Given that his reason for saying GPT-2 was too dangerous to release was that the world needed more time to prepare for the effects of this technology, and given that the following models were basically scaled-up versions of it and killed social media, news reporting and other kinds of communication, I'd say he was right about the dangers of it.
  - scratchyone 6mo ago
    funny how he didn't care about ethics the moment it was more profitable to release it than to talk about dangers.
yuhmahp 6mo ago
Assuming this article isn't written by an AI
- scratchyone 6mo ago
  I mean, it's the BBC and the article doesn't have any typical AI tells, where is this idea that it's AI written coming from???
  - yuhmahp 6mo ago
    I'm assuming it's possible that an AI can deduct the potential 3rd degree result from an article like this on the BBC. Not talking about the wording
    - MajorTakeaway 6mo ago
      >sees any random article headline, automatically assumes it's AI.
      - hxugufjfjf 6mo ago
        Pretty reasonable assumption these days unfortunately.
        Jtarii 6mo ago
        Just saying things with no evidence is not reasonable
    - philipwhiuk 6mo ago
      Roko's basilisk is a very tedious idea.
InputName 6mo ago
In lieu of a technological moat, companies search for regulatory capture.
SpicyLemonZest 6mo ago
> It's a strange way for any company to talk about its own work. You don't hear McDonald's announcing that it's created a burger so terrifyingly delicious that it would be unethical to grill it for the public. > Here's one theory. But the author never gets back to this! It's the main observation the theory has to account for; why don't we see other companies speak this way, if it's such an effective strategy for deflecting non-apocalyptic concerns?
- scratchyone 6mo ago
  tbf most companies don't have a potentially world-ending product. only real similar field is defense contractors who typically can't brag about unreleased ideas as they're classified.
  - SpicyLemonZest 6mo ago
    I agree, but the experts the author cites do not. Professor Valor believes that AI is a mirror and any existential fears of it are just reflected fear of ourselves; Professor Bender believes that AI is a con and all the people who say it's powerful enough to be world-ending are lying. Anyone who concedes the premise that AI has a genuine potential to be world-ending is, I think, on the AI labs' side of this debate.
    - jrumbut 6mo ago
      We are not such a bad thing to fear. This technology interacts socially, so even if it can't jailbreak itself on a technology-level (which feels like a tough guarantee to make at this point) it can simply ask someone to do a bad thing and there is some chance they'll do it. The same way a human leader does. The first kids who have only faint memories of a time before chatbots will be entering the military in 6-7 years. You have to assume they are acting as best friends, therapists, or even surrogate parents for a substantial number of kids right now. We are going to need years to figure out what to do about this technology. I think some impetus to get that process started is a good idea.
- tangotaylor 6mo ago
dicksent 6mo ago
gotta bring up the hype
sixtyj 6mo ago
Article mentions a book "The AI Con" that argues that much of what is labeled "artificial intelligence" is a misleading term that obscures ordinary automation while concentrating power in a small number of technology firms. So fear-mongering seems to be just a tool how to get attention and more customers. Hey ma, I use very dangerous tool now. I am OG.
- redsocksfan45 6mo ago
  [dead]
tangotaylor 6mo ago
Finally the media is catching on. Lee Vinsel's criti-hype article nailed this 5 years ago, before we even had the chatbot economy we do now: https://sts-news.medium.com/youre-doing-it-wrong-notes-on-criticism-and-technology-hype-18b08b4307e5 https://sts-news.medium.com/youre-doing-it-wrong-notes-on-cr...
- b65e8bee43c2ed0 6mo ago
  the media is not catching on, they've been looping through 'AI is going to kill us all!' when they want to sell fear and 'Look at all the energy and water AI companies are pointlessly wasting!' when they want to sell anger. the writers and the editors know exactly what they're doing - spreading FUD and creating controversy out of thin air. some of it is done for-profit, some for-agenda, and all of it with malicious intent.
baggachipz 6mo ago
Why wouldn't they continue crying wolf when it always gets them free advertising from a gullible/complicit press?
firefoxd 6mo ago
"you will all lose your jobs and it will wipe out half of humanity." If you lead with this, people will stop questioning why their sprint velocity hasn't increased 10 fold. Managers start asking leads, instead of hiring more devs can we add Agent.md to our repos? The Apocalypse sells. They are afraid that you'll find out that AI is just another useful tool. That's the real threat, not to humanity, but to their hype. Edit: i made a video about this recently: https://youtu.be/nB0Vz-fh8EI https://youtu.be/nB0Vz-fh8EI
bryan0 6mo ago
> Why do AI companies want us to be afraid of them? ... According to critics, it benefits AI companies to keep you fixated on apocalypse because it distracts from the very real damage they're already doing to the world. People seem unable to make up their mind if AI is very dangerous or is it not. I think what the AI companies and this author agree on, is that this technology is potentially extremely dangerous. AI impacts labor markets, the environment, warfare, mental health, etc... It's harder now to find things which it will not impact. So if we agree that AI is potentially dangerous, it makes the title question moot: Both AI companies and this author want people to be aware of the dangers that AI poses to society. The real question is what do we do about it? The nuance here is that AI can be incredible positive as well. It's like the invention of fire, you can use it for good or bad, and there will be many unintended consequences along the way. We could legislate and ban AI tech. People have proposed this seriously, yet this feels completely unrealistic. If the US bans AI research, then this research will move elsewhere. I think it is like trying to ban fire because it's dangerous: some groups will learn to work with fire and they will get an extreme advantage over those groups that don't. (or they will destroy themselves in the process). So maybe instead of demonizing the AI companies, we have a nuanced debate about this tech and propose solutions that our best for our society?
- dodu_ 6mo ago
  >So maybe instead of demonizing the AI companies, we have a nuanced debate about this tech and propose solutions that our best for our society? These are not mutually exclusive. Calling out the demonic behavior of trying to coerce people into using your product out of fear is not an indictment of the underlying technology itself.
  - bryan0 6mo ago
    One of the points I was trying to make is that the statement: > trying to coerce people into using your product out of fear is nonsense. Everyone agrees that there are legitimate reasons to be fearful of this technology, this is not a fabrication, but we need to figure out how to proceed in a safe and constructive way. What "coercion" is occurring here? Either you find the technology valuable and you want to pay for it, or you find it not useful (or worse harmful), and you do not want to pay for it. Maybe another way of putting it, what do you think the frontier AI companies should do in this situation? It seems that being straightforward with the dangers is correct thing to do, and probably being overly cautious is prudent. You could go further and argue they should slow down or stop development, but that is something that the govt should impose, we should not expect or trust the companies to do this themselves. Ironically, in the Anthropic / Pentagon case, we have Anthropic trying to pump the brakes and put up guardrails while the govt wants to go full-steam ahead with autonomous warfare. The other issue with slowing down / pausing development is it requires an unheard of level of agreement, even with companies in China, or else it will probably not be effective. You could argue this is not even possible at this point.
tptacek 6mo ago
I have never heard of "Heidy Khlaaf, chief AI scientist at the AI Now Institute", but the sentiment in this article is diametrically opposite that of the vulnerability research scene. There is contention among vulnerability researchers about the impact of Mythos! But it's not "are frontier models going to shake up vulnerability research and let loose a deluge of critical vulnerabilities" --- software security people overwhelmingly believe that to be true. Rather, it's whether Mythos is truly a step change from 4.7 and 5.5. For vulnerability researchers, the big "news" wasn't Mythos, but rather Carlini's talk from Unprompted, where he got on stage and showed his dumb-seeming "find me zero days" prompt, which actually worked. The big question for vulnerability people now isn't "AI or no AI"; it's "running directly off the model, or building fun and interesting harnesses". Later I spoke with someone who has been professionally acquainted with Khlaaf. Khlaaf is a serious researcher, but not a software security researcher; it's not their field. I think what's happening here is that the BBC doesn't know the difference between AI safety prognosis and software security prognosis, or who to talk to for each topic.
- adrian_b 6mo ago
  I doubt very much that a "find me zero days" prompt worked, because I am not aware of the slightest evidence about this. The Anthropic report that describes the bugs they have found with Mythos in various open-source projects admits that a prompt like "find me zero days" does not work with Mythos. To find bugs, they have run Mythos a large number of times on each file of the scanned project, with different prompts. They have started with a more generic prompt intended to discover whether there are chances to find bugs in that file, in order to decide whether it is worthwhile to run Mythos many times on that file. Then they have used more and more specific prompts, to identify various classes of bugs. Eventually, when it was reasonably certain that a bug exists, Mythos was run one more time, with a prompt requesting the confirmation that the identified bug exists (and the creation of an exploit or patch). Because what you say about Carlini is in obvious contradiction with the technical report about Mythos of Anthropic, I assume that is was just pure BS or some demo run on a fake program with artificial bugs. Or else the so-called prompt was not an LLM prompt, but just the name of a command for a bug-finding harness, which runs the LLM in a loop, with various suitable prompts, as described by Anthropic.
Sol- 6mo ago
You can gauge the quality of the article by seeing Emily Bender quoted, who will insist on stochastic parrots when AI does billions of dollars of economically useful work.
- therobots927 6mo ago
  Better be a lot of billions. There’s about $1 trillion that needs to be paid off.
- ethin 6mo ago
  Can you back this up with actual data, or is this "I believe it to be true" vibes?
- raincole 6mo ago
  What does this even mean? AI can be stochastic parrots and create billions of dollars of revenue at the same time. Steam machines are even dumber, but I'm quite sure that industrial revolution is a real thing.
feverzsj 6mo ago
That's exactly how religion works.
Imnimo 6mo ago
My read is not so much "if we say this is dangerously powerful, it will make people want to buy our product", but rather that there is a significant segment of AI researchers for whom x-risk, AI alignment, etc. is a deal-breaker issue. And so the Sam Altmans of the world have to treat these concerns as serious to attract and retain talent. See for example OpenAI's pledge to dedicate 20% of their compute to safety research. I don't get the sense that Sam ever intended to follow through on that, but it was very important to a segment of his employees. And it seems like trying to play both sides of this at least contributed to Ilya's departure. On the other hand, it seems like Dario is himself a bit more of a true believer.
- james2doyle 6mo ago
  There have been a number of people leaving them because of that bait and switch it seems. That 20% turned out to be something closer to 2% or even 1%
- chis 6mo ago
  Yeah I just don't buy that it would somehow help AI companies for everyone to be existentially afraid of their technology. It seems much more reasonable to think that they really believe the things they're saying, than that it's some kind of 4d chess. Additionally Dario has just been really accurate with his predictions so far. For instance in early 2025 he predicted that nearly 100% of code would be written with AI in 2026.
  - edbaskerville 6mo ago
    Does anyone have good estimates of what percent of real production code is currently being written by LLMs? (& presumably this is rather different for your typical SaaS backend vs. frontend vs. device drivers vs. kernel schedulers...)
    - mbesto 6mo ago
      By all companies? I'd say less than 10% of all LOC today are generated by LLMs.
throwaway911282 6mo ago
dario is the biggest proponent of fear mongering marketing playbook
FiberBundle 6mo ago
Another potential reason, not mentioned in the article, is that open source models obviously pose the biggest threat in the labs' ability to monetize their tech. Anthropic especially seems to be very anti open-source. If frontier models start to plateau and don't have capabilities that truly differentiate them, nobody will pay what the labs would want to charge. Posing the tech as a danger is a way for them to make the government regulate open source models.
- jrumbut 6mo ago
  This is a great point. I'm kind of surprised there isn't a greater proliferation of open source models to do things the public ones won't. I know such things exist, but imagine how many web browsers there would be if all the mainstream ones had the same content restrictions as LLMs. I guess since training them does take cash that raises the bar for what people will do as a prank or on principle.
  - adrian_b 6mo ago
    Training is time-consuming and/or expensive but it is not the main blocker. The main problem is obtaining a big enough training data set. Now, unless you are someone like Google or Microsoft, it has become much harder to scrap data from the Internet than by the time when OpenAI and Anthropic got most of their data.
boh 6mo ago
I think the big secret is that AI is just software. In the same way that a financial firm doesn't all of sudden make a bunch of money because Microsoft shipped an update to Excel, AI is inert without intention. If there's any major successes in AI output it's because a person got it to do that. Claude Code is great, but it will also wipe out a database even though it's instructed not to (I can confirm from experience). The idea that there's some secret innovation that will come out any minute doesn't change the fact that it's software that requires human interaction to work.
- codingdave 6mo ago
  Yes, and it has been said since day one of LLMs that all we need to do is keep things that way - no action without human intervention. Just like it was said that you should never grant AI direct access to change your production systems. But the stories of people who have done exactly that and had their systems damaged and deleted show that people aren't trying to even keep such basic safety nets in place. AI is getting strong enough that if people give some general direction as well as access to production systems of any kind, things can go badly. It is not true that all implementations of agentic AI requires human intervention for all action.
  - stuaxo 6mo ago
    Eh, these same people will attach openclaw to production systems soon and destroy their own companies.
    - flats 6mo ago
      One does not even need OpenClaw to achieve this outcome: https://x.com/lifeof_jer/status/2048103471019434248 https://x.com/lifeof_jer/status/2048103471019434248
      - ffsm8 6mo ago
        Yeeeehaaaaa, the vibes shall never end! On a more serious note, they were mostly f*cked by their paas provider imo. Claude will always do dumb shit. Especially if you tell it to not do something... By doing so you generally increase the likelihood of it doing it. It's even obvious why if you think about it, the pattern of "you had one job, but you failed" or "only this can't happen, it happened!" And all it's other forms is all over literature, online content etc. But their PaaS provider not scoping permissions properly is the root cause, all things considered. While Claude did cause this issue there, something else would've happened eventually otherwise.
throwaway132448 6mo ago
The same reason Palantir does: Its their brand - it’s just marketing. Glad people are finally catching on.
afh1 6mo ago
They want regulation for others but not them. Otherwise there might be competition.
- elondaits 6mo ago
  I think they want regulation for them as well, because they have the money to comply… but regulation eliminates the threat of open source models, foreign models, and small independent companies.
Micanthus 6mo ago
> According to critics, it benefits AI companies to keep you fixated on apocalypse because it distracts from the very real damage they're already doing to the world. Am I not allowed to be concerned about _both_? I do not believe that Sam Altman and other AI company execs believe that the singularity is imminent. If they did, they wouldn't behave so recklessly. Even if they don't care about the rest of humanity, there's too much risk to themselves if they actually believe what they're saying. But I think it's correct to be worried about a potential future AI apocalypse. Personally I doubt that LLMs will scale to full sentience, but I believe we'll get there eventually. And whether it's in 2 years or 200 years I'm worried about it. Plenty of smart people who aren't working for AI companies (and thus have no motive to use it as hype or distraction) hold this belief and it really doesn't seem that crazy. But yeah, obviously let's focus primarily on the real harms AI is causing in our society right now.
- ben_w 6mo ago
  > I do not believe that Sam Altman and other AI company execs believe that the singularity is imminent. If they did, they wouldn't behave so recklessly. Even if they don't care about the rest of humanity, there's too much risk to themselves if they actually believe what they're saying. I don't believe Zuckerberg believes in either the promise or the danger, his presentations are far too mundane. The leaked memos suggest he may simply not care about dangers, which is worse. Altman at least seems to think an LLM can be used as an effective tool for harm and is doing more than the bare minimum to avoid AI analogies of all the accidents and disasters from the industrial age which led to us having health and safety laws, building codes, and consumer product safety laws. Musk clearly thinks laws only exist for him to wield against others. Tries to keep active tools which cause widespread revulsion as if a freedom of speech argument is enough. Amodei seems to actually care even when it hurts Anthropic, as evidenced by saying "no" to the US government. It could be kayfabe, Trump is famous for it after all, but as yet I have no active reason to dismiss Amodei as merely that.
deepsquirrelnet 6mo ago
This is my own take, directly related to this that I posted a little while back. The one thing that I think the article missed is the geopolitical angle they’re also working: * We need to completely deregulate these US companies so China doesn't win and take us over * We need to heavily regulate anybody who is not following the rules that make us the de-facto winner * This is so powerful it will take all the jobs (and therefore if you lead a company that isn't using AI, you will soon be obsolete) * If you don't use AI, you will not be able to function in a future job * We need to lineup an excuse to call our friends in government and turn off the open source spigot when the time is right They have chosen fear as a motivator, and it is clearly working very well. It's easier to use fear now, while it's new and then flip the narrative once people are more familiar with it than to go the other direction. Companies are not just telling a story to hype their product, but why they alone are the ones that should be entrusted to build it.
- mofeien 6mo ago
  "The race to build smarter-than-human AI is a race with no winners." And specifically about the point on China, several people in power in China have also expressed the need to regulate AI and put international structures of governance in place to make sure it will benefit mankind: https://nowinners.ai/#s5-china https://nowinners.ai/#s5-china
  - junofan 6mo ago
    I’ll buy it when they stop lying in the history section of their UN bioweapons self-certification thing. They can do that any time.
- wise0wl 6mo ago
  The outcome of this is, in my opinion, the United States Government classifying and regulating LLMs as something akin to how the ATF classifies weapons, ie. requiring a license to operate an LLM (hosting), with different classifications and determinations on the relative "power" of a particular model and framework, and outright banning most open-source models, like how DIY machine guns or suppressors are banned. Think of a standard for classifying and regulating the self-hosting of open-source models similar to how an FFL works. You can do it, but you must have all your paperwork lined up, with background checks, a valid business license, and if you forget to dot an "i" or cross a "t" the Cyber version of the ATF shows up and shoots your fucking dog.
DalasNoin 6mo ago
Quote from the article: ""AI will probably most likely lead to the end of the world, but in the meantime, there'll be great companies," Altman said in 2015." Altman wasn't even at OpenAI at that point, so why would that be marketing?
- phainopepla2 6mo ago
  > "AI will probably most likely lead to the end of the world, but in the meantime, there'll be great companies," Altman said in 2015. Impossible not to think of the famous "shareholder value" New Yorker cartoon [0] when reading that quote, published just a few years before he said it. [0] https://www.newyorker.com/cartoon/a16995 https://www.newyorker.com/cartoon/a16995
mrwh 6mo ago
What happens when unemployment hits 25%, and youth unemployment hits 50%, in a democracy? That's the real terror here, not hacking.
gdiamos 6mo ago
I originally thought evil killer robots discussions in AI labs was an idea out of Hollywood. Then I saw how effective it was at raising money.
gdiamos 6mo ago
I used to think evil killer robot discussions among AI researchers was an idea based in Hollywood, not science. Then I realized how effective the fear was at fundraising...
staminade 6mo ago
AI company leaders didn't invent this concern about the potential dangers of AI, either as a cause of economic disruption, or as a potential extinction risk. Superintelligence was published in 2014, and even then it wasn't a new topic. Technologists, philosophers and science fiction authors have been discussing and speculating about AI risk for decades. Also, the idea that AI leadership seized on and amplified these concerns purely for marketing purposes isn't plausible. If you're attempting to market a new product to a mass audience, talking about how dangerous and potentially world-ending it is is the most insane strategy you could choose. Any advantage in terms of getting people's attention is going to be totally outweighed by the huge negative associations you are creating in the minds of people who you want to use your product, and the likelihood of bringing unwanted scrutiny and regulation to your nascent industry. (Can you imagine the entire railroad industry saying, "Our new trains are so fast, if they crash everybody on board will die! And all the people in the surrounding area will die! It'll be a catastrophe!" They would not do this. The rational strategy is to underplay the risks and attempt to reassure people. Even more so if you think genuinely believe the risks are being overstated.) Occam's razor suggests that when the AI industry warned about AI risk they believed what they were saying. They had a new, rapidly advancing technology, and absent practical experience of its dangers they referred to pre-existing discussions on the topic, and concluded it was potentially very risky. And so they talked about them in order to prepare the ground in case they turned out to be true. If you warn about AI causing mass unemployment, and then it actually does so, perhaps you can shift the blame to the governments who didn't pay attention and implement social policies to mitigate the effects. I don't think the AI industry deserve too much of our sympathy, but there is a definite "damned if you do, damned if you don't aspect" to AI safety. If they underplay it, they will get accused of ignoring the risks, and if they talk about it, they get accused scaremongering if the worst doesn't happen.
- AndrewKemendo 6mo ago
  Very well put and I think that covers pretty much everything that needs to be said here. In fact it has been AI people who have been leading discussions around AI ethics and the dangers of AI since 1955. This is not new and it is consistent. The new thing is that the average person is now entering into the debate around AI; And like pretty much everything else in the public sphere doing it with entirely no context. I always love when some total novice encounters a problem in a well studied field as though they’re the first one to encounter it. There’s nothing more narcissistic than some person thinking they are unique in their position with absolutely no demonstration of having done their homework on whether or not this is an established topic in an established field. That’s where I place 99.9999% of people who are opening their mouth on this topic. Most of the builders don’t care about this mess and are continuing to work like usual.
p0w3n3d 6mo ago
FUS - Fear Uncertainty Doubt
- p0w3n3d 6mo ago
  Also FUS - Fscking Auto Correct
twobitshifter 6mo ago
I’m afraid of AI but not because I think it’s going to become skynet tomorrow, it’s because of all the social ills that are already clearly attached to it. - Spam - Deep Fakes - Porn - Buggy Software - Economic Bubbles - Degradation in people’s abilities and learned dependence on ChatGPT for basic functions. - Job loss through enshittification ala AI interviews and Telemarketers - Climate Change, noise pollution etc. - Mass Surveillance It’s much more an Idiocracy AI than Terminator.
- __natty__ 6mo ago
  This. We don’t need AI searching for critical bugs in the software, enshittified life quality is already enough.
- pacomerh 6mo ago
  Yeah, this is definitely not sustainable. We're all getting tired of the content quality going downhill. If it's gonna to be like this for a while, I guess new social networks will have to emerge and moderate more? maybe, especially since the government definitely isn't interested in moderating anything. They just want to win races.
  - devilsdata 6mo ago
    How do you ever have a social media network that is immune to this from now? You could with the best intentions start a non-profit, defederated, open-source, grass roots social network. It would go great, until the moment it hits critical mass and becomes prey for people who are willing to piss in the pond to make money. There's no way to defend against it. You can just copy and paste text from an LLM into the reply box.
7777777phil 6mo ago
If your model converges to the same outputs as everyone else's because everyone trained on the same data, the only thing left to differentiate on is brand, and fear is great at building brand. "We are too dangerous to commoditize" pitches better than "we are mostly typical of the internet's median answer", those are kind of the same statement.
api 6mo ago
I’d say the same thing about Palantir. It’s very clear that they are playing into the hatred and speculation about them to puff themselves up and get attention in the “any attention is good attention” era. Being a literal comic book villain syndicate is sexier than being Millennial/GenZ TRW. (I am not saying I approve of all the stuff they are being used for or all the statements of its management.)
andai 6mo ago
Beating the drum of utopia and apocalypse was a suspiciously common tactic in the last century. Also, in a slightly different way, the twenty before that.
registeredcorn 6mo ago
Are you telling me that it's sexier to say, "In its current form, we cannot contain its power" rather than, "We're working out the last set of bugs before the start of Q3"?
skybrian 6mo ago
It’s an extraordinary situation and I’m wondering what sort of analogies make sense. If there were tobacco companies warning everyone who would listen in the 1950’s that cigarettes cause cancer, it would be, like, points for honesty, but why don’t you stop selling them then? The difference being that there are a lot of good uses for AI chat and it doesn’t directly harm most people. It seems like the customers who would misuse AI are getting left out of the discussion? It’s as if arms dealers were being solely blamed for war, or if arms dealers were expected to stop wars. The difference being that a single, general purpose product that can do such a wide variety of things isn’t really comparable to making weapons that are only good for one thing. Maybe it’s as if car manufacturers in the early 20th century were predicting highways, traffic, and pollution. Or imagine if early dot-com companies were predicting the various dangers of social networks?
netcan 6mo ago
I remember thinking Altman seemed to be over-reacting and fanning the flames about "Ai bias" circa gpt3. There was a when little panic about the fear of bigoted computers at that point. But... it got a lot of earned advertising and they also sort of did a "pre-burn." They saturating the space with "bigoted Ai concern" for a while, and now I don't ever see it come up. There's a "get ahead of the inevitable" thing going on. Also, obviously, prospectus hype. Besides all that, these are geeks and they're excited. This is what an excited geek looks like.
nyc_data_geek1 6mo ago
Because your fear is their marketing, is their valuation. There, saved you a click.
deferredgrant 6mo ago
[flagged]
dimva 6mo ago
They say AI will destroy humanity because they believe it. OpenAI and Anthropic were created by people who believed this. There's nothing nefarious about them saying this. Why are they still building it? Because each team thinks that THEY are the ones who can prevent it from destroying humanity, but they have to get to AGI first, before the other teams make an AI that does destroy humanity. But also, if AGI doesn't destroy humanity, it would be the most powerful weapon in the world, and they want to be the ones in control of it. Keeping the focus on Armageddon distracts from the real and severe problems that arise if a single person, or even a small group, controls an AGI.
- deleted 6mo ago
  [deleted]
fokweo433 6mo ago
[dead]
GuB-42 6mo ago
For me, there are two reasons: One is simple marketing. "We have a product that gives you superpowers, but we have to be careful, imagine if everyone had superpowers, it would be chaos". Most people reaction would be "I don't care about the others, give me these superpowers". An then, when you finally have it, you realize it is just not that super, but the next one will be, or so they say. The other is that established players want to build a moat. If by scare tactics, they can convince regulators to not give newcomers the same freedom they had when they started up, good for them. Bonus points if they are the ones who make the rules. They are they experts, we should follow them, right? And yes, it is touched on in the article, it is a way to hide smaller but real problems behind bigger but hypothetical problems. When we see Skynet, we don't see the copyright problems, slop and unreliability. While they talk about the vulnerabilities the new LLM find, they don't talk about the ones it introduces in your vibe-coded app.

290 points by rolph 6mo ago

Honestly we should have learned this claim from AI companies was purely fear-mongering back when GPT-2 was "too dangerous to release".

detectivestory 6mo ago
That's true but in reality I think people are far more afraid of AI in terms of how it is being used in warfare and policing. Automatic target detection and deployment of drones, or even how it might simply make their role at work redundant etc
- scratchyone 6mo ago
  100% agreed. That's part of the issue imo, these companies pretend their new models are "too dangerous" to seem like they care about the world, yet they have no qualms deploying existing models in warfare or bragging about impending mass-unemployment.
- notrealyme123 6mo ago
  I am to be honest not sure what I am more scared. AI shaping warfare Vs. Using AI to justify outrageous warfare
  - scratchyone 6mo ago
    Don't worry, these companies will make sure we get to experience both nightmare futures.
  - yieldcrv 6mo ago
    that’s not a war crime, that’s boundary setting, and honestly, that’s rare would you like me to list the applicable sections of the Geneva convention?
  - MSFT_Edging 6mo ago
    We sadly don't need AI to justify outrageous warfare. You just need to remember when the US invaded Iraq over WMDs, including a full investigation into the WMDs that never found any. We then invaded anyway, to the detriment of everyone except defense contractors.
- palmotea 6mo ago
  > That's true but in reality I think people are far more afraid of AI in terms of how it is being used in warfare and policing. Automatic target detection and deployment of drones, or even how it might simply make their role at work redundant etc I think the last one should be first on the list: regular people are afraid AI will negatively affect their economic security (i.e. knowledge and service workers will get the rust-belt factory worker treatment). And the potential of giving knowledge and service workers the rust-belt factory worker treatment is exactly what makes Wall Street excited about AI and has the AI company leaders salivating about the profit they can make. Warfare, policing, bio-engineered viruses are theoretical and far down the list.
  - detectivestory 6mo ago
    Its already being used in warfare though.
    - palmotea 6mo ago
      > Its already being used in warfare though. What I mean is theoretical to the common person. They don't have killbot drones hunting them down, and are unlikely to have that experience anytime soon. But most people have jobs, most people would be hard-hit if they lost theirs, lots of people lose theirs, and our elites are just itching to make that happen. I'm certainly most worried about AI: my employer started an ongoing silent layoff campaign about the same time they started enforcing AI usage. I don't think those are unconnected.
  - wongarsu 6mo ago
    Not to mention that "automatic target detection" was primarily enabled by the ~2016-2020 AI hype/boom around image recognition, not the 2022-current hype/boom around LLMs
- mossTechnician 6mo ago
  To me, the more interesting divergence in discussion is on its capabilities. AI industry insiders (including "safety" groups like ControlAI) talk about the dangers only in terms of its power: "Scheming", job loss, breaking containment, the New Cold War with China. Critics outside the industry talk in terms of its lack of power: Inaccuracy, erroneous translation of user intent, failure to deliver on its promises and investment, environmental cost from the former, and ultimately the danger of people in power (e.g. law enforcement, military officials) treating its output as valid and unbiased, or simply laundering their wishes through it.
- chasd00 6mo ago
  AI has been used in defense for a while now, a modern tomahawk cruise missile and its associated targeting systems is a good example. I think most people fear AI taking their job and only source of income.
  - sublinear 6mo ago
    Linear regression?
- sublinear 6mo ago
  These were all already very valid concerns long before this era of "AI" or computational power. The broader public is just now barely beginning to understand because all they have to do is ask a chatbot. AI does not enable new capabilities, but it does aggregate an idea into a rough sketch and do it quickly on-demand. None of this really means it will play out that way. The devil is in the details. What it does mean is much more nuanced attention on the politics and money because that's where the power always was.
  - detectivestory 6mo ago
    AI does enable new capabilities when it comes to constant mass surveillance, and automated weaponry.
    - sublinear 6mo ago
      No it doesn't. We already have all of that right now and have had it for decades. The big investment into Project Stargate is all about managing risk. The government contractor and security clearance situation is out of control. As well, every human mistake is costly and time consuming to address. If you instead blame it on AI, you can skip the court proceedings and postmortems. The other part of this is likely an attempt to surface information with summaries and shorten the chain of command. This is just a power grab and a dangerous dismissal of necessary implementation detail. It's a tantrum being thrown by ignorant people at the top being displaced. We live in an ever-complicated world that demands more experienced leadership than we have available. AI is their hail mary pass. LLMs are being abused as a political battering ram. They are not the technological breakthrough advertised. The AI label is borderline absurd. AGI even moreso. NLP is an accessibility tool at best.
maplethorpe 6mo ago
It seems like they were correct, to me.
- iugtmkbdfil834 6mo ago
  I was going to say.. I think people in general have this weird understanding of the word dangerous. Just because something is not movie level dramatic and/or does not generate over the top violence does not automatically make it less dangerous. In a sense, just the fact that is benign on the surface and allowed to embed in our day to day life is what makes the upcoming rug pull so painful. And I am saying this as a person who actually likes this tech.
- elar_verole 6mo ago
  Yes, I love how everyone uses this argument, when what they were saying was among the lines of "GPT-2 would make it too easy to generate spam, deepfakes, content to manipulate opinion..." (not the actual quote but something like that). Turns out it was completely correct if you look at the state of the internet right now. Obviously, they still overhype and oversell this end of humanity stuff, but this argument regurgitated ad-nauseam is not THAT great of an example when you think about it.
mofeien 6mo ago
Given that his reason for saying GPT-2 was too dangerous to release was that the world needed more time to prepare for the effects of this technology, and given that the following models were basically scaled-up versions of it and killed social media, news reporting and other kinds of communication, I'd say he was right about the dangers of it.
- scratchyone 6mo ago
  funny how he didn't care about ethics the moment it was more profitable to release it than to talk about dangers.

yuhmahp 6mo ago

Assuming this article isn't written by an AI

scratchyone 6mo ago
I mean, it's the BBC and the article doesn't have any typical AI tells, where is this idea that it's AI written coming from???
- yuhmahp 6mo ago
  I'm assuming it's possible that an AI can deduct the potential 3rd degree result from an article like this on the BBC. Not talking about the wording
  - MajorTakeaway 6mo ago
    >sees any random article headline, automatically assumes it's AI.
    - hxugufjfjf 6mo ago
      Pretty reasonable assumption these days unfortunately.
      - Jtarii 6mo ago
        Just saying things with no evidence is not reasonable
  - philipwhiuk 6mo ago
    Roko's basilisk is a very tedious idea.

InputName 6mo ago

In lieu of a technological moat, companies search for regulatory capture.

SpicyLemonZest 6mo ago

> It's a strange way for any company to talk about its own work. You don't hear McDonald's announcing that it's created a burger so terrifyingly delicious that it would be unethical to grill it for the public. > Here's one theory. But the author never gets back to this! It's the main observation the theory has to account for; why don't we see other companies speak this way, if it's such an effective strategy for deflecting non-apocalyptic concerns?

scratchyone 6mo ago
tbf most companies don't have a potentially world-ending product. only real similar field is defense contractors who typically can't brag about unreleased ideas as they're classified.
- SpicyLemonZest 6mo ago
  I agree, but the experts the author cites do not. Professor Valor believes that AI is a mirror and any existential fears of it are just reflected fear of ourselves; Professor Bender believes that AI is a con and all the people who say it's powerful enough to be world-ending are lying. Anyone who concedes the premise that AI has a genuine potential to be world-ending is, I think, on the AI labs' side of this debate.
  - jrumbut 6mo ago
    We are not such a bad thing to fear. This technology interacts socially, so even if it can't jailbreak itself on a technology-level (which feels like a tough guarantee to make at this point) it can simply ask someone to do a bad thing and there is some chance they'll do it. The same way a human leader does. The first kids who have only faint memories of a time before chatbots will be entering the military in 6-7 years. You have to assume they are acting as best friends, therapists, or even surrogate parents for a substantial number of kids right now. We are going to need years to figure out what to do about this technology. I think some impetus to get that process started is a good idea.
tangotaylor 6mo ago

dicksent 6mo ago

gotta bring up the hype

sixtyj 6mo ago

Article mentions a book "The AI Con" that argues that much of what is labeled "artificial intelligence" is a misleading term that obscures ordinary automation while concentrating power in a small number of technology firms. So fear-mongering seems to be just a tool how to get attention and more customers. Hey ma, I use very dangerous tool now. I am OG.

redsocksfan45 6mo ago
[dead]

tangotaylor 6mo ago

Finally the media is catching on. Lee Vinsel's criti-hype article nailed this 5 years ago, before we even had the chatbot economy we do now: https://sts-news.medium.com/youre-doing-it-wrong-notes-on-criticism-and-technology-hype-18b08b4307e5 https://sts-news.medium.com/youre-doing-it-wrong-notes-on-cr...

b65e8bee43c2ed0 6mo ago
the media is not catching on, they've been looping through 'AI is going to kill us all!' when they want to sell fear and 'Look at all the energy and water AI companies are pointlessly wasting!' when they want to sell anger. the writers and the editors know exactly what they're doing - spreading FUD and creating controversy out of thin air. some of it is done for-profit, some for-agenda, and all of it with malicious intent.

baggachipz 6mo ago

Why wouldn't they continue crying wolf when it always gets them free advertising from a gullible/complicit press?

firefoxd 6mo ago

"you will all lose your jobs and it will wipe out half of humanity." If you lead with this, people will stop questioning why their sprint velocity hasn't increased 10 fold. Managers start asking leads, instead of hiring more devs can we add Agent.md to our repos? The Apocalypse sells. They are afraid that you'll find out that AI is just another useful tool. That's the real threat, not to humanity, but to their hype. Edit: i made a video about this recently: https://youtu.be/nB0Vz-fh8EI https://youtu.be/nB0Vz-fh8EI

bryan0 6mo ago

> Why do AI companies want us to be afraid of them? ... According to critics, it benefits AI companies to keep you fixated on apocalypse because it distracts from the very real damage they're already doing to the world. People seem unable to make up their mind if AI is very dangerous or is it not. I think what the AI companies and this author agree on, is that this technology is potentially extremely dangerous. AI impacts labor markets, the environment, warfare, mental health, etc... It's harder now to find things which it will not impact. So if we agree that AI is potentially dangerous, it makes the title question moot: Both AI companies and this author want people to be aware of the dangers that AI poses to society. The real question is what do we do about it? The nuance here is that AI can be incredible positive as well. It's like the invention of fire, you can use it for good or bad, and there will be many unintended consequences along the way. We could legislate and ban AI tech. People have proposed this seriously, yet this feels completely unrealistic. If the US bans AI research, then this research will move elsewhere. I think it is like trying to ban fire because it's dangerous: some groups will learn to work with fire and they will get an extreme advantage over those groups that don't. (or they will destroy themselves in the process). So maybe instead of demonizing the AI companies, we have a nuanced debate about this tech and propose solutions that our best for our society?

dodu_ 6mo ago
>So maybe instead of demonizing the AI companies, we have a nuanced debate about this tech and propose solutions that our best for our society? These are not mutually exclusive. Calling out the demonic behavior of trying to coerce people into using your product out of fear is not an indictment of the underlying technology itself.
- bryan0 6mo ago
  One of the points I was trying to make is that the statement: > trying to coerce people into using your product out of fear is nonsense. Everyone agrees that there are legitimate reasons to be fearful of this technology, this is not a fabrication, but we need to figure out how to proceed in a safe and constructive way. What "coercion" is occurring here? Either you find the technology valuable and you want to pay for it, or you find it not useful (or worse harmful), and you do not want to pay for it. Maybe another way of putting it, what do you think the frontier AI companies should do in this situation? It seems that being straightforward with the dangers is correct thing to do, and probably being overly cautious is prudent. You could go further and argue they should slow down or stop development, but that is something that the govt should impose, we should not expect or trust the companies to do this themselves. Ironically, in the Anthropic / Pentagon case, we have Anthropic trying to pump the brakes and put up guardrails while the govt wants to go full-steam ahead with autonomous warfare. The other issue with slowing down / pausing development is it requires an unheard of level of agreement, even with companies in China, or else it will probably not be effective. You could argue this is not even possible at this point.

tptacek 6mo ago

I have never heard of "Heidy Khlaaf, chief AI scientist at the AI Now Institute", but the sentiment in this article is diametrically opposite that of the vulnerability research scene. There is contention among vulnerability researchers about the impact of Mythos! But it's not "are frontier models going to shake up vulnerability research and let loose a deluge of critical vulnerabilities" --- software security people overwhelmingly believe that to be true. Rather, it's whether Mythos is truly a step change from 4.7 and 5.5. For vulnerability researchers, the big "news" wasn't Mythos, but rather Carlini's talk from Unprompted, where he got on stage and showed his dumb-seeming "find me zero days" prompt, which actually worked. The big question for vulnerability people now isn't "AI or no AI"; it's "running directly off the model, or building fun and interesting harnesses". Later I spoke with someone who has been professionally acquainted with Khlaaf. Khlaaf is a serious researcher, but not a software security researcher; it's not their field. I think what's happening here is that the BBC doesn't know the difference between AI safety prognosis and software security prognosis, or who to talk to for each topic.

adrian_b 6mo ago
I doubt very much that a "find me zero days" prompt worked, because I am not aware of the slightest evidence about this. The Anthropic report that describes the bugs they have found with Mythos in various open-source projects admits that a prompt like "find me zero days" does not work with Mythos. To find bugs, they have run Mythos a large number of times on each file of the scanned project, with different prompts. They have started with a more generic prompt intended to discover whether there are chances to find bugs in that file, in order to decide whether it is worthwhile to run Mythos many times on that file. Then they have used more and more specific prompts, to identify various classes of bugs. Eventually, when it was reasonably certain that a bug exists, Mythos was run one more time, with a prompt requesting the confirmation that the identified bug exists (and the creation of an exploit or patch). Because what you say about Carlini is in obvious contradiction with the technical report about Mythos of Anthropic, I assume that is was just pure BS or some demo run on a fake program with artificial bugs. Or else the so-called prompt was not an LLM prompt, but just the name of a command for a bug-finding harness, which runs the LLM in a loop, with various suitable prompts, as described by Anthropic.

Sol- 6mo ago

You can gauge the quality of the article by seeing Emily Bender quoted, who will insist on stochastic parrots when AI does billions of dollars of economically useful work.

therobots927 6mo ago
Better be a lot of billions. There’s about $1 trillion that needs to be paid off.
ethin 6mo ago
Can you back this up with actual data, or is this "I believe it to be true" vibes?
raincole 6mo ago
What does this even mean? AI can be stochastic parrots and create billions of dollars of revenue at the same time. Steam machines are even dumber, but I'm quite sure that industrial revolution is a real thing.

feverzsj 6mo ago

That's exactly how religion works.

Imnimo 6mo ago

My read is not so much "if we say this is dangerously powerful, it will make people want to buy our product", but rather that there is a significant segment of AI researchers for whom x-risk, AI alignment, etc. is a deal-breaker issue. And so the Sam Altmans of the world have to treat these concerns as serious to attract and retain talent. See for example OpenAI's pledge to dedicate 20% of their compute to safety research. I don't get the sense that Sam ever intended to follow through on that, but it was very important to a segment of his employees. And it seems like trying to play both sides of this at least contributed to Ilya's departure. On the other hand, it seems like Dario is himself a bit more of a true believer.

james2doyle 6mo ago
There have been a number of people leaving them because of that bait and switch it seems. That 20% turned out to be something closer to 2% or even 1%
chis 6mo ago
Yeah I just don't buy that it would somehow help AI companies for everyone to be existentially afraid of their technology. It seems much more reasonable to think that they really believe the things they're saying, than that it's some kind of 4d chess. Additionally Dario has just been really accurate with his predictions so far. For instance in early 2025 he predicted that nearly 100% of code would be written with AI in 2026.
- edbaskerville 6mo ago
  Does anyone have good estimates of what percent of real production code is currently being written by LLMs? (& presumably this is rather different for your typical SaaS backend vs. frontend vs. device drivers vs. kernel schedulers...)
  - mbesto 6mo ago
    By all companies? I'd say less than 10% of all LOC today are generated by LLMs.

throwaway911282 6mo ago

dario is the biggest proponent of fear mongering marketing playbook

FiberBundle 6mo ago

Another potential reason, not mentioned in the article, is that open source models obviously pose the biggest threat in the labs' ability to monetize their tech. Anthropic especially seems to be very anti open-source. If frontier models start to plateau and don't have capabilities that truly differentiate them, nobody will pay what the labs would want to charge. Posing the tech as a danger is a way for them to make the government regulate open source models.

jrumbut 6mo ago
This is a great point. I'm kind of surprised there isn't a greater proliferation of open source models to do things the public ones won't. I know such things exist, but imagine how many web browsers there would be if all the mainstream ones had the same content restrictions as LLMs. I guess since training them does take cash that raises the bar for what people will do as a prank or on principle.
- adrian_b 6mo ago
  Training is time-consuming and/or expensive but it is not the main blocker. The main problem is obtaining a big enough training data set. Now, unless you are someone like Google or Microsoft, it has become much harder to scrap data from the Internet than by the time when OpenAI and Anthropic got most of their data.

boh 6mo ago

I think the big secret is that AI is just software. In the same way that a financial firm doesn't all of sudden make a bunch of money because Microsoft shipped an update to Excel, AI is inert without intention. If there's any major successes in AI output it's because a person got it to do that. Claude Code is great, but it will also wipe out a database even though it's instructed not to (I can confirm from experience). The idea that there's some secret innovation that will come out any minute doesn't change the fact that it's software that requires human interaction to work.

codingdave 6mo ago
Yes, and it has been said since day one of LLMs that all we need to do is keep things that way - no action without human intervention. Just like it was said that you should never grant AI direct access to change your production systems. But the stories of people who have done exactly that and had their systems damaged and deleted show that people aren't trying to even keep such basic safety nets in place. AI is getting strong enough that if people give some general direction as well as access to production systems of any kind, things can go badly. It is not true that all implementations of agentic AI requires human intervention for all action.
- stuaxo 6mo ago
  Eh, these same people will attach openclaw to production systems soon and destroy their own companies.
  - flats 6mo ago
    One does not even need OpenClaw to achieve this outcome: https://x.com/lifeof_jer/status/2048103471019434248 https://x.com/lifeof_jer/status/2048103471019434248
    - ffsm8 6mo ago
      Yeeeehaaaaa, the vibes shall never end! On a more serious note, they were mostly f*cked by their paas provider imo. Claude will always do dumb shit. Especially if you tell it to not do something... By doing so you generally increase the likelihood of it doing it. It's even obvious why if you think about it, the pattern of "you had one job, but you failed" or "only this can't happen, it happened!" And all it's other forms is all over literature, online content etc. But their PaaS provider not scoping permissions properly is the root cause, all things considered. While Claude did cause this issue there, something else would've happened eventually otherwise.

throwaway132448 6mo ago

The same reason Palantir does: Its their brand - it’s just marketing. Glad people are finally catching on.

afh1 6mo ago

They want regulation for others but not them. Otherwise there might be competition.

elondaits 6mo ago
I think they want regulation for them as well, because they have the money to comply… but regulation eliminates the threat of open source models, foreign models, and small independent companies.

Micanthus 6mo ago

> According to critics, it benefits AI companies to keep you fixated on apocalypse because it distracts from the very real damage they're already doing to the world. Am I not allowed to be concerned about _both_? I do not believe that Sam Altman and other AI company execs believe that the singularity is imminent. If they did, they wouldn't behave so recklessly. Even if they don't care about the rest of humanity, there's too much risk to themselves if they actually believe what they're saying. But I think it's correct to be worried about a potential future AI apocalypse. Personally I doubt that LLMs will scale to full sentience, but I believe we'll get there eventually. And whether it's in 2 years or 200 years I'm worried about it. Plenty of smart people who aren't working for AI companies (and thus have no motive to use it as hype or distraction) hold this belief and it really doesn't seem that crazy. But yeah, obviously let's focus primarily on the real harms AI is causing in our society right now.

ben_w 6mo ago
> I do not believe that Sam Altman and other AI company execs believe that the singularity is imminent. If they did, they wouldn't behave so recklessly. Even if they don't care about the rest of humanity, there's too much risk to themselves if they actually believe what they're saying. I don't believe Zuckerberg believes in either the promise or the danger, his presentations are far too mundane. The leaked memos suggest he may simply not care about dangers, which is worse. Altman at least seems to think an LLM can be used as an effective tool for harm and is doing more than the bare minimum to avoid AI analogies of all the accidents and disasters from the industrial age which led to us having health and safety laws, building codes, and consumer product safety laws. Musk clearly thinks laws only exist for him to wield against others. Tries to keep active tools which cause widespread revulsion as if a freedom of speech argument is enough. Amodei seems to actually care even when it hurts Anthropic, as evidenced by saying "no" to the US government. It could be kayfabe, Trump is famous for it after all, but as yet I have no active reason to dismiss Amodei as merely that.

deepsquirrelnet 6mo ago

This is my own take, directly related to this that I posted a little while back. The one thing that I think the article missed is the geopolitical angle they’re also working: * We need to completely deregulate these US companies so China doesn't win and take us over * We need to heavily regulate anybody who is not following the rules that make us the de-facto winner * This is so powerful it will take all the jobs (and therefore if you lead a company that isn't using AI, you will soon be obsolete) * If you don't use AI, you will not be able to function in a future job * We need to lineup an excuse to call our friends in government and turn off the open source spigot when the time is right They have chosen fear as a motivator, and it is clearly working very well. It's easier to use fear now, while it's new and then flip the narrative once people are more familiar with it than to go the other direction. Companies are not just telling a story to hype their product, but why they alone are the ones that should be entrusted to build it.

mofeien 6mo ago
"The race to build smarter-than-human AI is a race with no winners." And specifically about the point on China, several people in power in China have also expressed the need to regulate AI and put international structures of governance in place to make sure it will benefit mankind: https://nowinners.ai/#s5-china https://nowinners.ai/#s5-china
- junofan 6mo ago
  I’ll buy it when they stop lying in the history section of their UN bioweapons self-certification thing. They can do that any time.
wise0wl 6mo ago
The outcome of this is, in my opinion, the United States Government classifying and regulating LLMs as something akin to how the ATF classifies weapons, ie. requiring a license to operate an LLM (hosting), with different classifications and determinations on the relative "power" of a particular model and framework, and outright banning most open-source models, like how DIY machine guns or suppressors are banned. Think of a standard for classifying and regulating the self-hosting of open-source models similar to how an FFL works. You can do it, but you must have all your paperwork lined up, with background checks, a valid business license, and if you forget to dot an "i" or cross a "t" the Cyber version of the ATF shows up and shoots your fucking dog.

DalasNoin 6mo ago

Quote from the article: ""AI will probably most likely lead to the end of the world, but in the meantime, there'll be great companies," Altman said in 2015." Altman wasn't even at OpenAI at that point, so why would that be marketing?

phainopepla2 6mo ago
> "AI will probably most likely lead to the end of the world, but in the meantime, there'll be great companies," Altman said in 2015. Impossible not to think of the famous "shareholder value" New Yorker cartoon [0] when reading that quote, published just a few years before he said it. [0] https://www.newyorker.com/cartoon/a16995 https://www.newyorker.com/cartoon/a16995

mrwh 6mo ago

What happens when unemployment hits 25%, and youth unemployment hits 50%, in a democracy? That's the real terror here, not hacking.

gdiamos 6mo ago

I originally thought evil killer robots discussions in AI labs was an idea out of Hollywood. Then I saw how effective it was at raising money.

gdiamos 6mo ago

I used to think evil killer robot discussions among AI researchers was an idea based in Hollywood, not science. Then I realized how effective the fear was at fundraising...

staminade 6mo ago

AI company leaders didn't invent this concern about the potential dangers of AI, either as a cause of economic disruption, or as a potential extinction risk. Superintelligence was published in 2014, and even then it wasn't a new topic. Technologists, philosophers and science fiction authors have been discussing and speculating about AI risk for decades. Also, the idea that AI leadership seized on and amplified these concerns purely for marketing purposes isn't plausible. If you're attempting to market a new product to a mass audience, talking about how dangerous and potentially world-ending it is is the most insane strategy you could choose. Any advantage in terms of getting people's attention is going to be totally outweighed by the huge negative associations you are creating in the minds of people who you want to use your product, and the likelihood of bringing unwanted scrutiny and regulation to your nascent industry. (Can you imagine the entire railroad industry saying, "Our new trains are so fast, if they crash everybody on board will die! And all the people in the surrounding area will die! It'll be a catastrophe!" They would not do this. The rational strategy is to underplay the risks and attempt to reassure people. Even more so if you think genuinely believe the risks are being overstated.) Occam's razor suggests that when the AI industry warned about AI risk they believed what they were saying. They had a new, rapidly advancing technology, and absent practical experience of its dangers they referred to pre-existing discussions on the topic, and concluded it was potentially very risky. And so they talked about them in order to prepare the ground in case they turned out to be true. If you warn about AI causing mass unemployment, and then it actually does so, perhaps you can shift the blame to the governments who didn't pay attention and implement social policies to mitigate the effects. I don't think the AI industry deserve too much of our sympathy, but there is a definite "damned if you do, damned if you don't aspect" to AI safety. If they underplay it, they will get accused of ignoring the risks, and if they talk about it, they get accused scaremongering if the worst doesn't happen.

AndrewKemendo 6mo ago
Very well put and I think that covers pretty much everything that needs to be said here. In fact it has been AI people who have been leading discussions around AI ethics and the dangers of AI since 1955. This is not new and it is consistent. The new thing is that the average person is now entering into the debate around AI; And like pretty much everything else in the public sphere doing it with entirely no context. I always love when some total novice encounters a problem in a well studied field as though they’re the first one to encounter it. There’s nothing more narcissistic than some person thinking they are unique in their position with absolutely no demonstration of having done their homework on whether or not this is an established topic in an established field. That’s where I place 99.9999% of people who are opening their mouth on this topic. Most of the builders don’t care about this mess and are continuing to work like usual.

p0w3n3d 6mo ago

FUS - Fear Uncertainty Doubt

p0w3n3d 6mo ago
Also FUS - Fscking Auto Correct

twobitshifter 6mo ago

I’m afraid of AI but not because I think it’s going to become skynet tomorrow, it’s because of all the social ills that are already clearly attached to it. - Spam - Deep Fakes - Porn - Buggy Software - Economic Bubbles - Degradation in people’s abilities and learned dependence on ChatGPT for basic functions. - Job loss through enshittification ala AI interviews and Telemarketers - Climate Change, noise pollution etc. - Mass Surveillance It’s much more an Idiocracy AI than Terminator.

__natty__ 6mo ago
This. We don’t need AI searching for critical bugs in the software, enshittified life quality is already enough.
pacomerh 6mo ago
Yeah, this is definitely not sustainable. We're all getting tired of the content quality going downhill. If it's gonna to be like this for a while, I guess new social networks will have to emerge and moderate more? maybe, especially since the government definitely isn't interested in moderating anything. They just want to win races.
- devilsdata 6mo ago
  How do you ever have a social media network that is immune to this from now? You could with the best intentions start a non-profit, defederated, open-source, grass roots social network. It would go great, until the moment it hits critical mass and becomes prey for people who are willing to piss in the pond to make money. There's no way to defend against it. You can just copy and paste text from an LLM into the reply box.

7777777phil 6mo ago

If your model converges to the same outputs as everyone else's because everyone trained on the same data, the only thing left to differentiate on is brand, and fear is great at building brand. "We are too dangerous to commoditize" pitches better than "we are mostly typical of the internet's median answer", those are kind of the same statement.

api 6mo ago

I’d say the same thing about Palantir. It’s very clear that they are playing into the hatred and speculation about them to puff themselves up and get attention in the “any attention is good attention” era. Being a literal comic book villain syndicate is sexier than being Millennial/GenZ TRW. (I am not saying I approve of all the stuff they are being used for or all the statements of its management.)

andai 6mo ago

Beating the drum of utopia and apocalypse was a suspiciously common tactic in the last century. Also, in a slightly different way, the twenty before that.

registeredcorn 6mo ago

Are you telling me that it's sexier to say, "In its current form, we cannot contain its power" rather than, "We're working out the last set of bugs before the start of Q3"?

skybrian 6mo ago

It’s an extraordinary situation and I’m wondering what sort of analogies make sense. If there were tobacco companies warning everyone who would listen in the 1950’s that cigarettes cause cancer, it would be, like, points for honesty, but why don’t you stop selling them then? The difference being that there are a lot of good uses for AI chat and it doesn’t directly harm most people. It seems like the customers who would misuse AI are getting left out of the discussion? It’s as if arms dealers were being solely blamed for war, or if arms dealers were expected to stop wars. The difference being that a single, general purpose product that can do such a wide variety of things isn’t really comparable to making weapons that are only good for one thing. Maybe it’s as if car manufacturers in the early 20th century were predicting highways, traffic, and pollution. Or imagine if early dot-com companies were predicting the various dangers of social networks?

netcan 6mo ago

I remember thinking Altman seemed to be over-reacting and fanning the flames about "Ai bias" circa gpt3. There was a when little panic about the fear of bigoted computers at that point. But... it got a lot of earned advertising and they also sort of did a "pre-burn." They saturating the space with "bigoted Ai concern" for a while, and now I don't ever see it come up. There's a "get ahead of the inevitable" thing going on. Also, obviously, prospectus hype. Besides all that, these are geeks and they're excited. This is what an excited geek looks like.

nyc_data_geek1 6mo ago

Because your fear is their marketing, is their valuation. There, saved you a click.

deferredgrant 6mo ago

[flagged]

dimva 6mo ago

They say AI will destroy humanity because they believe it. OpenAI and Anthropic were created by people who believed this. There's nothing nefarious about them saying this. Why are they still building it? Because each team thinks that THEY are the ones who can prevent it from destroying humanity, but they have to get to AGI first, before the other teams make an AI that does destroy humanity. But also, if AGI doesn't destroy humanity, it would be the most powerful weapon in the world, and they want to be the ones in control of it. Keeping the focus on Armageddon distracts from the real and severe problems that arise if a single person, or even a small group, controls an AGI.

deleted 6mo ago
[deleted]

fokweo433 6mo ago

[dead]

GuB-42 6mo ago

For me, there are two reasons: One is simple marketing. "We have a product that gives you superpowers, but we have to be careful, imagine if everyone had superpowers, it would be chaos". Most people reaction would be "I don't care about the others, give me these superpowers". An then, when you finally have it, you realize it is just not that super, but the next one will be, or so they say. The other is that established players want to build a moat. If by scare tactics, they can convince regulators to not give newcomers the same freedom they had when they started up, good for them. Bonus points if they are the ones who make the rules. They are they experts, we should follow them, right? And yes, it is touched on in the article, it is a way to hide smaller but real problems behind bigger but hypothetical problems. When we see Skynet, we don't see the copyright problems, slop and unreliability. While they talk about the vulnerabilities the new LLM find, they don't talk about the ones it introduces in your vibe-coded app.