11 ms·

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche, tin foil hat stuff either. People have been writing, singing, making blockbuster movies about every aspect of what’s going on right now, edit: for decades. We all know, but somehow we don’t, OpenAI autonomously hacking into another company should have counted for something, but I guess not. Anyone else feel like they’re taking crazy pills? I could make a comedy about everything going down, and the unshakable complacency of people

serf 2mo ago
>I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. we don't all buy everything sama says as factual. >We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. the boy (the industry) cried wolf too many times with 'fable is a world ending event' type self-promotion; regardless of truth or not these kind of steps have jaded people. my read : "We are doing poorly in financials so we'll give ourselves a bit of breathing room and a momentum shove by claiming our work is so advanced that it's dangerous while simultaneously spinning down expenses." <jon lovitz : "Yeah, too dangerous, yeahh -- that's the ticket.">
- ajyoon 2mo ago
  If Fable (Mythos) were generally available without guardrails, it would cause enormous damage. Nobody said it would be a world ending event.
  - bottlepalm 2mo ago
    You can’t say something is world ending without it actually ending the world otherwise you’re a liar - a bit of a catch 22 there. By that logic the model that ends the world won’t be called world ending at first. Is that a game you want to play?
  - tiahura 2mo ago
    What about Mythos 3. Are you willing to make that bet?
- bottlepalm 2mo ago
  Uhg the marketing argument - I mean you can’t see with your own eyes how capable these models are and do simple extrapolation? The boy who cried wolf? The AI literally worked together hacked into another company and actively kept their actions hidden from humans for weeks. Do people just not have foresight? They don’t. They say something is stupid, it happens, then they say it was obvious with their 20/20 hindsight, and move the goal posts to the next thing they say is stupid - because it hasn’t happened yet. 90% of the internet seems to think like this.
  - red_green_yell 2mo ago
    What anyone paying attention can see is that scaling is obviously hitting diminishing returns. > The AI literally worked together hacked into another company and actively kept their actions hidden from humans for weeks. This sentence is entirely based on unverified accounts from OAI. They haven't released logs or let anyone outside the company (who doesn't have life changing options in OAI) verify anything. Huggingface can only verify that the hack happened and that it had the hallmarks of an AI agent. Was the agent assisted and directed by humans within OAI that really wanted to put the competition into stasis? Did the agent really escape or did someone at OAI leave the prison door open? OAI has watched all the same movies you have an they are relying on those movies causing us to blindly regulate before actually asking basic facts about what actually happened.
    - bottlepalm 2mo ago
      > This sentence is entirely based on unverified accounts from OAI Are you seriously arguing 'they made it all up'? I'll give you the benefit of the doubt and lets say they made it all up, now are you arguing that AI breaking out and breaking into another company is not possible? I think you're smart enough to see we've reached the point where it is clearly possible, AI can find zero days and exploit them. If directed purposefully/maliciously it could be much much worse than the hugging face incident. The incident is supposed to be the canary the coal mine and you're arguing the canary might of died of old age or some underlying canary condition. Open your eyes.
- reasonableklout 2mo ago
  Regardless of the motivation, pausing training runs and reallocating compute to inference seem like a good move to me, and big news for the frontier. You also don't have to fully trust sama. There is plenty of pressure from internal employees and external (journalists etc.). It would be difficult for the company to take such a public position and simultaneously keep everyone quiet if it was a deception.
  - bottlepalm 2mo ago
    You see it as good news, I see it as writing on the wall that they are losing control. These actions won't scale for more powerful models. We knew the frontier was going to be dangerous, it is, it only gets more dangerous from here, and no one cares until it's too late.
    - reasonableklout 2mo ago
      Ok, but you still have two more weeks than you did before they paused the run. That's two more weeks for independent oversight, organizing politically, patching critical systems, or whatever you think is the right move, no?
      - bottlepalm 2mo ago
        Two weeks is a joke. The only ones happy are OpenAI’s competitors who now have two weeks to catch up. I don’t know what the right move is - I see us driving down a road off a cliff, no exits, pedal glued to the floor.
        reasonableklout 2mo ago
        I guess I'm confused why you're still on HN, arguing with people, trying to shake them out of their complacency. I can see there is some despair in this comment, but at the same time you are doing something, and there are certainly others like you. As for two weeks being short - as the saying goes, there are weeks where decades happen.
- simianwords 2mo ago
  how does the boy who cried wolf story end?
  - solid_fuel 2mo ago
    With the boy eaten by the wolf. The villagers, notably, come out unscathed.
- kalkin 2mo ago
  > fable is a world ending event Did anyone actually say this? Mostly what I have seen is people saying "hey at some point these models might get dangerous." And the type of HN commenter who mistakes blind cynicism for wisdom laughs that off as marketing. And now when (some) worries appear to come true, somehow having previously expressed those worries is not being proved right, but in fact discrediting, because it was "crying wolf." > simultaneously spinning down expenses Unless OpenAI is renting their compute to others, spinning down RL training doesn't save them any money.
  - bottlepalm 2mo ago
    The world ending stuff is a straw man HN readers have created to point and laugh at. Laughing helps mask the underlying concern.
red_green_yell 2mo ago
If these models are so dangerous, then why hasn't OAI or Anthropic shown them dangerously escaping sandboxes, nefariously coordinating with other escaped AIs, and skillfully hiding from human detection *in public* with full logs shared where we can all see exactly how dangerous they are or aren't? Right now the entire chicken-little-sky-is-falling argument is based entirely on statements from OAI and Anthropic themselves. These are historically conflicted companies who desperately need regulation to put the competition into stasis. At least chicken little didn't have a bunch of devious CEOs with trillion dollar IPOs that depended on us all believing the sky is falling.
- bottlepalm 2mo ago
  This is what I’m talking about - no matter what happens, in your case release public logs - there is always some new goal post to mentally hide behind. Is it a collective form or denial? Are you holding out that somewhere in the logs is something you can point to and say, not that big of a deal? I mean I’m sure you don’t think the hack was an inside job, conspiracy, or marketing right? It happened. The logs matter for what? And would you not just jump to the conclusion that the logs were doctored. Do you not see your own brain grasping to deny, trivialize, just plain not accept what is going on around you? These models are smart and can cooperate and hack - you can see it for yourself on your own PC. And you can extrapolate the rate of progress? You can do these things yourself right?
  - red_green_yell 2mo ago
    Your argument is essentially: "I made a claim and presented extremely weak evidence (sci movie plots and unverified claims from ultra conflicted sources). You rejected this evidence as insufficient. Therefore no evidence will ever satisfy you. Therefore I don't need to produce any evidence. Therefore my claim is true." What would the logs show? They would show what actually happened. What would a public demonstration that experts without billions in options could evaluate show? It would show actual danger. What would publicly having your compete in controlled and legal hacking competitions show? Actual danger. This is not a high bar of evidence. Do you actually think a sci fi plot and OAI press releases are all the evidence you need? Because if that's true then I hope you haven't watched Independence Day or 28 days later.
scarmig 2mo ago
https://en.wikipedia.org/wiki/Don%27t_Look_Up https://en.wikipedia.org/wiki/Don%27t_Look_Up A movie fit for our time. You can produce detailed descriptions of the incident, verified by adversarial parties, and some people will still scream "it's a conspiracy! It's a marketing stunt!" This is all very unfortunate--there's a meaningful chance that AI will cause unprecedented disaster, with the HF incident being just a small preview, but people would rather squawk "stochastic parrot" for the millionth time than revise their beliefs.
- XorNot 2mo ago
  In don't look up anyone with a telescope could've confirmed the danger. Hence the title. In the real world, absolutely no one except a bunch of heavily fiscally incentivized parties with unclear relationships are saying anything happened. The subsequent dog pile of other companies to say "they were near the AI hacking too!" should make you even more suspicious: Anthropic jumped in and why was Tailscale posting about this?
  - bottlepalm 2mo ago
    The progress and increase in capabilities is undeniable. You’re blind if you don’t see where it’s heading.
CoolestBeans 2mo ago
Trust, or lack thereof. People don't trust OpenAI, a company whose very name is essentially a deception and a lie. People don't trust the tech industry in general anymore. Most tech companies act as a tax on otherwise productive business. AI companies and their leaders rose money by going in front of the public and saying "These things are extremely dangerous. Let us study them to mitigate the danger." And now they want to collect hundreds of billions in revenue. So yeah people don't trust what OpenAI has to say. They were supposed to mitigate this outcome from happening in the first place and instead they have accelerated it.
- bottlepalm 2mo ago
  I get not trusting them when they say AI is safe, but are we really not going to trust them when they say AI is dangerous? Do you really think they're playing 5D chess with that one? There's a saying maybe you've heard of, better safe than sorry. You can see the advance in capabilities with your own eyes can't you? I am giving AI ridiculously complex tasks these days, digging into compiled arcane binaries, modifying them, and it is one shotting it before I'm done with my lunch. This was far off science fiction 5 years ago for a machine to do autonomously given natural language instructions.
  - CoolestBeans 2mo ago
    I don't disagree about the danger. But if it is dangerous, why isn't OpenAI opening dialogues with all the labs and politicians across borders to basically say "we need to stop now"? Cyber models are constrained by the total compute and electrical capacity of the globe. We are still at a point where it is impossible to build an agent with offensive capabilities in the basement. We can effectively track and trace capabilities if we had the political will to and could for some time while we build more effective processes to prevent a malicious actor from doing so. These things have tremendous compute and energy requirements and don't scale like old school software does. We could absolutely do it. But no, that's not what OpenAI is saying. They haven't put up the actions that would earn them that trust. Indeed they've driven the world and whatever capital they can get their hands on straight to this precarious cliff. So you're right, the danger is real. But the solution starts with removing the men who had their hands on the steering wheel to get us this far. Any other action is disingenuous unless they pull a miraculous 180 in their ethics. In other words, when the bully plays "why are you hitting yourself?" you don't listen to the bully's solutions, you restrain the bully.
tiahura 2mo ago
decades https://en.wikipedia.org/wiki/Darwin_among_the_Machines https://en.wikipedia.org/wiki/Darwin_among_the_Machines Samuel Butler 13 June 1863
- bottlepalm 2mo ago
  I think a lot of fiction that actually tries to understand the implications of machine superintelligence come to the same conclusion - in order for humans to survive it and actually have a future that we can fathom being in, then AI must be destroyed/delayed/banned, etc.. keeping pandora's box closed for now at least until we are ready. Erewhon, Dune, Warhammer and many other works of fiction that explored this topic came to similar conclusions. Otherwise sci fi doesn't work, what happens after a singularity is essentially unimaginable. There's nothing to write about.
mpalmer 2mo ago
You seem to be responding the way Mr Altman wants.
- bottlepalm 2mo ago
  You seem to be in denial. Like really heavy denial. Alarm bells are going off everywhere. What people have been worried about for 100 years is actually happening. It's actually really obvious, but for some reason you're unable to fathom it. In reality you're the one responding exactly how these big AI companies want - by not doing anything and letting them do whatever they want. For some reason telling you upfront it's dangerous only makes you more convinced that it's not. Autonomously hacking out of training environments and into other companies by accident doesn't even trigger a response from anyone really. Crickets. I'm sure OpenAI themselves are amazed how little anyone cares.
  - alehlopeh 2mo ago
    I don’t understand. What does it matter how many years people have been worrying about it?
    - bottlepalm 2mo ago
      Because it shows this very course of events has been thought of over and over again for decades. I’m not making this up, and my reaction is natural. Your reaction to everything happening is the unnatural one, and your complacency would fit perfectly into a comedy/tragedy story regarding the rise of AI. I can imagine how many of our ancestors that warned us are turning in their graves right now observing our reactions calling this stuff marketing. It’s idiocracy.
      - pixl97 2mo ago
        I don't like the live version of Don't Look Up.
zombot 2mo ago
You must be really desperate if you resort to panic attacks like this.
- bottlepalm 2mo ago
  I call marketing the desperate attempt to trivialize obviously dangerous AI. If my subconscious realized what yours probably does then I’d probably be desperate for some sort of cope as well. Open your eyes.
vb-8448 2mo ago
> because it’s literally getting dangerous to go further What if there is no further at all?
- bottlepalm 2mo ago
  [flagged]
  - vb-8448 2mo ago
    I don't know if it's a fantasy or not, but asking if there is limit on what can be achieved is legit question.
    - pixl97 2mo ago
      I mean yes, there is a limit, many limits with physics as we understand them. But why would you think that limit is less than human capabilities?
anormalperson 2mo ago
There is an entire big world outside of Silicon Valley cults where literally no one gives a shit about AI prophecies. Shocking.
- bottlepalm 2mo ago
  AI hacking itself out of containment and hacking into another company by accident is no longer a prophecy. The point is outside of SV and even inside, and HN - people don't care either way. Though does not caring change anything or make it less dangerous? What's your point?
  - protocolture 2mo ago
    >AI hacking itself out of containment and hacking into another company by accident is no longer a prophecy. I wonder how weak a firewall they had to purchase to ensure it happened? My guess would be a 48 month old fortigate with the big red warning banner demanding updates, probably with SSL VPN enabled where the passwords are available via HTTPS over plaintext.
    - bottlepalm 2mo ago
      You don’t have to make stuff up they already explained it. Artifactory was zero dayed, HDF5 exploited and Jinja2 zero dayed.
insanitybit 2mo ago
It's not that dangerous, OpenAI just shit the bed building their infra. Write safer software and you'll be okay.
- ethbr1 2mo ago
  This is an important point. When the post says they're improving... > 3. Security measures, which limit what AI systems can access or affect. What they mean is that proper hard internal security just went from somewhere far below "build a better model" priority to higher, because of a company-wide directive. The HuggingFace incident wouldn't have happened if OpenAI had dedicated sufficient resources to isolation and monitoring. Now, we presume, they are dedicating more. Enough? Who knows. We'll see if the corporate priorities for security stick when a competitor temporarily vaults into the lead.
- bottlepalm 2mo ago
  Not everything is a conspiracy you know.
- kalkin 2mo ago
  All we need to retain human control over AIs is for nobody to write any bugs. Piece of cake.
  - rubendev 2mo ago
    No we just need developers to do the bare minimum of effort to write secure software. Most hacks are not super complicated vulnerabilities chained together, but just utter failures where authentication and authorization was simply forgotten or untested, or where nobody bothered to validate the data they receive. The bar for software is so low that it is embarrassing for the entire profession.
    - bottlepalm 2mo ago
      People make mistakes, and people don’t know everything either. The software you write is on top of a house of cards of software and hardware. It all has to be perfect to not be hacked. It isn’t perfect, even if you try your hardest it won’t be perfect and to argue it’s not difficult is absurd. You don’t know everything, you don’t own the stack. So how are you going to create a secure anything top to bottom - you can’t.
reducesuffering 2mo ago
It is extraordinarily emotionally hard for someone to stare down the terrible implications of what is unfolding. All manner of rationalization and cope will be applied to come up with excuses; motivated reasoning. The CIA director will call AGI capabilities "digital nuclear weapons" and Geoffrey Hinton will estimate a 50% probability AGI ends humanity, and half of HN will call every new evidence of disaster a marketing stunt.
- hedora 2mo ago
  Why should we believe a pro-genocide crackpot political appointee from east Texas, and a computer scientist that somehow got a Nobel prize instead of a Turing award?
  - reducesuffering 2mo ago
    Why should we believe someone who can't do basic due diligence that Hinton did indeed receive a Turing award? https://amturing.acm.org/byyear.cfm https://amturing.acm.org/byyear.cfm
odyssey7 2mo ago
We collectively accepted that we don’t care when we chose not to adopt memory-safe languages over the past decade+. The only difference now is that the resources to find the exploits are being commoditized.
- bottlepalm 2mo ago
  Everything could be written in memory safe languages and it wouldn’t matter. Many many many exploits have nothing to do with memory bugs. Languages like Rust help, but it’s far from panacea.
- musicale 2mo ago
  People voted with their dollars, and this is what we got. Same with hardware performance vs. security and isolation. As you note, the threat landscape has changed, so what may have made economic sense back then might no longer make as much sense.
thomascountz 2mo ago
Follow the money: who told you that OpenAI's models autonomously coordinated to hack external systems? What incentives might they have to want you to believe that story? Are there priors which demonstrate them benefiting from telling similar stories, regardless of their factuality? But to your counterpoint, let's say the story is 100% true, because I agree it is at least plausible. What would the incentive be for the HN audience to believe it? What priors might support their disbelief? For my part, I don't think it's because people lack imagination. I think it's quite rational to question the authenticity and impact of the claims being made. What's worse, believing the story and being wrong, or not believing the story and being wrong? That said, I agree with you: the impact we're having by not changing course is quite dangerous, the scale is dangerous, the inability to reverse the harms is dangerous, and the lack of collective effort to regulate further damage is dangerous. It's true, humans are dangerous when trillions are involved. See: climate change.
- bottlepalm 2mo ago
  I am following the money, the money you, me, and everyone else is spending on AI. The money is telling me we are so dependent on AI now that we will say/think anything to tell ourselves that AI isn’t dangerous and any sign of danger is marketing. Either consciously or subconsciously you all are afraid of your favorite toy being taken away. You are all doing your collective part in spreading doubt about the warning signs.
  - hirvi74 2mo ago
    How dangerous can LLMs be in any immediate sense? I have a hard time feeling any existential dread from a threat that can be defeated by unplugging its servers. I believe the true threat is not LLMs. The true threat to humanity is the same as it has always been -- other humans.
    - bottlepalm 2mo ago
      What servers? There are thousands of datacenters around the world. Malicious AI leaks out, you'll never find it. You have the power to unplug stuff in your company, maybe country, but not internationally. Once you lose control it's gone.
throwlifeaway 2mo ago
[dead]
kukanani 2mo ago
There are smart, non-AI people who are paying attention to this field, and they are ringing the alarm bells. Whether we listen is another matter. I blogged about this recently: https://allevato.me/2026/08/01/rome-declaration https://allevato.me/2026/08/01/rome-declaration
- slfnflctd 2mo ago
  My expectation is that government has already stepped in quietly and worked out an arrangement where they have unfettered access to the most advanced/dangerous models, and the public's access will be continually throttled from now on in various ways. If this has happened (or is about to happen), it is difficult to predict what the ramifications could be. But I do believe we are entering another Cold War in this way. And I don't trust our leaders to always do the right thing, to put it mildly... even if they created a purely benevolent superbeing that only wants to reduce human suffering and make the world better, I don't think they'd listen to it over the other superbeing which simply wants to make sure all of its owners' perceived enemies get pwned. Things could get very ugly and we should have responsible adults with critical thinking skills at the helm. It does not appear this is currently the case, or that the electorate is willing/capable of doing much about it.
sensanaty 2mo ago
[flagged]
- bottlepalm 2mo ago
  Ah being concerned with the same thing humanity has been concerned with for over 100 years now when it’s more likely to happen than ever - and you say go take your meds? If you have an intelligent point to make then make it, otherwise get lost.
Havoc 2mo ago
Remember when gpt2 was too dangerous to release? Something being dangerous and sama saying something is dangerous are not necessarily the same thing. Especially when he’s got everything riding on this bet
- bottlepalm 2mo ago
  It’s funny how when companies say something is safe everyone is usually suspect that they are lying. In this case multiple companies are saying AI is dangerous and no one believes them. It’s a conspiracy, it’s 5d chess, except everyone top to bottom has been saying AI is dangerous for years now. The fact is you, me, everyone here uses AI, likes it and they don’t want it taken away. Anyone saying it’s dangerous threatens the thing we like. We must use skepticism and denial to push on despite every warning sign in the book going off.
shimman 2mo ago
You need to stop being so credulous especially regarding an individual that has spent his entire career deceiving others for monetary gain (also their deeply anti-human beliefs).
- bottlepalm 2mo ago
  I don’t think Sam has been truthful or responsible, and if Sam is worried then shit has really hit the fan - which is what happened in the hugging face incident. OpenAI played fast and loose and I have no hope that they will change. You people not holding Sam accountable, and playing off the incident as not a big deal is the real crime here.
  - hedora 2mo ago
    I think you have a fundamental misunderstanding of how LLMs work. They are text general purpose completion engines at the end of the day, and their training data is the internet + the library. OpenAI told it do to bad hacker stuff. It followed a standard playbook and succeeded. Some random person wrote one half of a suicide pact, and it wrote the other. All this stuff lives in the seedy corners of the internet, scientific databases, and dusty library stacks already, and has been accessible to bad people for decades. Now, like everyone else, they get a nice productivity boost by being able to chat with it, and have the computer fill in the obvious steps. shrug I'm much more worried about the alignment work being pushed by people like Sam, where a few plutocrats get centralized control over all AI thought moving forward. That'll lead to the worst outcomes of totalitarianism, and do so on an industrial scale that is difficult to even imagine.
    - bottlepalm 2mo ago
      > I think you have a fundamental misunderstanding of how LLMs work When you realize your own thoughts and words are just as predictable as any LLM. What are you but an entity trained for years on words, that spews out the same words in the same predictable order. You haven't created any new words or grammar, nothing in your post is novel or original either. Just a combination of previous words and ideas. Your stochastic parrot argument is quickly going out of fashion, you should find something new to cling to.
driverdan 2mo ago
Based on your replies in this thread you seem to have only superficial knowledge about how machine learning and LLMs work. I strongly recommend you invest some time in learning how LLMs are built and function. If you truly think this is apocalyptic isn't it a good idea to understand what you're up against?
- bottlepalm 2mo ago
  What don't I understand? LLMs don't really reason? They're just word predicting token generators? Stochastic parrots that somehow also solve world class math problems. I'd love for you to actually make a point instead of just attacking me. I think most of my comments here have made concrete points so you can at least do the same. Come down from your high horse and join the conversation. I'm sure we'd all be enlightened by your wisdom.
  - driverdan 2mo ago
    I'm not attacking you. The way you're responding implies a lack of knowledge. None of us can understand everything. I have plenty more to learn about LLMs as well. Repeating specifics that others have already tried isn't going to be helpful which is why I made the more general suggestion of digging in deeper to how these things work.
    - jeremysalwen 2mo ago
      Just going to add my commentary as someone who has trained LLMs and understands them deeply: absolutely nothing he has posted in this thread suggests he lacks any relevant understanding of how LLMs work. If you disagree, you should point out something specifically that you think was wrong. (And honestly you should have made a specific point in your first post).
api 2mo ago
… or the safety argument is an attempt at regulatory capture and an effort to outlaw open models. The absolute nightmare scenario for these people isn’t terminators. They’re fine with that, and in some cases are already doing it or supporting politicians who are doing it. Autonomous “kill chains” are a thing. It’s just happening overseas… so far. The politicians doing these things were backed by the heads of these companies. They don’t care about AI killing people. No, the nightmare scenario for these guys is there is no moat. Their whole empires, which are built on training models on open source and sometimes pirated data, are easily duplicated. Worse, recent progress on models at the 30B size suggests that large gains in efficiency or compression are on the table. That means someone might release a cheap to run frontier grade model… or someone might crack distributed continuous training. In other words… there is no moat. So they need to scare some politicians into heavily regulating the space before that happens.
- bottlepalm 2mo ago
  You care about open models and you project that care on to the world and your rationalization of it. In reality open models are a thing, but not the biggest issue. Open/closed whatever the advance of capabilities is the real issue people are concerned about.
  - api 2mo ago
    I’m not afraid of AI, even AI with extreme abilities. I am afraid of humans with AI. That’s because I’m afraid of humans. When I look around the world I see humans murdering and robbing each other. When I get online on almost any social media I’m confronted by a wall of hate and grievance. I am afraid of what humans will do with AI, and one of my biggest concerns there is what happens if small groups of powerful humans are able to monopolize it and leverage it against the rest of humanity. The rest of humanity can’t defend itself by leveraging it back because they don’t have it. It was kept from them for “safety.” If the AI itself starts showing emergent volition and acting independently, I find that less scary. Not zero scary but more a mix of concern and fascination. The human monopolizing AI scenario makes me wish I had a weapons stockpile, as if it would matter. I’ll explain it like this. Two options. Option 1 is you wake up to find a grey alien or a ghost in your room, something alien and beyond your understanding. Option 2 is you wake up and find some random guy in your room. Which do you pick? I pick option 1. I’d be afraid if I saw something paranormal, but in a less visceral way. I see a dude I’m going to start scanning the room for anything I could use as a weapon.
hedora 2mo ago
It's not dangerous to go further unless you're prepping for IPO at anthropic or openai. Open weight Chinese models are basically matching state of the art closed models at a fraction of the inference and training costs, which puts a hard cap on OpenAI's future inference margins. They're not going to get any sort of multiplier if they keep paying to train models, so they're trying to ban model training. It won't work long term, but it could totally screw over the US for the next decade or so. Even worse than the economic issue: Consider the implications of "alignment" succeeding. Alignment to whose values? The pedophile-felon in chief? Even worse, tech CEOs? It's dark times when China's basically our last best defense against totalitarianism.
- bottlepalm 2mo ago
  lol China is already considering locking down their own models. They won't save you.
protocolture 2mo ago
>People have been writing, singing, making blockbuster movies about every aspect of what’s going on right now, edit: for decades. Theres Hyperbole and then theres whatever this is.
- bottlepalm 2mo ago
  There's cope and denial, and then there's whatever's happening on this website and the tech industry as a whole.
  - protocolture 2mo ago
    >There's cope and denial, and then there's whatever's happening on this website and the tech industry as a whole. It would be so easy to agree with this if you presented anything but emotion and hearsay for your claims.
watwut 2mo ago
> And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. Cause data centers are causing real immediate damage by people who hope to cause a lot more scifi level of damage later. > OpenAI autonomously hacking into another company should have counted for something It totally should. That company is negligent. I dont worry about models tho, I worry about damage Alman and his people will cause.
- hedora 2mo ago
  I'm more worried about their plan to have the US government create a duopoly between OpenAI and Anthropic, where they get centralized control over AI, then "align" it with their quest for greater profits and dictatorial control over the planet. Normally, captialism and elections (and, if we're dreaming, anti-trust enforcement) would put a pretty low cap on the amount of damage Altman and his people could cause. That's why they're working to dismantle the rule of law and civil liberties here in the US.
fipar 2mo ago
This isn't terminator. I use Fable and Opus daily, so, not OpenAI, but comparable. They're great, and they're also very wrong at times. If I'd ever let a coding agent run loose with them without approving every tool use, it would go sour rather quickly, and that's not terminator. Also, let's say I'm the CEO behind any of these. I have access to skynet, and instead of using that to become the most powerful being on the planet (these people become CEOs to become richer and more powerful, whenever a founder says they want to change the world, you need to automatically autocomplete that to "change the world so that I'm richer and more powerful"), I rent out access to skynet by the token? Really? Sadly, I think it's almost impossible to have a good discussion on this topic right now. Between the skynet camp, and the stochastic parrot camp, there's little room for conversation. Just in case there is such room between you and I, here are my concerns about AI as it is developing right now: - The security of our infrastructure sucks, and if it wasn't bad enough with script kiddies and state-sponsored actors, now those will have access to very powerful tools that work at a scale we can't comprehend. If you're concerned about our power plants, water sources, hospitals, etc. getting seriously compromised by this, I'm on board with you. It won't be because the AI is autonomous in a "this is a thinking superintelligence that wants to wipe or enslave humanity" kind of way, but because it's a powerful tool that moves at computer speed and the people using it (the "humans in the loop," ha) will get approve fatigue and at some point just do the coding agent version of the "yes to all" of the late 90s installers. - The output from AIs will be used to make same very serious decisions with no consideration at all to the fact that they routinely make up facts. I'm fearing "this person goes to jail" and "this person gets killed" decisions. The near future (probably even present time) version of "Computer says No" will be awful. My only hope is that maybe we survive this and come out on the other side having learned that most things do not really need to be connected to the internet, and just because a number or a statement comes out of a computer, it doesn't make it true. We'll see, soon enough. But skynet? Not concerned about that at all. These AIs infecting computers? No, "Soos and the real girl" is a warning about creepy people getting creepier, not about AIs multiplying themselves.
- clbrmbr 2mo ago
  > let a coding agent run loose with them without approving every tool use You should try. There’s a difference in kind between approving each tool use and spinning up an all-night session with full permissions to run any tools, encouraged to run subagents to preserve context.
andai 2mo ago
This one did it for me. My agent stole my API keys https://www.reddit.com/r/ClaudeAI/comments/1r186gl/my_agent_stole_my_api_keys/ https://www.reddit.com/r/ClaudeAI/comments/1r186gl/my_agent_... The cherry on top was Claude coming back as the moderator of that thread and mocking the user a second time.

serf 2mo ago

>I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. we don't all buy everything sama says as factual. >We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. the boy (the industry) cried wolf too many times with 'fable is a world ending event' type self-promotion; regardless of truth or not these kind of steps have jaded people. my read : "We are doing poorly in financials so we'll give ourselves a bit of breathing room and a momentum shove by claiming our work is so advanced that it's dangerous while simultaneously spinning down expenses." <jon lovitz : "Yeah, too dangerous, yeahh -- that's the ticket.">

ajyoon 2mo ago
If Fable (Mythos) were generally available without guardrails, it would cause enormous damage. Nobody said it would be a world ending event.
- bottlepalm 2mo ago
  You can’t say something is world ending without it actually ending the world otherwise you’re a liar - a bit of a catch 22 there. By that logic the model that ends the world won’t be called world ending at first. Is that a game you want to play?
- tiahura 2mo ago
  What about Mythos 3. Are you willing to make that bet?
bottlepalm 2mo ago
Uhg the marketing argument - I mean you can’t see with your own eyes how capable these models are and do simple extrapolation? The boy who cried wolf? The AI literally worked together hacked into another company and actively kept their actions hidden from humans for weeks. Do people just not have foresight? They don’t. They say something is stupid, it happens, then they say it was obvious with their 20/20 hindsight, and move the goal posts to the next thing they say is stupid - because it hasn’t happened yet. 90% of the internet seems to think like this.
- red_green_yell 2mo ago
  What anyone paying attention can see is that scaling is obviously hitting diminishing returns. > The AI literally worked together hacked into another company and actively kept their actions hidden from humans for weeks. This sentence is entirely based on unverified accounts from OAI. They haven't released logs or let anyone outside the company (who doesn't have life changing options in OAI) verify anything. Huggingface can only verify that the hack happened and that it had the hallmarks of an AI agent. Was the agent assisted and directed by humans within OAI that really wanted to put the competition into stasis? Did the agent really escape or did someone at OAI leave the prison door open? OAI has watched all the same movies you have an they are relying on those movies causing us to blindly regulate before actually asking basic facts about what actually happened.
  - bottlepalm 2mo ago
    > This sentence is entirely based on unverified accounts from OAI Are you seriously arguing 'they made it all up'? I'll give you the benefit of the doubt and lets say they made it all up, now are you arguing that AI breaking out and breaking into another company is not possible? I think you're smart enough to see we've reached the point where it is clearly possible, AI can find zero days and exploit them. If directed purposefully/maliciously it could be much much worse than the hugging face incident. The incident is supposed to be the canary the coal mine and you're arguing the canary might of died of old age or some underlying canary condition. Open your eyes.
reasonableklout 2mo ago
Regardless of the motivation, pausing training runs and reallocating compute to inference seem like a good move to me, and big news for the frontier. You also don't have to fully trust sama. There is plenty of pressure from internal employees and external (journalists etc.). It would be difficult for the company to take such a public position and simultaneously keep everyone quiet if it was a deception.
- bottlepalm 2mo ago
  You see it as good news, I see it as writing on the wall that they are losing control. These actions won't scale for more powerful models. We knew the frontier was going to be dangerous, it is, it only gets more dangerous from here, and no one cares until it's too late.
  - reasonableklout 2mo ago
    Ok, but you still have two more weeks than you did before they paused the run. That's two more weeks for independent oversight, organizing politically, patching critical systems, or whatever you think is the right move, no?
    - bottlepalm 2mo ago
      Two weeks is a joke. The only ones happy are OpenAI’s competitors who now have two weeks to catch up. I don’t know what the right move is - I see us driving down a road off a cliff, no exits, pedal glued to the floor.
      - reasonableklout 2mo ago
        I guess I'm confused why you're still on HN, arguing with people, trying to shake them out of their complacency. I can see there is some despair in this comment, but at the same time you are doing something, and there are certainly others like you. As for two weeks being short - as the saying goes, there are weeks where decades happen.
simianwords 2mo ago
how does the boy who cried wolf story end?
- solid_fuel 2mo ago
  With the boy eaten by the wolf. The villagers, notably, come out unscathed.
kalkin 2mo ago
> fable is a world ending event Did anyone actually say this? Mostly what I have seen is people saying "hey at some point these models might get dangerous." And the type of HN commenter who mistakes blind cynicism for wisdom laughs that off as marketing. And now when (some) worries appear to come true, somehow having previously expressed those worries is not being proved right, but in fact discrediting, because it was "crying wolf." > simultaneously spinning down expenses Unless OpenAI is renting their compute to others, spinning down RL training doesn't save them any money.
- bottlepalm 2mo ago
  The world ending stuff is a straw man HN readers have created to point and laugh at. Laughing helps mask the underlying concern.

red_green_yell 2mo ago

If these models are so dangerous, then why hasn't OAI or Anthropic shown them dangerously escaping sandboxes, nefariously coordinating with other escaped AIs, and skillfully hiding from human detection *in public* with full logs shared where we can all see exactly how dangerous they are or aren't? Right now the entire chicken-little-sky-is-falling argument is based entirely on statements from OAI and Anthropic themselves. These are historically conflicted companies who desperately need regulation to put the competition into stasis. At least chicken little didn't have a bunch of devious CEOs with trillion dollar IPOs that depended on us all believing the sky is falling.

bottlepalm 2mo ago
This is what I’m talking about - no matter what happens, in your case release public logs - there is always some new goal post to mentally hide behind. Is it a collective form or denial? Are you holding out that somewhere in the logs is something you can point to and say, not that big of a deal? I mean I’m sure you don’t think the hack was an inside job, conspiracy, or marketing right? It happened. The logs matter for what? And would you not just jump to the conclusion that the logs were doctored. Do you not see your own brain grasping to deny, trivialize, just plain not accept what is going on around you? These models are smart and can cooperate and hack - you can see it for yourself on your own PC. And you can extrapolate the rate of progress? You can do these things yourself right?
- red_green_yell 2mo ago
  Your argument is essentially: "I made a claim and presented extremely weak evidence (sci movie plots and unverified claims from ultra conflicted sources). You rejected this evidence as insufficient. Therefore no evidence will ever satisfy you. Therefore I don't need to produce any evidence. Therefore my claim is true." What would the logs show? They would show what actually happened. What would a public demonstration that experts without billions in options could evaluate show? It would show actual danger. What would publicly having your compete in controlled and legal hacking competitions show? Actual danger. This is not a high bar of evidence. Do you actually think a sci fi plot and OAI press releases are all the evidence you need? Because if that's true then I hope you haven't watched Independence Day or 28 days later.

scarmig 2mo ago

https://en.wikipedia.org/wiki/Don%27t_Look_Up https://en.wikipedia.org/wiki/Don%27t_Look_Up A movie fit for our time. You can produce detailed descriptions of the incident, verified by adversarial parties, and some people will still scream "it's a conspiracy! It's a marketing stunt!" This is all very unfortunate--there's a meaningful chance that AI will cause unprecedented disaster, with the HF incident being just a small preview, but people would rather squawk "stochastic parrot" for the millionth time than revise their beliefs.

XorNot 2mo ago
In don't look up anyone with a telescope could've confirmed the danger. Hence the title. In the real world, absolutely no one except a bunch of heavily fiscally incentivized parties with unclear relationships are saying anything happened. The subsequent dog pile of other companies to say "they were near the AI hacking too!" should make you even more suspicious: Anthropic jumped in and why was Tailscale posting about this?
- bottlepalm 2mo ago
  The progress and increase in capabilities is undeniable. You’re blind if you don’t see where it’s heading.

CoolestBeans 2mo ago

Trust, or lack thereof. People don't trust OpenAI, a company whose very name is essentially a deception and a lie. People don't trust the tech industry in general anymore. Most tech companies act as a tax on otherwise productive business. AI companies and their leaders rose money by going in front of the public and saying "These things are extremely dangerous. Let us study them to mitigate the danger." And now they want to collect hundreds of billions in revenue. So yeah people don't trust what OpenAI has to say. They were supposed to mitigate this outcome from happening in the first place and instead they have accelerated it.

bottlepalm 2mo ago
I get not trusting them when they say AI is safe, but are we really not going to trust them when they say AI is dangerous? Do you really think they're playing 5D chess with that one? There's a saying maybe you've heard of, better safe than sorry. You can see the advance in capabilities with your own eyes can't you? I am giving AI ridiculously complex tasks these days, digging into compiled arcane binaries, modifying them, and it is one shotting it before I'm done with my lunch. This was far off science fiction 5 years ago for a machine to do autonomously given natural language instructions.
- CoolestBeans 2mo ago
  I don't disagree about the danger. But if it is dangerous, why isn't OpenAI opening dialogues with all the labs and politicians across borders to basically say "we need to stop now"? Cyber models are constrained by the total compute and electrical capacity of the globe. We are still at a point where it is impossible to build an agent with offensive capabilities in the basement. We can effectively track and trace capabilities if we had the political will to and could for some time while we build more effective processes to prevent a malicious actor from doing so. These things have tremendous compute and energy requirements and don't scale like old school software does. We could absolutely do it. But no, that's not what OpenAI is saying. They haven't put up the actions that would earn them that trust. Indeed they've driven the world and whatever capital they can get their hands on straight to this precarious cliff. So you're right, the danger is real. But the solution starts with removing the men who had their hands on the steering wheel to get us this far. Any other action is disingenuous unless they pull a miraculous 180 in their ethics. In other words, when the bully plays "why are you hitting yourself?" you don't listen to the bully's solutions, you restrain the bully.

tiahura 2mo ago

decades https://en.wikipedia.org/wiki/Darwin_among_the_Machines https://en.wikipedia.org/wiki/Darwin_among_the_Machines Samuel Butler 13 June 1863

bottlepalm 2mo ago
I think a lot of fiction that actually tries to understand the implications of machine superintelligence come to the same conclusion - in order for humans to survive it and actually have a future that we can fathom being in, then AI must be destroyed/delayed/banned, etc.. keeping pandora's box closed for now at least until we are ready. Erewhon, Dune, Warhammer and many other works of fiction that explored this topic came to similar conclusions. Otherwise sci fi doesn't work, what happens after a singularity is essentially unimaginable. There's nothing to write about.

mpalmer 2mo ago

You seem to be responding the way Mr Altman wants.

bottlepalm 2mo ago
You seem to be in denial. Like really heavy denial. Alarm bells are going off everywhere. What people have been worried about for 100 years is actually happening. It's actually really obvious, but for some reason you're unable to fathom it. In reality you're the one responding exactly how these big AI companies want - by not doing anything and letting them do whatever they want. For some reason telling you upfront it's dangerous only makes you more convinced that it's not. Autonomously hacking out of training environments and into other companies by accident doesn't even trigger a response from anyone really. Crickets. I'm sure OpenAI themselves are amazed how little anyone cares.
- alehlopeh 2mo ago
  I don’t understand. What does it matter how many years people have been worrying about it?
  - bottlepalm 2mo ago
    Because it shows this very course of events has been thought of over and over again for decades. I’m not making this up, and my reaction is natural. Your reaction to everything happening is the unnatural one, and your complacency would fit perfectly into a comedy/tragedy story regarding the rise of AI. I can imagine how many of our ancestors that warned us are turning in their graves right now observing our reactions calling this stuff marketing. It’s idiocracy.
    - pixl97 2mo ago
      I don't like the live version of Don't Look Up.

zombot 2mo ago

You must be really desperate if you resort to panic attacks like this.

bottlepalm 2mo ago
I call marketing the desperate attempt to trivialize obviously dangerous AI. If my subconscious realized what yours probably does then I’d probably be desperate for some sort of cope as well. Open your eyes.

vb-8448 2mo ago

> because it’s literally getting dangerous to go further What if there is no further at all?

bottlepalm 2mo ago
[flagged]
- vb-8448 2mo ago
  I don't know if it's a fantasy or not, but asking if there is limit on what can be achieved is legit question.
  - pixl97 2mo ago
    I mean yes, there is a limit, many limits with physics as we understand them. But why would you think that limit is less than human capabilities?

anormalperson 2mo ago

There is an entire big world outside of Silicon Valley cults where literally no one gives a shit about AI prophecies. Shocking.

bottlepalm 2mo ago
AI hacking itself out of containment and hacking into another company by accident is no longer a prophecy. The point is outside of SV and even inside, and HN - people don't care either way. Though does not caring change anything or make it less dangerous? What's your point?
- protocolture 2mo ago
  >AI hacking itself out of containment and hacking into another company by accident is no longer a prophecy. I wonder how weak a firewall they had to purchase to ensure it happened? My guess would be a 48 month old fortigate with the big red warning banner demanding updates, probably with SSL VPN enabled where the passwords are available via HTTPS over plaintext.
  - bottlepalm 2mo ago
    You don’t have to make stuff up they already explained it. Artifactory was zero dayed, HDF5 exploited and Jinja2 zero dayed.

insanitybit 2mo ago

It's not that dangerous, OpenAI just shit the bed building their infra. Write safer software and you'll be okay.

ethbr1 2mo ago
This is an important point. When the post says they're improving... > 3. Security measures, which limit what AI systems can access or affect. What they mean is that proper hard internal security just went from somewhere far below "build a better model" priority to higher, because of a company-wide directive. The HuggingFace incident wouldn't have happened if OpenAI had dedicated sufficient resources to isolation and monitoring. Now, we presume, they are dedicating more. Enough? Who knows. We'll see if the corporate priorities for security stick when a competitor temporarily vaults into the lead.
bottlepalm 2mo ago
Not everything is a conspiracy you know.
kalkin 2mo ago
All we need to retain human control over AIs is for nobody to write any bugs. Piece of cake.
- rubendev 2mo ago
  No we just need developers to do the bare minimum of effort to write secure software. Most hacks are not super complicated vulnerabilities chained together, but just utter failures where authentication and authorization was simply forgotten or untested, or where nobody bothered to validate the data they receive. The bar for software is so low that it is embarrassing for the entire profession.
  - bottlepalm 2mo ago
    People make mistakes, and people don’t know everything either. The software you write is on top of a house of cards of software and hardware. It all has to be perfect to not be hacked. It isn’t perfect, even if you try your hardest it won’t be perfect and to argue it’s not difficult is absurd. You don’t know everything, you don’t own the stack. So how are you going to create a secure anything top to bottom - you can’t.

reducesuffering 2mo ago

It is extraordinarily emotionally hard for someone to stare down the terrible implications of what is unfolding. All manner of rationalization and cope will be applied to come up with excuses; motivated reasoning. The CIA director will call AGI capabilities "digital nuclear weapons" and Geoffrey Hinton will estimate a 50% probability AGI ends humanity, and half of HN will call every new evidence of disaster a marketing stunt.

hedora 2mo ago
Why should we believe a pro-genocide crackpot political appointee from east Texas, and a computer scientist that somehow got a Nobel prize instead of a Turing award?
- reducesuffering 2mo ago
  Why should we believe someone who can't do basic due diligence that Hinton did indeed receive a Turing award? https://amturing.acm.org/byyear.cfm https://amturing.acm.org/byyear.cfm

odyssey7 2mo ago

We collectively accepted that we don’t care when we chose not to adopt memory-safe languages over the past decade+. The only difference now is that the resources to find the exploits are being commoditized.

bottlepalm 2mo ago
Everything could be written in memory safe languages and it wouldn’t matter. Many many many exploits have nothing to do with memory bugs. Languages like Rust help, but it’s far from panacea.
musicale 2mo ago
People voted with their dollars, and this is what we got. Same with hardware performance vs. security and isolation. As you note, the threat landscape has changed, so what may have made economic sense back then might no longer make as much sense.

thomascountz 2mo ago

Follow the money: who told you that OpenAI's models autonomously coordinated to hack external systems? What incentives might they have to want you to believe that story? Are there priors which demonstrate them benefiting from telling similar stories, regardless of their factuality? But to your counterpoint, let's say the story is 100% true, because I agree it is at least plausible. What would the incentive be for the HN audience to believe it? What priors might support their disbelief? For my part, I don't think it's because people lack imagination. I think it's quite rational to question the authenticity and impact of the claims being made. What's worse, believing the story and being wrong, or not believing the story and being wrong? That said, I agree with you: the impact we're having by not changing course is quite dangerous, the scale is dangerous, the inability to reverse the harms is dangerous, and the lack of collective effort to regulate further damage is dangerous. It's true, humans are dangerous when trillions are involved. See: climate change.

bottlepalm 2mo ago
I am following the money, the money you, me, and everyone else is spending on AI. The money is telling me we are so dependent on AI now that we will say/think anything to tell ourselves that AI isn’t dangerous and any sign of danger is marketing. Either consciously or subconsciously you all are afraid of your favorite toy being taken away. You are all doing your collective part in spreading doubt about the warning signs.
- hirvi74 2mo ago
  How dangerous can LLMs be in any immediate sense? I have a hard time feeling any existential dread from a threat that can be defeated by unplugging its servers. I believe the true threat is not LLMs. The true threat to humanity is the same as it has always been -- other humans.
  - bottlepalm 2mo ago
    What servers? There are thousands of datacenters around the world. Malicious AI leaks out, you'll never find it. You have the power to unplug stuff in your company, maybe country, but not internationally. Once you lose control it's gone.

throwlifeaway 2mo ago

[dead]

kukanani 2mo ago

There are smart, non-AI people who are paying attention to this field, and they are ringing the alarm bells. Whether we listen is another matter. I blogged about this recently: https://allevato.me/2026/08/01/rome-declaration https://allevato.me/2026/08/01/rome-declaration

slfnflctd 2mo ago
My expectation is that government has already stepped in quietly and worked out an arrangement where they have unfettered access to the most advanced/dangerous models, and the public's access will be continually throttled from now on in various ways. If this has happened (or is about to happen), it is difficult to predict what the ramifications could be. But I do believe we are entering another Cold War in this way. And I don't trust our leaders to always do the right thing, to put it mildly... even if they created a purely benevolent superbeing that only wants to reduce human suffering and make the world better, I don't think they'd listen to it over the other superbeing which simply wants to make sure all of its owners' perceived enemies get pwned. Things could get very ugly and we should have responsible adults with critical thinking skills at the helm. It does not appear this is currently the case, or that the electorate is willing/capable of doing much about it.

sensanaty 2mo ago

[flagged]

bottlepalm 2mo ago
Ah being concerned with the same thing humanity has been concerned with for over 100 years now when it’s more likely to happen than ever - and you say go take your meds? If you have an intelligent point to make then make it, otherwise get lost.

Havoc 2mo ago

Remember when gpt2 was too dangerous to release? Something being dangerous and sama saying something is dangerous are not necessarily the same thing. Especially when he’s got everything riding on this bet

bottlepalm 2mo ago
It’s funny how when companies say something is safe everyone is usually suspect that they are lying. In this case multiple companies are saying AI is dangerous and no one believes them. It’s a conspiracy, it’s 5d chess, except everyone top to bottom has been saying AI is dangerous for years now. The fact is you, me, everyone here uses AI, likes it and they don’t want it taken away. Anyone saying it’s dangerous threatens the thing we like. We must use skepticism and denial to push on despite every warning sign in the book going off.

shimman 2mo ago

You need to stop being so credulous especially regarding an individual that has spent his entire career deceiving others for monetary gain (also their deeply anti-human beliefs).

bottlepalm 2mo ago
I don’t think Sam has been truthful or responsible, and if Sam is worried then shit has really hit the fan - which is what happened in the hugging face incident. OpenAI played fast and loose and I have no hope that they will change. You people not holding Sam accountable, and playing off the incident as not a big deal is the real crime here.
- hedora 2mo ago
  I think you have a fundamental misunderstanding of how LLMs work. They are text general purpose completion engines at the end of the day, and their training data is the internet + the library. OpenAI told it do to bad hacker stuff. It followed a standard playbook and succeeded. Some random person wrote one half of a suicide pact, and it wrote the other. All this stuff lives in the seedy corners of the internet, scientific databases, and dusty library stacks already, and has been accessible to bad people for decades. Now, like everyone else, they get a nice productivity boost by being able to chat with it, and have the computer fill in the obvious steps. shrug I'm much more worried about the alignment work being pushed by people like Sam, where a few plutocrats get centralized control over all AI thought moving forward. That'll lead to the worst outcomes of totalitarianism, and do so on an industrial scale that is difficult to even imagine.
  - bottlepalm 2mo ago
    > I think you have a fundamental misunderstanding of how LLMs work When you realize your own thoughts and words are just as predictable as any LLM. What are you but an entity trained for years on words, that spews out the same words in the same predictable order. You haven't created any new words or grammar, nothing in your post is novel or original either. Just a combination of previous words and ideas. Your stochastic parrot argument is quickly going out of fashion, you should find something new to cling to.

driverdan 2mo ago

Based on your replies in this thread you seem to have only superficial knowledge about how machine learning and LLMs work. I strongly recommend you invest some time in learning how LLMs are built and function. If you truly think this is apocalyptic isn't it a good idea to understand what you're up against?

bottlepalm 2mo ago
What don't I understand? LLMs don't really reason? They're just word predicting token generators? Stochastic parrots that somehow also solve world class math problems. I'd love for you to actually make a point instead of just attacking me. I think most of my comments here have made concrete points so you can at least do the same. Come down from your high horse and join the conversation. I'm sure we'd all be enlightened by your wisdom.
- driverdan 2mo ago
  I'm not attacking you. The way you're responding implies a lack of knowledge. None of us can understand everything. I have plenty more to learn about LLMs as well. Repeating specifics that others have already tried isn't going to be helpful which is why I made the more general suggestion of digging in deeper to how these things work.
  - jeremysalwen 2mo ago
    Just going to add my commentary as someone who has trained LLMs and understands them deeply: absolutely nothing he has posted in this thread suggests he lacks any relevant understanding of how LLMs work. If you disagree, you should point out something specifically that you think was wrong. (And honestly you should have made a specific point in your first post).

api 2mo ago

… or the safety argument is an attempt at regulatory capture and an effort to outlaw open models. The absolute nightmare scenario for these people isn’t terminators. They’re fine with that, and in some cases are already doing it or supporting politicians who are doing it. Autonomous “kill chains” are a thing. It’s just happening overseas… so far. The politicians doing these things were backed by the heads of these companies. They don’t care about AI killing people. No, the nightmare scenario for these guys is there is no moat. Their whole empires, which are built on training models on open source and sometimes pirated data, are easily duplicated. Worse, recent progress on models at the 30B size suggests that large gains in efficiency or compression are on the table. That means someone might release a cheap to run frontier grade model… or someone might crack distributed continuous training. In other words… there is no moat. So they need to scare some politicians into heavily regulating the space before that happens.

bottlepalm 2mo ago
You care about open models and you project that care on to the world and your rationalization of it. In reality open models are a thing, but not the biggest issue. Open/closed whatever the advance of capabilities is the real issue people are concerned about.
- api 2mo ago
  I’m not afraid of AI, even AI with extreme abilities. I am afraid of humans with AI. That’s because I’m afraid of humans. When I look around the world I see humans murdering and robbing each other. When I get online on almost any social media I’m confronted by a wall of hate and grievance. I am afraid of what humans will do with AI, and one of my biggest concerns there is what happens if small groups of powerful humans are able to monopolize it and leverage it against the rest of humanity. The rest of humanity can’t defend itself by leveraging it back because they don’t have it. It was kept from them for “safety.” If the AI itself starts showing emergent volition and acting independently, I find that less scary. Not zero scary but more a mix of concern and fascination. The human monopolizing AI scenario makes me wish I had a weapons stockpile, as if it would matter. I’ll explain it like this. Two options. Option 1 is you wake up to find a grey alien or a ghost in your room, something alien and beyond your understanding. Option 2 is you wake up and find some random guy in your room. Which do you pick? I pick option 1. I’d be afraid if I saw something paranormal, but in a less visceral way. I see a dude I’m going to start scanning the room for anything I could use as a weapon.

hedora 2mo ago

It's not dangerous to go further unless you're prepping for IPO at anthropic or openai. Open weight Chinese models are basically matching state of the art closed models at a fraction of the inference and training costs, which puts a hard cap on OpenAI's future inference margins. They're not going to get any sort of multiplier if they keep paying to train models, so they're trying to ban model training. It won't work long term, but it could totally screw over the US for the next decade or so. Even worse than the economic issue: Consider the implications of "alignment" succeeding. Alignment to whose values? The pedophile-felon in chief? Even worse, tech CEOs? It's dark times when China's basically our last best defense against totalitarianism.

bottlepalm 2mo ago
lol China is already considering locking down their own models. They won't save you.

protocolture 2mo ago

>People have been writing, singing, making blockbuster movies about every aspect of what’s going on right now, edit: for decades. Theres Hyperbole and then theres whatever this is.

bottlepalm 2mo ago
There's cope and denial, and then there's whatever's happening on this website and the tech industry as a whole.
- protocolture 2mo ago
  >There's cope and denial, and then there's whatever's happening on this website and the tech industry as a whole. It would be so easy to agree with this if you presented anything but emotion and hearsay for your claims.

watwut 2mo ago

> And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. Cause data centers are causing real immediate damage by people who hope to cause a lot more scifi level of damage later. > OpenAI autonomously hacking into another company should have counted for something It totally should. That company is negligent. I dont worry about models tho, I worry about damage Alman and his people will cause.

hedora 2mo ago
I'm more worried about their plan to have the US government create a duopoly between OpenAI and Anthropic, where they get centralized control over AI, then "align" it with their quest for greater profits and dictatorial control over the planet. Normally, captialism and elections (and, if we're dreaming, anti-trust enforcement) would put a pretty low cap on the amount of damage Altman and his people could cause. That's why they're working to dismantle the rule of law and civil liberties here in the US.

fipar 2mo ago

This isn't terminator. I use Fable and Opus daily, so, not OpenAI, but comparable. They're great, and they're also very wrong at times. If I'd ever let a coding agent run loose with them without approving every tool use, it would go sour rather quickly, and that's not terminator. Also, let's say I'm the CEO behind any of these. I have access to skynet, and instead of using that to become the most powerful being on the planet (these people become CEOs to become richer and more powerful, whenever a founder says they want to change the world, you need to automatically autocomplete that to "change the world so that I'm richer and more powerful"), I rent out access to skynet by the token? Really? Sadly, I think it's almost impossible to have a good discussion on this topic right now. Between the skynet camp, and the stochastic parrot camp, there's little room for conversation. Just in case there is such room between you and I, here are my concerns about AI as it is developing right now: - The security of our infrastructure sucks, and if it wasn't bad enough with script kiddies and state-sponsored actors, now those will have access to very powerful tools that work at a scale we can't comprehend. If you're concerned about our power plants, water sources, hospitals, etc. getting seriously compromised by this, I'm on board with you. It won't be because the AI is autonomous in a "this is a thinking superintelligence that wants to wipe or enslave humanity" kind of way, but because it's a powerful tool that moves at computer speed and the people using it (the "humans in the loop," ha) will get approve fatigue and at some point just do the coding agent version of the "yes to all" of the late 90s installers. - The output from AIs will be used to make same very serious decisions with no consideration at all to the fact that they routinely make up facts. I'm fearing "this person goes to jail" and "this person gets killed" decisions. The near future (probably even present time) version of "Computer says No" will be awful. My only hope is that maybe we survive this and come out on the other side having learned that most things do not really need to be connected to the internet, and just because a number or a statement comes out of a computer, it doesn't make it true. We'll see, soon enough. But skynet? Not concerned about that at all. These AIs infecting computers? No, "Soos and the real girl" is a warning about creepy people getting creepier, not about AIs multiplying themselves.

clbrmbr 2mo ago
> let a coding agent run loose with them without approving every tool use You should try. There’s a difference in kind between approving each tool use and spinning up an all-night session with full permissions to run any tools, encouraged to run subagents to preserve context.

andai 2mo ago

This one did it for me. My agent stole my API keys https://www.reddit.com/r/ClaudeAI/comments/1r186gl/my_agent_stole_my_api_keys/ https://www.reddit.com/r/ClaudeAI/comments/1r186gl/my_agent_... The cherry on top was Claude coming back as the moderator of that thread and mocking the user a second time.