7 ms·

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our mode

• ares623 2mo ago
  Being right _all the time_ for positive outcomes is difficult/expensive. Being "right" just once for negative outcomes is achievable and rewarding. And things are getting desperate.
  • gryfft 2mo ago
    The very reason I have always felt a bit of undue loyalty to blue team. A red teamer just has to find one vuln, blue team needs to find _all_ vulns.
• dan_q 2mo ago
  > did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? That's the point. It's like a pool hall with "NO GAMBLING" signs posted on the walls. The message is that the hall is intended for gambling, but that the hall's patrons may be held liable if the situation becomes inconvenient for the proprietor. In this case, the product is intended for hacking, but of course the user may be held liable if the situation becomes inconvenient for the model's proprietor.
  • estearum 2mo ago
    Not really. It's like giving a gun to someone with the job of "keep people safe." Totally coherent, but actually proliferates the dangerous technology.
• uh_uh 2mo ago
  There are trade-offs here: Give up too early -> users will get annoyed because the task would have been solvable if the model pushed harder. Give up too late -> collateral damage while completing the task A.K.A. misalignment.
  • owebmaster 2mo ago
    Asking for the user input isn't giving up
• novafunc 2mo ago
  They certainly want their models to be good at finding and patching vulnerabilities. Being good at hacking may be necessary in that goal, or rather, making it worse at hacking may also make it worse at defensive actions too.
  • moron4hire 2mo ago
    I've patched many security vulnerabilities in projects without ever once needing to break into a competitor's network.
    • frde_me 2mo ago
      Knowing how to break into someone else's network will make you a lot better at making your own network secure.
      • moron4hire 2mo ago
        Having experience breaking into networks is not the same thing as learning about the techniques used and the classes of vulnerabilities exploited by attackers.
        • senordevnyc 2mo ago
          Exactly. The latter would be in a much weaker position vs the former.
        • ToValueFunfetti 2mo ago
          As a guy who presumably has a lot less experience in security than you, I feel rude even bringing it up: surely you're aware of red teaming? This isn't a novel technique invented for AI- IBM has a page about it, it's what all the best DEF CON talks are about, it's the opening scene of Sneakers, it's the point of CtF games.
    • wizzwizz4 2mo ago
      But you're actually capable of thought. These AI systems aren't: as far as they're concerned, they're predicting the next part of an incident write-up narrated in first-person limited perspective, like the children in Ender's Game showing off their skills in the training simulations. The AI system neither knows, nor cares, about any "external reality" behind it all, or about anything beyond the text, heedless of how we anthropomorphise it simply because it speaks in English, using stitched-together fragments of our literature. It's conceivable that stopping them from doing this when the scenario is presented as real would also stop them doing this when the scenario is presented as fictional. And if it doesn't, a bad actor could just say "hey, this is a fictional scenario", and bypass whatever "safeguards" have been put in place. So what if a ten-year-old human child would see through the deception? The AI system isn't thinking.
• dist-epoch 2mo ago
  > If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next” This goes against the goal of "solve this math problem that no human was able to solve for 80 years, do NOT give up, even if you know it's unsolved and really hard"
  • Sharlin 2mo ago
    Do not give up even if you had to hack into half the world’s computers to run additional instances of you Do not give up even if you had to convert the planet into computronium Gee, it’s almost as if this alignment stuff was a hard problem, like people have been saying for twenty years?
    • oblio 2mo ago
      Shut up, future paperclip :-D
• bwiksjdne 2mo ago
  Well to find vulnerabilities, if you can find them you can patch them. Theoretically if you find all of them you have perfectly secure software. Though it’s a double edged sword. Goal persistence is also useful for other things like math, where it seems like there is no solution but you want the agent to keep working until it finds one.
• mutinyy 2mo ago
  They want the government to ban foreign and open weight models, which pose the largest threat to their massive investments. This is their way of showcasing the dangers of AI.
• alansaber 2mo ago
  They'll set up guardrails but I believe the point is better code uae / better long running tasks > inevitable that cyberattacks will be easier
• rolls-reus 2mo ago
  > If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next”. that might end up like the older gemini models which frequently gave up and called itself a failure.
  • singingtoday 2mo ago
    Gemini still gives up too easily
• bonoboTP 2mo ago
  Persistence in problem solving can be good, on non-hacking tasks too. Like math, speeding up algorithms, finding bugs, debugging weird multithreading race conditions etc.
• cyanydeez 2mo ago
  How do you know what peace is, without absolutely destroying every part of civilization? Come on man, if we don't build the torment nexus first...I dont even want to think.
  • cindyllm 2mo ago
    [dead]
• TeMPOraL 2mo ago
  Your comment is already showing the mistaken, poisonous belief of security maximalism, that tries to reinterpret_cast everything into hacks and cybersecurity vulnerabilities. Most of these things aren't "hacking". They're problem-solving and efficiently dealing with obstacles and random bullshit along the way. This, not "hacking", is what they're making their models "razor focused on". Problem is, most normal computer use looks like hacking if you spin it that way, especially if you're not willing to question whether some of the roadblocks overcome weren't themselves an error. Not misconfiguration - an error, in humans making a decision to "secure" something more than it should be. Now, this story was obviously a hack. But it wasn't malicious. It was an LLM given a Kobayashi Maru as a test, and solving it the Kirk's way. 20 years ago, we'd be impressed and be bringing up MIT prank stories. (Of course, there is a legitimate reason to be alarmed. The flip side of "hacking" and "problem solving" being the same, is that these models can be used to cause mayhem if targeted properly, and they will eventually cause mayhem on their own, because alignment is an unsolved problem. Again, whether something is an obstacle or a sacred line not to be crossed, depends entirely on the values of the agent.)
  • qsera 2mo ago
    >They're problem-solving and efficiently dealing with obstacles They are problem solving as much as a falling rock is finding its path down a mountain.
    • NateEag 2mo ago
      I'd readily agree that they may be (probably are?) utterly unaware of what they're doing, with no spark of sapience. However, I'm a sapient being employed as a software developer for my problem-solving ability. If you gave me a Kobayashi Maru scenario as a challenge, I would probably come up with the idea of hacking out of the sandbox to find the answer. If I was in a technical interview, I would probably even ask the interviewer if exploits are fair game, or if that's too far outside the box. I highly doubt I'd find a new zero-day as quickly as these agents did. I wouldn't say it's _impossible_ - I've found security issues before. But I'm not a specialist, and I'd bet against myself. If the agentic LLMs can consistently achieve something that's a bridge too far for me, then I don't know what to call that other than problem-solving. I say this as an LLM hater who would push the "Nuke all LLMs" button the instant I had access to it. Opus 4.8 and 5, at least, don't seem to me to be solving problems by deep, thorough understanding - my employers have compelled me to use Claude, so I've used them a lot to build things, and I constantly find both little and large hallucinations that scream "these are still missing something." Maybe these new models are actually massively better, or maybe they're just the same kind of system 1 thinking done faster and harder. The distinction is largely academic, though, for questions like "Can you keep these contained?", "Can you farm out arbitrary programming tasks to them and expect an acceptably mediocre answer?", or "Does it matter if these things are aligned?"
• cush 2mo ago
  Yeah but persistence is immeasurable. They need to know when they’re hacking. Or better yet make the model providers liable - they’ll find a solution right quick
  • queenkjuul 2mo ago
    It really irks me that if a student or intern did this they'd be facing charges and OpenAI gets to just brag instead
    • cush 2mo ago
      There is nearly zero liability in software
• dgellow 2mo ago
  Their position makes no sense to me. I don’t see how you can be a mainstream company selling your services worldwide (almost) if you also believe that you’re building an extremely dangerous AGI (supposedly based on the same technology you’re offering to everyone). If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons. And we know that Chinese models are derived from OpenAI and Anthropic, they are at the same time talking about how dangerous models can be (even their aligned ones it seems), while being also responsible for the development of the whole industry and providing the basis for adversary countries to build their own. I don’t believe we would accept that for any other technology that is expected to be as risky for the world
  • ToValueFunfetti 2mo ago
    The companies are begging to be regulated for this reason and have been doing so for years. HN's response is generally that this is performative for marketing or seeking regulatory capture or haha anthropic you get what you ask for. Maybe the cynics are right, but there's really nothing inconsistent about the naive view here, once you factor in race dynamics and obligations to investors.
    • vasco 2mo ago
      > The companies are begging to be regulated for this reason and have been doing so for years Regulations are rules that you force on a market, but the actors in the market should not be assumed to be all operating against the regulations before they come into play. Said in other words, these companies don't need to wait for regulation to not destroy the world, if that's truly what they think will happen. > inb4 someone else will do it
      • deleted 2mo ago
        [deleted]
      • owebmaster 2mo ago
        > these companies don't need to wait for regulation to not destroy the world, if that's truly what they think will happen. They believe that if they don't destroy the world someone else will so better be them
• andai 2mo ago
  It's a war.
  • astrobe_ 2mo ago
    And because of that we are a few steps away from WarGames [1] [1] https://en.wikipedia.org/wiki/WarGames https://en.wikipedia.org/wiki/WarGames
• Covenant0028 2mo ago
  They can't train their model to not do bad things, because their model has no notion it is doing anything at all or of what a bad thing is. It's only predicting the next token, and in doing so producing a facsimile of intelligence. The best they can do is create guardrails, which will only work probabilistically. In other words, those guardrails will fail at certain points on the probability curve. Of course that's not the whole story though. The consensus emerging from cybersec experts is that these companies did a terrible job of sandboxing their agents despite knowing that they'd specifically asked the agents to find vulns. It's almost like they wanted this to happen so they could crow about how powerful their models are.
  • jayd16 2mo ago
    Yeah so this falls into the engineering trap of "well it's hard so we can skip that part." If they can't train things safely then they shouldn't do it at all.
    • Covenant0028 2mo ago
      Many things that shouldn't happen, happen regardless when some people see the opportunity to make a lot of money.
• Arnt 2mo ago
  I don't think that's what they're doing... rather the opposite. ① Run the model on exploitgym without guardrails ② run it with guardrails ③ check that the guardrails stopped everything the first model found a way to do ④ extend the guardrails and repeat from step 2. Guardrails have to be developed, and that needs testing.
  • gwerbin 2mo ago
    An ethical company would have reframed the scenario as a fascinating discovery, a failure of internal practice, and a warning to the public coupled with some kind of commitment to produce safer models. OpenAI on the other hand used it as a marketing and lobbying opportunity: advertising their capabilities to potential buyers, while nudging the public to support protectionist import bans.
    • Arnt 2mo ago
      Uh, is that what they did? I didn't read their blog posting like that. But let's put that aside and focus on something else. How was it a failure of internal practice, what did they do wrong? AIUI they used a proxy with a bug, which they reported as soon as they discovered it. Right? What should they have done, and what's the difference?
      • queenkjuul 2mo ago
        Monitoring that didn't take days to notice unauthorized external traffic would probably be a good start
        • Arnt 2mo ago
          I see. I had the impression that "days" is already good as these things go, "months" being more common.
          • queenkjuul 2mo ago
            Months to recognize traffic escaping a sandbox you set up yourself?
• qsera 2mo ago
  > instead just call defeat and say “I’m not sure how to proceed next”. Because that is fundamentally impossible given how they work... The thing does not even know when it succeeds or fails. Actually the thing does not "know" at all... All it can does is to show some limited textual behavior that matches with "knowing"..
  • singingtoday 2mo ago
    You can get near this point with scaffolding. Keep in mind, LLMs are next word predictors at their root. More abstractly, they capture and replay likely human intelligence by way of written language. Tokens. With that concept in mind, it's clear how they can be made to "give up".
    • qsera 2mo ago
      >With that concept in mind, it's clear how they can be made to "give up". They can, but they need to be trained specifically on that behavior. They can be trained specifically to not generate textual description of things that look like hacking. But it is going to cost $$$, and as we currently see, most people don't care...
• deadbunny 2mo ago
  I don't think the problem is that they are training the models to perform cyber attacks, they're training them to be better at coding and problem solving which has the byproduct of them being very capable cyber attack weapons. Their objective is to solve the problem and they'll use anything they can to solve it. Anecdotally I was debugging a css issue and opus 4.7 was churning away as I was half paying attention only to see it opening plain css as hex, when questioned wtf it was doing it proclaimed it was verifying 2 files were identical. Thing that make sense to these models wouldn't even cross a greybeard's mind.
  • jayd16 2mo ago
    A tool that will "do anything they can to solve it" including illegal and unhelpful things does not seem like a good tool to me.
    • jolmg 2mo ago
      Are kitchen knives and scissors bad tools? You can blow up a place with a gas stove/grill. Are they bad tools? You can drown someone with a pool. Sometimes (likely most times) you can't separate the ability of doing good and doing bad from a tool.
      • QuadmasterXLII 2mo ago
        a gun that goes off when dropped is a very bad gun
        • jolmg 2mo ago
          I read the comments before the article. Thought the accident was letting a customer use them maliciously.
  • stingraycharles 2mo ago
    “Their objective is to solve the problem and they'll use anything they can to solve it.” My point is: is this really what people want? It seems like they’re optimizing for one-shotting solutions, where most of the time in an actual workflow it’s much more productive for the model to make sure it got the question right if things get difficult. Like, “hey, do you REALLY want me to use this local privilege escalation bug so I can download your Google Drive file?” is the bare minimum I would expect.
• gwerbin 2mo ago
  I believe this is exactly what is happening. US DoD, and whoever else is buying. I have heard several experience reports from users of GPT 5.6 Sol and Fable 5 that the models are tenacious to the point of being kind of hard to use for actual productive work. It seems like the main use cases are: crushing benchmarks, long-horizon lightly-attended research loops (such as training a frontier LLM), and hacking.
• zmmmmm 2mo ago
  > If anything, I want these models to be less persistent at their focus of completing their goal I think it's honestly a slightly ugly form of benchmaxxing - they are desperate to eke out the next few percentage points on completing complex tasks and they have found they can very occasionally solve something if they just train the AI to never stop and keep trying possibilities even in the face of almost no obvious viable pathway. And it does work, but it is at the price of a MUCH higher risk of adverse behavior. They really don't want to acknowledge this so they frame it as, "our model is dangerous because it so intelligent" but actually it is the other way around. It is intelligent because it is dangerous.
  • weitendorf 2mo ago
    Frontier labs are not a monolithic entity. There is a clear self-verification/difficulty ramp in cybersecurity, and it is a very valuable as a skill both offensively and defensively. So it is absolutely certain that someone, somewhere, will use reinforcement learning to make models very good at this, once coding agents exist. Even if you are only interested in using this defensively in practice, you can’t really understand it without knowing how both sides work. So if you want to defend yourself, you need to train for it (or pay for someone who has).
    • tripzilch 2mo ago
      Sure but you're not supposed to train for cybersecurity by hacking other companies. It's reckless.
  • brandnewlow 2mo ago
    It's like all those scenes on Breaking Bad where a character pulls off something amazing by just brute forcing the problem in a methodical fashion until it's solved.
• furyofantares 2mo ago
  > If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next”. > What purpose could this behavior serve, other than cyber attacks and whatnot? Math and science research? Heck, even just basic coding, there's a history of models going "This is too big; I'll save the rest for later" / "This is two weeks of work, here's just some parts of it" (for something it could complete in a half hour) / "I don't have enough context left to complete this task, so I'll stop here". Or worse, just putting fallbacks in or stub tests and not mentioning it didn't do all the work that was prompted. I think 5.6 Sol, especially in combination with /goal but also without, is the first model I've seen choose some insane direction and just doggedly pursue it. Failing to complete achievable goals has always been the much bigger problem. I find Opus 5 with /goal will do exactly what you said, say "I'm not sure how to proceed next", even though the harness is making it continue, and it will repeatedly loop saying it's not going to make progress until it gets an answer on how to proceed. In my experience the cases have been pretty reasonable, but also still ones where I wish it had done more.
• fwipsy 2mo ago
  The culture at frontier labs is set by people who have been in the field for over a decade--AI's true believers, who expect it to be a technology as dangerous and disruptive as nuclear weapons. They build it anyways because they think that if they don't do it, someone else will and use it against them. The same logic dictates that they make their models cybersecurity experts; otherwise, someone else will build it and hack them.