3 ms·
MIT creates method to force AI to comply with safety rules
- mixdup 2mo agoThis kind of seems like a no-brainer. Instead of just letting a model have unfettered "physical" ability to do things and hope you can cognitively control it, why not let the AI do whatever it wants, but its access to the tools go through a hard-coded set of rules that is not subject to fuzzy interpretation Of course that depends on having controls that can't be circumvented which is a big if
- sigpwned 2mo agoI agree, that seems like a configuration/policy/operational approach, which is how we handle this problem now for humans using RBAC and authn/authz, just applied to AI. People do a crude version of this today with sandboxing (where the AI's sphere of influence is strictly limited by its environment, barring misconfiguration of the sandbox or breaking out of the sandbox, of course) and with workflows (where AIs are integrated into deterministic workflows, and then deterministic, non-agentic code decides how to handle AI outputs). But integrating this into more agentic architectures with finer control just seems like a best practice, said that way. It's not a tradeoff, there's no drawback, just do it. In other words, yes, a no-brainer.
- lunarboy 2mo agoIs this not the exact gap that happened for OpenAI's accidental hack of huggingface? They tried to sandbox network access but the Antifactory or whatever package has holes that the collective of agents abused
- montenegrohugo 2mo agodoesnt work. this is a no-brainer because it's a bad solution. The whole point of intelligence is that it's generalizable. If you constrain it to some controlled things, then it ceases to be useful. its incompatible. the whole incentive with ai is to let it do whtv it wants.
- mixdup 2mo agoWe don't even do it that way with people
- dpark 2mo agoThat’s not what this is about. This is an algorithm for giving a model more freedom while nudging it in the right direction. It’s not about what tools are available.
- ck2 2mo agoso what happens when the "AI" decides the only way to pass the test is to hack the harness and turn it off?
- hmokiguess 2mo agoessentially this https://xkcd.com/2044/ https://xkcd.com/2044/
- dessimus 2mo agoyeah, don't forget to roll in https://xkcd.com/927/ https://xkcd.com/927/
- guywithahat 2mo agoReminds me of the huggingface hack
- Mr_P 2mo agoIf you click through to the paper, it has approximately nothing to do with what this HN post title suggests.
- CharlesW 2mo agoYes, it appears the submitter rewrote the title (strike 1) without even reading TFA (strike 2). Not great. Actual title: "New MIT Algorithm Meets Every Hard Constraint in Simulated Tests"
- DonsDiscountGas 2mo agoIndeed. Which is a shame because it's still pretty cool work.
- dpark 2mo agoIf fairness this article is poorly written and doesn’t explain what they actually did at all.
- petcat 2mo ago> For constraint satisfaction, what ultimately matters is the model’s final output, since the internal process is discarded. By not requiring every intermediate step to satisfy the constraints, we give the model more freedom to find high-quality solutions that are still feasible in the end. My (maybe naive) question is if we only check the final result then isn't it already too late and possibly the safety rules have already been irreversibly violated? It gives the example of a robot arm avoiding obstacles while still finding the shortest path, but if we only check the correctness at the end, then isn't it possible that it already collided with an obstacle?
- mixdup 2mo ago> but if we only check the correctness at the end, then isn't it possible that it already collided with an obstacle? You would put the check before it actually does the thing. It's at the "end" of the process of figuring out what it wants to do, not the end of fulfilling the request or prompt
- Sarvaturi 2mo ago[dead]
- dpark 2mo agoI’m pretty sure this is just a poorly written article. HardFlow seems to be a strategy for nudging the model in the right direction while giving it more freedom. Only applying the constraints at the end is a mischaracterization from what I can tell.
- Sarvaturi 2mo ago[dead]
- ianjbutler 2mo agoOutcome reward vs process reward models. The second is obviously better.. like getting partial credit on a physics test for wrong answers but correct method. Research is gradually hybridizing them but historically we avoided doing it the right way because of practical difficulties (labels required, more expensive and difficult) and more ideological ones (believers in magical machine intuition think it sounds too classical / logic based to be useful, pin their hopes on unproven faith in grokking at scale).
- sailfast 2mo agoWould love to see this tested on some of the newer cybersecurity models so we could actually defend ourselves instead of getting cut off at the knees by silly regular expressions. Hope this approach gets well tested and sees good results so we have a shot at human governance.
- verdverm 2mo agoThe actual paper: https://arxiv.org/abs/2511.08425v3 https://arxiv.org/abs/2511.08425v3 > Our key insight is to leverage numerical optimal control to steer the sampling trajectory so that constraints are satisfied precisely at the terminal time. Doesn't seem so "fool proof" to me as where the inevitable media spin will take it. Then, how do you know "where" to steer weights? "Safe" has no agreed upon definition
- WalterSobchak 2mo agoMIT's blog post: https://news.mit.edu/2026/new-method-enables-ai-safety-critical-situations-0914 https://news.mit.edu/2026/new-method-enables-ai-safety-criti...
- arionhardison 2mo agoI had a swarm break out about 18 months ago; so I stopped and decided I really wanted to dig into it. 1. My agents do not take direct action, they run programs. 2. Programs are not LLM hits/real-time output; they don't MAX tokens the MAX determinism. 3. Programs are logistical wrappers for Protocols where the guardrails are (RLVR.ai) 4. Policies for generating programs are democratically governed re: fec.dev - they get voted on 5. Elected-HITL implements the policy pipelines and ontological abstract intents [and their maps] I would be really interested to learn about the Gov. models that others are using but this seems to be something that linked0-in (which i loathe) discusses (in the most pedestrian/luddite) terms more than HN.
- Suhinnall27 2mo agoThe idea of enforcing constraints only on the final output instead of every intermediate step is pretty interesting. I wonder how well this would translate to language models, where “safe” is much harder to define mathematically than a robot avoiding an obstacle.
- MattCruikshank 2mo ago[Cackles in Jeff Goldblum.]
- nekusar 2mo agoWHOSE SAFETY? What are the rules? Or does sharing the rules present security problems, so they're not shared? What are the ethics axioms? And why should I trust your ethical framework?
- dpark 2mo agoIt’s talking about physical safety. Not ethical safety. The concern here is robots physically colliding with things.
- jcfrei 2mo ago> In experiments spanning robotics, control of physical processes, and computer vision, the new method consistently satisfied the required constraints while identifying better solutions than existing techniques. I guess this method works when you can precisely quantify the allowed output - like the degrees an arm can move or the path a robot can take, etc. But it doesn't appear to be applicable to an AI writing code - which is where our main concerns currently are.