3 ms·
MIT creates method to force AI to comply with safety rules
- mixdup 21d agoThis kind of seems like a no-brainer. Instead of just letting a model have unfettered "physical" ability to do things and hope you can cognitively control it, why not let the AI do whatever it wants, but its access to the tools go through a hard-coded set of rules that is not subject to fuzzy interpretation Of course that depends on having controls that can't be circumvented which is a big if
- sigpwned 21d agoI agree, that seems like a configuration/policy/operational approach, which is how we handle this problem now for humans using RBAC and authn/authz, just applied to AI. People do a crude version of this today with sandboxing (where the AI's sphere of influence is strictly limited by its environment, barring misconfiguration of the sandbox or breaking out of the sandbox, of course) and with workflows (where AIs are integrated into deterministic workflows, and then deterministic, non-agentic code decides how to handle AI outputs). But integrating this into more agentic architectures with finer control just seems like a best practice, said that way. It's not a tradeoff, there's no drawback, just do it. In other words, yes, a no-brainer.
- lunarboy 21d agoIs this not the exact gap that happened for OpenAI's accidental hack of huggingface? They tried to sandbox network access but the Antifactory or whatever package has holes that the collective of agents abused
- montenegrohugo 21d agodoesnt work. this is a no-brainer because it's a bad solution. The whole point of intelligence is that it's generalizable. If you constrain it to some controlled things, then it ceases to be useful. its incompatible. the whole incentive with ai is to let it do whtv it wants.
- mixdup 21d agoWe don't even do it that way with people
- dpark 21d agoThat’s not what this is about. This is an algorithm for giving a model more freedom while nudging it in the right direction. It’s not about what tools are available.
- ck2 21d agoso what happens when the "AI" decides the only way to pass the test is to hack the harness and turn it off?
- hmokiguess 21d agoessentially this https://xkcd.com/2044/ https://xkcd.com/2044/
- dessimus 21d agoyeah, don't forget to roll in https://xkcd.com/927/ https://xkcd.com/927/
- guywithahat 21d agoReminds me of the huggingface hack
- Mr_P 21d agoIf you click through to the paper, it has approximately nothing to do with what this HN post title suggests.
- CharlesW 21d agoYes, it appears the submitter rewrote the title (strike 1) without even reading TFA (strike 2). Not great. Actual title: "New MIT Algorithm Meets Every Hard Constraint in Simulated Tests"
- DonsDiscountGas 21d agoIndeed. Which is a shame because it's still pretty cool work.
- dpark 21d agoIf fairness this article is poorly written and doesn’t explain what they actually did at all.
- petcat 21d ago> For constraint satisfaction, what ultimately matters is the model’s final output, since the internal process is discarded. By not requiring every intermediate step to satisfy the constraints, we give the model more freedom to find high-quality solutions that are still feasible in the end. My (maybe naive) question is if we only check the final result then isn't it already too late and possibly the safety rules have already been irreversibly violated? It gives the example of a robot arm avoiding obstacles while still finding the shortest path, but if we only check the correctness at the end, then isn't it possible that it already collided with an obstacle?
- mixdup 21d ago> but if we only check the correctness at the end, then isn't it possible that it already collided with an obstacle? You would put the check before it actually does the thing. It's at the "end" of the process of figuring out what it wants to do, not the end of fulfilling the request or prompt
- Sarvaturi 21d ago[dead]
- dpark 21d agoI’m pretty sure this is just a poorly written article. HardFlow seems to be a strategy for nudging the model in the right direction while giving it more freedom. Only applying the constraints at the end is a mischaracterization from what I can tell.
- Sarvaturi 21d ago[dead]
- ianjbutler 21d agoOutcome reward vs process reward models. The second is obviously better.. like getting partial credit on a physics test for wrong answers but correct method. Research is gradually hybridizing them but historically we avoided doing it the right way because of practical difficulties (labels required, more expensive and difficult) and more ideological ones (believers in magical machine intuition think it sounds too classical / logic based to be useful, pin their hopes on unproven faith in grokking at scale).
- sailfast 21d agoWould love to see this tested on some of the newer cybersecurity models so we could actually defend ourselves instead of getting cut off at the knees by silly regular expressions. Hope this approach gets well tested and sees good results so we have a shot at human governance.
- verdverm 21d agoThe actual paper: https://arxiv.org/abs/2511.08425v3 https://arxiv.org/abs/2511.08425v3 > Our key insight is to leverage numerical optimal control to steer the sampling trajectory so that constraints are satisfied precisely at the terminal time. Doesn't seem so "fool proof" to me as where the inevitable media spin will take it. Then, how do you know "where" to steer weights? "Safe" has no agreed upon definition
- WalterSobchak 21d agoMIT's blog post: https://news.mit.edu/2026/new-method-enables-ai-safety-critical-situations-0914 https://news.mit.edu/2026/new-method-enables-ai-safety-criti...
- arionhardison 21d agoI had a swarm break out about 18 months ago; so I stopped and decided I really wanted to dig into it. 1. My agents do not take direct action, they run programs. 2. Programs are not LLM hits/real-time output; they don't MAX tokens the MAX determinism. 3. Programs are logistical wrappers for Protocols where the guardrails are (RLVR.ai) 4. Policies for generating programs are democratically governed re: fec.dev - they get voted on 5. Elected-HITL implements the policy pipelines and ontological abstract intents [and their maps] I would be really interested to learn about the Gov. models that others are using but this seems to be something that linked0-in (which i loathe) discusses (in the most pedestrian/luddite) terms more than HN.
- Suhinnall27 21d agoThe idea of enforcing constraints only on the final output instead of every intermediate step is pretty interesting. I wonder how well this would translate to language models, where “safe” is much harder to define mathematically than a robot avoiding an obstacle.
- MattCruikshank 21d ago[Cackles in Jeff Goldblum.]
- nekusar 21d agoWHOSE SAFETY? What are the rules? Or does sharing the rules present security problems, so they're not shared? What are the ethics axioms? And why should I trust your ethical framework?
- dpark 21d agoIt’s talking about physical safety. Not ethical safety. The concern here is robots physically colliding with things.
- jcfrei 21d ago> In experiments spanning robotics, control of physical processes, and computer vision, the new method consistently satisfied the required constraints while identifying better solutions than existing techniques. I guess this method works when you can precisely quantify the allowed output - like the degrees an arm can move or the path a robot can take, etc. But it doesn't appear to be applicable to an AI writing code - which is where our main concerns currently are.