5 ms·
Before LLMs we had a pretty good idea of security boundaries in software. Applications didn’t trust user input. Operating systems didn’t trust applications. Ser
by paxys 2mo ago
Before LLMs we had a pretty good idea of security boundaries in software. Applications didn’t trust user input. Operating systems didn’t trust applications. Services and processes didn’t trust each other. There were always tokens, scopes, delegated grants.
Suddenly every AI company’s security model seems to be to say “pretty please” to a non-deterministic machine and hope for the best. And if there is a security failure instead of accepting blame they go “well we can’t help it, our model is too intelligent”.
- mannanj 2mo agoBefore LLMs we didnt have much accountability from leadership, that was eroding over time. After LLMs we still dont.
- orbital-decay 2mo agoSuddenly determinism has other meanings than "same input leads to the same output". I'm still confused by this and not sure how it happened so easily, but it seems to be accepted by everyone now.
- pixl97 2mo agoI may say you're thinking of the term "deterministic algorithm" as determinism is more of a philosophical definition that has evolved over the years.
- orbital-decay 2mo agoPhilosophical determinism is pretty similar in spirit to physics/CS (roughly same cause = same effect, no free will/side effects), and what parent calls determinism seems to be neither. It's rather about natural language prompts not being formally specified in the first place. Unreliability inherent to any intelligence (artificial or natural)? but not determinism I think. The word somehow got universally hijacked.
- jimbokun 2mo agoAmen. Why can’t we give agents a shell with permissions for programs and file system access controlled by Unix permissions? This seemed to be a solved problem back in the systems where many users were logged into one machine and the admins had to keep everyone from impacting each other.
- winstonwinston 2mo agoIt would loose automagic? In a recent full discloure I was reading about a CVE of an LLM agent, the vendor installed a “secure sandbox VM” and then just shared a host’s filesystem read/write to the agent’s VM.
- cryptonector 2mo agoWe never solved the restricted shell problem for humans.
- jimbokun 2mo agoCan you elaborate? Read, write, execute privileges on files and directories goes a long way. What's missing? The biggest is Internet access, or networking in general, I suppose.
- cryptonector 2mo agoTwo problems. First, it is remarkably difficult to come up with a set of programs that it is safe to let the restricted user use. Second, it is remarkably difficult to make the restricted environment useful enough if you're really serious about allowing only safe programs to be used. Try to make it useful enough and you end up with escapes everywhere.
- jimbokun 2mo agoSure but then the file protections also apply to the programs used by the agent. If you can strictly limit the agent to only modifying files in the source code directory of a single repository, that greatly limits the blast radius of damage.
- cryptonector 2mo agoAnd the "oh noes! our model is too intelligent!!" thing is advertising.