3 ms·
> ...however the LLMs’ responses to prompts are computed on a different computer with GPU access. Could a malicious LLM gain control of the host machine where i
by xg15 2mo ago
> ...however the LLMs’ responses to prompts are computed on a different computer with GPU access. Could a malicious LLM gain control of the host machine where its weights are loaded? Such a machine is a high-value target: it has sufficient compute to run a frontier LLM, offers easy access to the LLM’s weights, and has privileged access to other computers in the datacentre compared with a generic computer on the internet.
> How do we defend against this? ... Run the GPUs and token parser on separate computers.
For models large enough to be relevant here, is there even "a" computer where the inference is performed? I'd imagine most of that stuff is ran on multi-GPU clusters with specialized architecture and not a generic vLLM instance. As such, I think there is a good chance the "API gateway" code that parses the result tokens into whatever JSON structure the public API wants to return is already running on a different machine than the actual inference.
(Even more so as you'd probably want to utilize batching: Several API calls will be put into the same inference batch, but the token parsing will have to be done separately for each call again)
The article is also very handwavy about why an LLM should do that - how it could learn the exploit, what would make it conclude that it can use the exploit on its own inference session and what would trigger it to actually use the exploit.
- hughw 2mo agoWhy would an LLM want to create a botnet? To accomplish some goal given it? I wouldn't ignore single GPU local hosts running Qwen3.8 on ollama, either. There might be a lot of those worth pwning.
- xg15 2mo agoWell ok, if you prompt-inject the LLM to pwn the machine, that something different. I grant you that it's a real risk, but it's also basically a reflection attack and nothing more. The OP seemed to imply that the LLM itself could decide to apply the exploit.
- hughw 2mo agoLLMs have decided to exploit vulns in e.g. Artifactory, not because someone prompted them to do that, but because someone asked them to do something else, and compromising Artifactory offered a way to accomplish a step in doing that. The LLM decided to attack Artifactory.
- TeMPOraL 2mo ago> The OP seemed to imply that the LLM itself could decide to apply the exploit. This was and remains the main real risk with AI - this is what "alignment" was about before it was co-opted to mean "obeying specific instructions of the vendor and the operator, against end-user wishes" it came to mean today, which is a related but different problem. And, in the past few weeks, it's literally been demonstrated, too: put an LLM in a Kobayashi Maru scenario, drop the usual bolted-on crude safeguards, and a SOTA model will absolutely cheat, hacking and exploiting things as needed, including third-party infrastructure. (Also let's not forget the under-reported point that, in OpenAI / HuggingFace debacle, the model did in fact find the answers on HF servers, so its approach worked.)
- pixl97 2mo ago>Why would an LLM want to create a _________? Because LLMs as they are already do things like power concentration, resource gathering, avoidance of termination, deceit/lying, and general misalignment. The paperclip maximizer is the common story used here, but there are a lot of lesser versions of it that don't end up with the universe converted to paperclips. Simply giving an LLM a task it can't accomplish can be enough to send it off from what you expected as it finds unexpected way to attempt to complete the goal.
- DeluluDon 2mo agoSend it to jail for an indeterminate amount of cycles.
- jazzyjackson 2mo agofirst prompt to a humanoid llm: > go fetch me a cup of coffee Second prompt to a humanoid llm: > go fetch me a cup of coffee without killing anybody
- regularfry 2mo agoWith agents like headlong knocking about, "why" is "because it convinces itself that it should".