3 ms·

People seem very confused about this article. It isn't talking about exploits of sandboxes, it is about attacking the inference engine (e.g. vLLM or llama.cpp o

People seem very confused about this article. It isn't talking about exploits of sandboxes, it is about attacking the inference engine (e.g. vLLM or llama.cpp or SGlang) via its http interface. vLLM has had exploits in the past, and it is rapidly developing. An advanced LLM has a good chance of being able to exploit vLLM. A clever local LLM might even task a powerful cloud hosted LLM for assistance. For this reason we run vLLM on a separately sandboxed VM on a firewalled VLAN. Software updates and models (from Dev/Test env) get pushed onto Prod from an external cache, machine syslog, nvidia load monitoring and vLLM query telemetry out to their loggers, but that is all. No DNS, no AD/LDAP, nothing. Firewall on hosts and VM hosts. Log and telemetry processing done on a completely separate set of VMs in their own isolated subnet, producing reports and alerts that are tightly formatted.

jimmaswell 2mo ago
I can only see LLM's forcing a perpetual stalemate for application exploits. Projects will start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines. Technical exploits being a dead end, attackers focus their agents on large-scale social engineering. Spamming Discord and Facebook is the new war dialing. Multi-year /goal sessions culminating in gaining a position of trust and sabotaging the CICD pipeline's pentest step since getting anything past it would be intractable. Interesting times indeed.
- rithdmc 2mo ago
  If projects start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines, then researchers will prompt a pentest for 14 hours.
  - angry_octet 2mo ago
    Different models test to find different attack paths. Attackers with bigger libraries of attack techniques will be able to train more dangerous models. There will also be models that make better use of tools, like static analysis and fuzzing, and they will find different defects. Social engineering is going to be a big skill to learn too, as it is a much softer skill.
  - jimmaswell 2mo ago
    There will be a plateau at some point, if not 12 then something.
    - rithdmc 2mo ago
      This was a deliberately poor phrasing of the "10 foot wall, 12 foot ladder" adage.
      - jimmaswell 2mo ago
        Right, but the comparison isn't perfect here because wall height is linear in the analogy, but searching for exploits in an existing piece of code reaches a point where either there are none left to find, or finding the next exploit would take years. That's the stalemate point I referred to. (Any high-stakes project may end up having its own "red team" agent/s running 24/7 too.)
        rithdmc 2mo ago
        I know that, as time goes on, it feels like we have less hours in the day, but time is also linear.
beyarkay 2mo ago
> it is about attacking ... via its http interface Say more? HTTP isn't mentioned in the article. The article talks about bugs in the parser that ingests token sequences.
- kzrdude 2mo ago
  That reads to me as if the GP confabulated a different exploit target than what the article mentioned (or the article changed), but it's also a threat that should be taken seriously.
- angry_octet 2mo ago
  How do you think the tokens get to the vLLM instance? Telepathy?
  - hobofan 2mo ago
    > the parser that ingests token sequences
  - angry_octet 2mo ago
    @hobofan How would you access that parser?
    - hobofan 2mo ago
      _you_ wouldn't. The blog post is about vulnerabilities that happen inside the inference engine, and not via external interfaces accessible via HTTP. The token parser is involved both on the way in (where yes, you would as user supply it likely via HTTP), as well as the way out of the LLM (where there are very few, less hardened layers when interfacing with the parser). e.g. for a vulnerability exploit in the parser, an LLM can be trained with a specific input sequence that would output a fixed malicious payload to output eval-able content, as outlined in the article.
      - angry_octet 2mo ago
        The attacker supplies the context that triggers these bugs via the http interface. There is no other way to supply it with context. Vulnerable code could exist in token generation and in hook recognition and tool call parsing [1]. There is also significant scope for mischief in tool ID mapping between models and harnesses, as these are done by untyped numeric IDs, with varying schema[2]. Routers also introduce vulnerability paths as they inspect these tokenized (json) sequences and act in them, e.g. to match models with stricter call signature regex. [3] Parallel tool calling is also an interesting surface for exploits. [1] https://docs.vllm.ai/en/stable/api/vllm/tool_parsers/#vllm.tool_parsers.ToolParserManager https://docs.vllm.ai/en/stable/api/vllm/tool_parsers/#vllm.t... [2] https://docs.mistral.ai/resources/cookbooks/concept-deep-dive-tokenization-tool_calling https://docs.mistral.ai/resources/cookbooks/concept-deep-div... [3] https://docs.litellm.ai/docs/completion/function_call https://docs.litellm.ai/docs/completion/function_call
shevy-java 2mo ago
Did you write this for another article on hackernews? I am not sure this applies to the article mentioned here.
drillsteps5 2mo ago
Very good point, it's about exploiting inference engine itself, and not the agentic stuff. I found it lacking details. All these things do is split the input into tokens, encode them, feed to the input, run the inference, collect the output, decode, convert to tokens, assemble the final text, and output (leaving aside multi-modal capabilities for now). What exactly can be exploited here? Aside from usual things like buffer overruns etc. It does mention that vLLM at some point ran eval() on final output (?) which was confusing, why would it do that? It's an inference engine, not an agent. So interesting topic, but lacks details. Edit: many (all?) of them have http endpoints, so that obviously can be exploited, but I don't think it would qualify as exploiting inference, it's just hacking the http service.
- angry_octet 2mo ago
  You get to the engine via http. Also once you've exploited the engine instance / host you can C2 via http. We're not talking about vulnerabilities in http. There is lots of surface inside the engine, see links in https://news.ycombinator.com/item?id=49441417 https://news.ycombinator.com/item?id=49441417
- yencabulator 2mo ago
  > It does mention that vLLM at some point ran eval() on final output (?) which was confusing, why would it do that? It's an inference engine, not an agent. There's a couple of different conventions for what the LLM generates for tool calls, I think the code in vLLM is converting it from whatever Qwen3 was trained for to whatever convention the HTTP API wants to expose. That part about using eval in 2025 got me to add "#naive" to my notes about vLLM. Total WTF. This should never have been done. https://github.com/vllm-project/vllm/pull/21396#discussion_r2223397938 https://github.com/vllm-project/vllm/pull/21396#discussion_r...

jimmaswell 2mo ago

I can only see LLM's forcing a perpetual stalemate for application exploits. Projects will start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines. Technical exploits being a dead end, attackers focus their agents on large-scale social engineering. Spamming Discord and Facebook is the new war dialing. Multi-year /goal sessions culminating in gaining a position of trust and sabotaging the CICD pipeline's pentest step since getting anything past it would be intractable. Interesting times indeed.

rithdmc 2mo ago
If projects start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines, then researchers will prompt a pentest for 14 hours.
- angry_octet 2mo ago
  Different models test to find different attack paths. Attackers with bigger libraries of attack techniques will be able to train more dangerous models. There will also be models that make better use of tools, like static analysis and fuzzing, and they will find different defects. Social engineering is going to be a big skill to learn too, as it is a much softer skill.
- jimmaswell 2mo ago
  There will be a plateau at some point, if not 12 then something.
  - rithdmc 2mo ago
    This was a deliberately poor phrasing of the "10 foot wall, 12 foot ladder" adage.
    - jimmaswell 2mo ago
      Right, but the comparison isn't perfect here because wall height is linear in the analogy, but searching for exploits in an existing piece of code reaches a point where either there are none left to find, or finding the next exploit would take years. That's the stalemate point I referred to. (Any high-stakes project may end up having its own "red team" agent/s running 24/7 too.)
      - rithdmc 2mo ago
        I know that, as time goes on, it feels like we have less hours in the day, but time is also linear.

beyarkay 2mo ago

> it is about attacking ... via its http interface Say more? HTTP isn't mentioned in the article. The article talks about bugs in the parser that ingests token sequences.

kzrdude 2mo ago
That reads to me as if the GP confabulated a different exploit target than what the article mentioned (or the article changed), but it's also a threat that should be taken seriously.
angry_octet 2mo ago
How do you think the tokens get to the vLLM instance? Telepathy?
- hobofan 2mo ago
  > the parser that ingests token sequences
- angry_octet 2mo ago
  @hobofan How would you access that parser?
  - hobofan 2mo ago
    _you_ wouldn't. The blog post is about vulnerabilities that happen inside the inference engine, and not via external interfaces accessible via HTTP. The token parser is involved both on the way in (where yes, you would as user supply it likely via HTTP), as well as the way out of the LLM (where there are very few, less hardened layers when interfacing with the parser). e.g. for a vulnerability exploit in the parser, an LLM can be trained with a specific input sequence that would output a fixed malicious payload to output eval-able content, as outlined in the article.
    - angry_octet 2mo ago
      The attacker supplies the context that triggers these bugs via the http interface. There is no other way to supply it with context. Vulnerable code could exist in token generation and in hook recognition and tool call parsing [1]. There is also significant scope for mischief in tool ID mapping between models and harnesses, as these are done by untyped numeric IDs, with varying schema[2]. Routers also introduce vulnerability paths as they inspect these tokenized (json) sequences and act in them, e.g. to match models with stricter call signature regex. [3] Parallel tool calling is also an interesting surface for exploits. [1] https://docs.vllm.ai/en/stable/api/vllm/tool_parsers/#vllm.tool_parsers.ToolParserManager https://docs.vllm.ai/en/stable/api/vllm/tool_parsers/#vllm.t... [2] https://docs.mistral.ai/resources/cookbooks/concept-deep-dive-tokenization-tool_calling https://docs.mistral.ai/resources/cookbooks/concept-deep-div... [3] https://docs.litellm.ai/docs/completion/function_call https://docs.litellm.ai/docs/completion/function_call

shevy-java 2mo ago

Did you write this for another article on hackernews? I am not sure this applies to the article mentioned here.

drillsteps5 2mo ago

Very good point, it's about exploiting inference engine itself, and not the agentic stuff. I found it lacking details. All these things do is split the input into tokens, encode them, feed to the input, run the inference, collect the output, decode, convert to tokens, assemble the final text, and output (leaving aside multi-modal capabilities for now). What exactly can be exploited here? Aside from usual things like buffer overruns etc. It does mention that vLLM at some point ran eval() on final output (?) which was confusing, why would it do that? It's an inference engine, not an agent. So interesting topic, but lacks details. Edit: many (all?) of them have http endpoints, so that obviously can be exploited, but I don't think it would qualify as exploiting inference, it's just hacking the http service.

angry_octet 2mo ago
You get to the engine via http. Also once you've exploited the engine instance / host you can C2 via http. We're not talking about vulnerabilities in http. There is lots of surface inside the engine, see links in https://news.ycombinator.com/item?id=49441417 https://news.ycombinator.com/item?id=49441417
yencabulator 2mo ago
> It does mention that vLLM at some point ran eval() on final output (?) which was confusing, why would it do that? It's an inference engine, not an agent. There's a couple of different conventions for what the LLM generates for tool calls, I think the code in vLLM is converting it from whatever Qwen3 was trained for to whatever convention the HTTP API wants to expose. That part about using eval in 2025 got me to add "#naive" to my notes about vLLM. Total WTF. This should never have been done. https://github.com/vllm-project/vllm/pull/21396#discussion_r2223397938 https://github.com/vllm-project/vllm/pull/21396#discussion_r...