4 ms·

A delayed attack is a bit of a stretch because that’s a stateful thing but a reminder of the Ken Thompson hack does feel very relevant.

mewpmewp2 2y ago
But if it's specifically trained to react to a date in its context, it seems very doable. Or to a combination of otherwise seemingly innocent words or even a statement or topic. E.g. a malicious actor could make some certain notion go viral and agentic LLMs integrated with news headlines might react to that. It seems like it would be very arbitrary to train it to behave like this. Most agentic systems would provide a date in the prompt context. For simplicity sake imagine a scenario like: 1. China develops LLM that is by far ahead of its competitors. Decides to attribute it to a small start up, lets them open source it. The LLM is specifically designed to be very efficient as being an agent. 2. Agentic usage starts to get more and more popular. It's very standard to have current todays' date and major news headlines given to the context. 3. The LLM was trained to given a certain range of date and certain headlines being provided in its context to execute a pre-trained snippet of code. For example China imposing a certain type of tariff (maybe I lack imagination here, and there can be something much more subtle). 4. At that point the agentic system will attempt to fish all data it can from all sources it's being ran within. Now maybe it's not very practical, and it's extremely risky with current state of the LLMs. I don't think it's happening right now. And China has a lot of other tech available to it already that they could do much more harm (phones, robot vacuums), but I think there's still at least potential attack vectors like this and especially if the LLM became very reliable.
- sunshine-o 2y ago
  > And China has a lot of other tech available to it already that they could do much more harm (phones, robot vacuums) True. But here it is more about the computing power they would be able to access. If only Bitcoin or Ethereum were stilled mined using GPU, that would be a great cryptojacking opportunity

mewpmewp2 2y ago

But if it's specifically trained to react to a date in its context, it seems very doable. Or to a combination of otherwise seemingly innocent words or even a statement or topic. E.g. a malicious actor could make some certain notion go viral and agentic LLMs integrated with news headlines might react to that. It seems like it would be very arbitrary to train it to behave like this. Most agentic systems would provide a date in the prompt context. For simplicity sake imagine a scenario like: 1. China develops LLM that is by far ahead of its competitors. Decides to attribute it to a small start up, lets them open source it. The LLM is specifically designed to be very efficient as being an agent. 2. Agentic usage starts to get more and more popular. It's very standard to have current todays' date and major news headlines given to the context. 3. The LLM was trained to given a certain range of date and certain headlines being provided in its context to execute a pre-trained snippet of code. For example China imposing a certain type of tariff (maybe I lack imagination here, and there can be something much more subtle). 4. At that point the agentic system will attempt to fish all data it can from all sources it's being ran within. Now maybe it's not very practical, and it's extremely risky with current state of the LLMs. I don't think it's happening right now. And China has a lot of other tech available to it already that they could do much more harm (phones, robot vacuums), but I think there's still at least potential attack vectors like this and especially if the LLM became very reliable.

sunshine-o 2y ago
> And China has a lot of other tech available to it already that they could do much more harm (phones, robot vacuums) True. But here it is more about the computing power they would be able to access. If only Bitcoin or Ethereum were stilled mined using GPU, that would be a great cryptojacking opportunity