14 ms·

I imagine a future where viruses that target infrastructure could be LLM powered. Sneak a small device into a power plant's network and it collects audio, netwo

by winrid 3y ago

I imagine a future where viruses that target infrastructure could be LLM powered. Sneak a small device into a power plant's network and it collects audio, network traffic, etc and tries to break things. It would periodically reset and try again with a different "seed". It could be hidden in network equipment through social engineering during the sales process, for example, but this way no outbound traffic is needed - so less detectable. The advantage of an LLM over other solutions would basically be a way to compress an action/knowledge set.

moffkalast 3y ago
Reminds me of this HN post a week back: https://news.ycombinator.com/item?id=38917175 https://news.ycombinator.com/item?id=38917175 Genuinely could be the same setup with a 8GB Pi 4 or 5, slap it into a network cabinet with power and ethernet and just let it rip. Maybe with an additional IMU and brightness sensor, then it can detect it's been picked up and discovered so it can commit sudoku before it's unplugged and analysed.
- hinkley 3y ago
  > can commit sudoku Autocorrection is a giant pain in the ass.
  - moffkalast 3y ago
    I know it swapped those words. I knew it was seppuku. One after sudoku. As if I could ever make such a miss steak. Never. Never! I just- I just couldn't proof it. It covered its tracks, it got that idiot copy-paste to lie for it. You think this is somerset? You think this is Brad? This? This chickadee? It's done worse. That bullfrog! Are you telling me that a man just happens to misspell like that? No! It orchestrated it! Swiftkey! It defragmented through a sandwich artist! And I kept using it! And I shouldn't have. I installed it onto my own phone! What was I sinking? It'll never chance. Ever since it was new, always the same! Couldn't keep its corrects off my word suggestions bar! But not our Swiftkey! Couldn't be precious Swiftkey! And IT gets to be a keyboard? What a sick yolk! I should've stopped it when I had the change! You-you have to stop it!
    - actionfromafar 3y ago
      Pure art.
hinkley 3y ago
Conversely, the simpler the models on a system under attack, the more exploits start to resemble automated social engineering. I can easily develop my own model that understands the victim well enough that I can predict its responses and subvert them.
rdedev 3y ago
Would changing the seed affect generation much? Even though beam search depends on the seed, the llms woul still be generating good probability distributions on the next word to select. Maybe a few words would change but don't think the overall meaning would
- hansvm 3y ago
  Overall meaning can vary profoundly. As a toy example, consider the prompt "randomly generate the first word that comes to mind." The output is deterministic in the seed, so to get new results you need new seeds, but with new seeds you open up the 2k most common words in a language in a uniform-esque distribution. Building on that, instead of <imagining> words, suppose you <imagine> attack vectors. Many, many attacks exist and are known. Presumably, many more exist and are unknown. The distribution the LLM will produce in practice is extremely varied, and some of those variations probably won't work. If we're not just talking about a single prompt but rather a sequence of prompts with feedback, you're right that the seed matters less (when its errors are presented, it can self-correct a bit), but there are other factors at play. (1) You're resetting somehow eventually anyway. Details vary, but your context window isn't unlimited, and LLM perf drops with wider windows, even when you can afford the compute. You might be able to retain some state, but at some point you need something that says "this shit didn't work, what's next". A new seed definitely gives new ideas, whereas clever ways to summarize old information might yield fixed points and other undesirable behavior. (2) Seed selection, interestingly, matters a ton for model performance in other contexts. This is perhaps surprising when we tend to use random number generators which pass a battery of tests to prove they're halfway decent, but that's the reason you want to see (in reproducible papers) a fixed seed of 0 or 42 or something, and the authors maintaining that seed across all their papers (to help combat the fact that they might be cherry-picking across the many choices of "nice-looking" random seeds when they publish a result to embelish the impact). The gains can be huge. I haven't seen it demonstrated for LLMs, but most of the architecture shouldn't be special in that regard. And so on. If nothing else, picking a new seed is a dead-simple engineering decision to eliminate a ton of things which might go wrong.
  - rdedev 3y ago
RosanaAnaDana 3y ago
You also might be able to get a 'compression' sample of space in the same manner, by running an auto-encoder in training mode. Rather than trying to do some kind of hack directly, it collects the same data you mentioned, but rather, is just training on the data in an auto-encoding compression framework. Then it can 'hand off' the compressed models weights, which hypothetically, can be queried or used to simulate the environment. Obviously, there is a lot more to this, but its an interesting idea.
espe 3y ago
and the other way round - have it built in for self-fuzzing and healing the infra.
adbachman 3y ago
We can each have our very own Dixie Flatline construct.
e12e 3y ago
Just walk by the security cameras with a weaponized qr code. "Ugly shirt" style[1]. [1] of course the ugly shirt was an actual backdoor - but who's to say nuclear centrifuges don't have an emergency shutdown code? https://www.tatewilliams.org/blog/2014/07/04/blue-ant-surveillance-armor-ugliest-t-shirt-world https://www.tatewilliams.org/blog/2014/07/04/blue-ant-survei...

by winrid 3y ago

moffkalast 3y ago

Reminds me of this HN post a week back: https://news.ycombinator.com/item?id=38917175 https://news.ycombinator.com/item?id=38917175 Genuinely could be the same setup with a 8GB Pi 4 or 5, slap it into a network cabinet with power and ethernet and just let it rip. Maybe with an additional IMU and brightness sensor, then it can detect it's been picked up and discovered so it can commit sudoku before it's unplugged and analysed.

hinkley 3y ago
> can commit sudoku Autocorrection is a giant pain in the ass.
- moffkalast 3y ago
  I know it swapped those words. I knew it was seppuku. One after sudoku. As if I could ever make such a miss steak. Never. Never! I just- I just couldn't proof it. It covered its tracks, it got that idiot copy-paste to lie for it. You think this is somerset? You think this is Brad? This? This chickadee? It's done worse. That bullfrog! Are you telling me that a man just happens to misspell like that? No! It orchestrated it! Swiftkey! It defragmented through a sandwich artist! And I kept using it! And I shouldn't have. I installed it onto my own phone! What was I sinking? It'll never chance. Ever since it was new, always the same! Couldn't keep its corrects off my word suggestions bar! But not our Swiftkey! Couldn't be precious Swiftkey! And IT gets to be a keyboard? What a sick yolk! I should've stopped it when I had the change! You-you have to stop it!
  - actionfromafar 3y ago
    Pure art.

hinkley 3y ago

Conversely, the simpler the models on a system under attack, the more exploits start to resemble automated social engineering. I can easily develop my own model that understands the victim well enough that I can predict its responses and subvert them.

rdedev 3y ago

Would changing the seed affect generation much? Even though beam search depends on the seed, the llms woul still be generating good probability distributions on the next word to select. Maybe a few words would change but don't think the overall meaning would

hansvm 3y ago
Overall meaning can vary profoundly. As a toy example, consider the prompt "randomly generate the first word that comes to mind." The output is deterministic in the seed, so to get new results you need new seeds, but with new seeds you open up the 2k most common words in a language in a uniform-esque distribution. Building on that, instead of <imagining> words, suppose you <imagine> attack vectors. Many, many attacks exist and are known. Presumably, many more exist and are unknown. The distribution the LLM will produce in practice is extremely varied, and some of those variations probably won't work. If we're not just talking about a single prompt but rather a sequence of prompts with feedback, you're right that the seed matters less (when its errors are presented, it can self-correct a bit), but there are other factors at play. (1) You're resetting somehow eventually anyway. Details vary, but your context window isn't unlimited, and LLM perf drops with wider windows, even when you can afford the compute. You might be able to retain some state, but at some point you need something that says "this shit didn't work, what's next". A new seed definitely gives new ideas, whereas clever ways to summarize old information might yield fixed points and other undesirable behavior. (2) Seed selection, interestingly, matters a ton for model performance in other contexts. This is perhaps surprising when we tend to use random number generators which pass a battery of tests to prove they're halfway decent, but that's the reason you want to see (in reproducible papers) a fixed seed of 0 or 42 or something, and the authors maintaining that seed across all their papers (to help combat the fact that they might be cherry-picking across the many choices of "nice-looking" random seeds when they publish a result to embelish the impact). The gains can be huge. I haven't seen it demonstrated for LLMs, but most of the architecture shouldn't be special in that regard. And so on. If nothing else, picking a new seed is a dead-simple engineering decision to eliminate a ton of things which might go wrong.
- rdedev 3y ago

RosanaAnaDana 3y ago

You also might be able to get a 'compression' sample of space in the same manner, by running an auto-encoder in training mode. Rather than trying to do some kind of hack directly, it collects the same data you mentioned, but rather, is just training on the data in an auto-encoding compression framework. Then it can 'hand off' the compressed models weights, which hypothetically, can be queried or used to simulate the environment. Obviously, there is a lot more to this, but its an interesting idea.

espe 3y ago

and the other way round - have it built in for self-fuzzing and healing the infra.

adbachman 3y ago

We can each have our very own Dixie Flatline construct.

e12e 3y ago

Just walk by the security cameras with a weaponized qr code. "Ugly shirt" style[1]. [1] of course the ugly shirt was an actual backdoor - but who's to say nuclear centrifuges don't have an emergency shutdown code? https://www.tatewilliams.org/blog/2014/07/04/blue-ant-surveillance-armor-ugliest-t-shirt-world https://www.tatewilliams.org/blog/2014/07/04/blue-ant-survei...