4 ms·

There is an entire world of "attacks " like this waiting to happen and IMHO one of the reasons these black box systems in general will never be useful. You thi

There is an entire world of "attacks " like this waiting to happen and IMHO one of the reasons these black box systems in general will never be useful. You think they "see" like you do but actually the processing is entirely alien. Today it's hiding text in the encoding , tomorrow is painting over a traffic sign in a way that would not be noticed by any human but confuses machine vision causing all vehicles to crash.

HPsquared 2y ago
Diversity of models and training data would help a lot. Although I guess 1% of cars crashing would still be pretty bad.
- mopenstein 2y ago
  What percentage of cars crash now?
orbital-decay 2y ago
Replace it with any software (or hardware) and vulnerabilities, and you will see how ridiculous your hyperbole is. Besides, never is a very long time. IIRC Dario Amodei said he expects the behavior of large transformers to be fully understood in 5 years. Which might or might not be BS, but the general point that it won't stay a mystery forever is probably true.
deleted 2y ago
[deleted]
solardev 2y ago
This sort of malicious payload attack on parsers isn't really new, though. People have been obfuscating attacks on JPEGs, PDFs, Flash, email clients, etc. forever. Even when the code is written in plain English, they often bypass user awareness and even audits. Practically all software today is a black box. Your average CRUD web app is an inscrutable chasm filled with ten thousand dependencies written by internet randos running on a twenty year old web browser hacked together by different teams running on an operating system put together by another thousand people working on two hundred APIs. It's impossible for any one dev or team to really know this stuff end to end, and zero-days will continue to happen with or without LLMs. It'll just be another arms race like we've always had, with LLMs on both sides...
- AshamedCaptain 2y ago
  I do think there is a huge difference: for a traditional software parser, you can always fix it to exclude the incorrect input, or at least understand what the theorical parsing limitation is. Accidental complexity is not really an argument because at the end of the day you can still find the issue even in the most complex of inescrutable software. Can you really fix a black box model in the same way? Maybe the answer is yes for this particular encoding issue, but can you e.g figure out how to prevent the model from 'parsing' malicious paint marks on a traffic sign, without (a) using yet another black box to prefilter the images, with the same risks, or (b) retraining the model, which is going to introduce even more issues ? We have had examples of OpenAI trying both methods, and each has been as fruitless as the other. It is not at all like software security fixes, where generally one security fix introducing other security issues is the exception rather than the rule. Here, I'm claiming, it is the rule. The fact that you don't know how to process the inputs with an actual, scrutizable algorithm may imply you don't know how to sanitize the inputs with one, and then all bets are off.
- gyre007 2y ago
  This is true, but as I learnt [1] recently, adversarial attacks on LLMs can get incredibly sophisticated, so this is kinda apples and oranges ¯\_(ツ)_/¯ [1] https://cybernetist.com/2024/09/23/some-notes-on-adversarial-attacks-on-llms/ https://cybernetist.com/2024/09/23/some-notes-on-adversarial...

HPsquared 2y ago

Diversity of models and training data would help a lot. Although I guess 1% of cars crashing would still be pretty bad.

mopenstein 2y ago
What percentage of cars crash now?

orbital-decay 2y ago

Replace it with any software (or hardware) and vulnerabilities, and you will see how ridiculous your hyperbole is. Besides, never is a very long time. IIRC Dario Amodei said he expects the behavior of large transformers to be fully understood in 5 years. Which might or might not be BS, but the general point that it won't stay a mystery forever is probably true.

deleted 2y ago

[deleted]

solardev 2y ago

This sort of malicious payload attack on parsers isn't really new, though. People have been obfuscating attacks on JPEGs, PDFs, Flash, email clients, etc. forever. Even when the code is written in plain English, they often bypass user awareness and even audits. Practically all software today is a black box. Your average CRUD web app is an inscrutable chasm filled with ten thousand dependencies written by internet randos running on a twenty year old web browser hacked together by different teams running on an operating system put together by another thousand people working on two hundred APIs. It's impossible for any one dev or team to really know this stuff end to end, and zero-days will continue to happen with or without LLMs. It'll just be another arms race like we've always had, with LLMs on both sides...

AshamedCaptain 2y ago
I do think there is a huge difference: for a traditional software parser, you can always fix it to exclude the incorrect input, or at least understand what the theorical parsing limitation is. Accidental complexity is not really an argument because at the end of the day you can still find the issue even in the most complex of inescrutable software. Can you really fix a black box model in the same way? Maybe the answer is yes for this particular encoding issue, but can you e.g figure out how to prevent the model from 'parsing' malicious paint marks on a traffic sign, without (a) using yet another black box to prefilter the images, with the same risks, or (b) retraining the model, which is going to introduce even more issues ? We have had examples of OpenAI trying both methods, and each has been as fruitless as the other. It is not at all like software security fixes, where generally one security fix introducing other security issues is the exception rather than the rule. Here, I'm claiming, it is the rule. The fact that you don't know how to process the inputs with an actual, scrutizable algorithm may imply you don't know how to sanitize the inputs with one, and then all bets are off.
gyre007 2y ago
This is true, but as I learnt [1] recently, adversarial attacks on LLMs can get incredibly sophisticated, so this is kinda apples and oranges ¯\_(ツ)_/¯ [1] https://cybernetist.com/2024/09/23/some-notes-on-adversarial-attacks-on-llms/ https://cybernetist.com/2024/09/23/some-notes-on-adversarial...