3 ms·

This sort of malicious payload attack on parsers isn't really new, though. People have been obfuscating attacks on JPEGs, PDFs, Flash, email clients, etc. forev

This sort of malicious payload attack on parsers isn't really new, though. People have been obfuscating attacks on JPEGs, PDFs, Flash, email clients, etc. forever. Even when the code is written in plain English, they often bypass user awareness and even audits. Practically all software today is a black box. Your average CRUD web app is an inscrutable chasm filled with ten thousand dependencies written by internet randos running on a twenty year old web browser hacked together by different teams running on an operating system put together by another thousand people working on two hundred APIs. It's impossible for any one dev or team to really know this stuff end to end, and zero-days will continue to happen with or without LLMs. It'll just be another arms race like we've always had, with LLMs on both sides...

AshamedCaptain 2y ago
I do think there is a huge difference: for a traditional software parser, you can always fix it to exclude the incorrect input, or at least understand what the theorical parsing limitation is. Accidental complexity is not really an argument because at the end of the day you can still find the issue even in the most complex of inescrutable software. Can you really fix a black box model in the same way? Maybe the answer is yes for this particular encoding issue, but can you e.g figure out how to prevent the model from 'parsing' malicious paint marks on a traffic sign, without (a) using yet another black box to prefilter the images, with the same risks, or (b) retraining the model, which is going to introduce even more issues ? We have had examples of OpenAI trying both methods, and each has been as fruitless as the other. It is not at all like software security fixes, where generally one security fix introducing other security issues is the exception rather than the rule. Here, I'm claiming, it is the rule. The fact that you don't know how to process the inputs with an actual, scrutizable algorithm may imply you don't know how to sanitize the inputs with one, and then all bets are off.
gyre007 2y ago
This is true, but as I learnt [1] recently, adversarial attacks on LLMs can get incredibly sophisticated, so this is kinda apples and oranges ¯\_(ツ)_/¯ [1] https://cybernetist.com/2024/09/23/some-notes-on-adversarial-attacks-on-llms/ https://cybernetist.com/2024/09/23/some-notes-on-adversarial...

AshamedCaptain 2y ago

I do think there is a huge difference: for a traditional software parser, you can always fix it to exclude the incorrect input, or at least understand what the theorical parsing limitation is. Accidental complexity is not really an argument because at the end of the day you can still find the issue even in the most complex of inescrutable software. Can you really fix a black box model in the same way? Maybe the answer is yes for this particular encoding issue, but can you e.g figure out how to prevent the model from 'parsing' malicious paint marks on a traffic sign, without (a) using yet another black box to prefilter the images, with the same risks, or (b) retraining the model, which is going to introduce even more issues ? We have had examples of OpenAI trying both methods, and each has been as fruitless as the other. It is not at all like software security fixes, where generally one security fix introducing other security issues is the exception rather than the rule. Here, I'm claiming, it is the rule. The fact that you don't know how to process the inputs with an actual, scrutizable algorithm may imply you don't know how to sanitize the inputs with one, and then all bets are off.

gyre007 2y ago

This is true, but as I learnt [1] recently, adversarial attacks on LLMs can get incredibly sophisticated, so this is kinda apples and oranges ¯\_(ツ)_/¯ [1] https://cybernetist.com/2024/09/23/some-notes-on-adversarial-attacks-on-llms/ https://cybernetist.com/2024/09/23/some-notes-on-adversarial...