3 ms·

Well it is true, the XZ hack was a major social-engineering feat. Regardless, we should not be underestimating the power of these technologies. If an LLM can re

by af3d 3y ago

Well it is true, the XZ hack was a major social-engineering feat. Regardless, we should not be underestimating the power of these technologies. If an LLM can respond to a request to generate code, what makes you think the attacker couldn't simply add things like "Insert a buffer overflow in the code..." or what have you. And especially as this technology progresses, the capabilities will no doubt dramatically increase. Having said that, I do hope you are right!

talldayo 3y ago
> If an LLM can respond to a request to generate code, what makes you think the attacker couldn't simply add things like "Insert a buffer overflow in the code..." or what have you. What makes you think an attacker couldn't do that without an LLM? I'm having a hard time understanding what changes in this scenario by introducing an AI. Pretty much anyone can write a buffer overflow exploit with enough research. The harder part is getting your patches through code review, which is probably only hindered by having ChatGPT help you. Again... the past few decades have mostly gone off without a hitch, and there have been a lot of monkeys on typewriters hooked up to the internet.
- af3d 3y ago
  I don't know, LLM's are very good at embedding subtle changes in their responses. They are also good at researching things like "best possible place to hide bugs", etc. Is it really a stretch to imagine that these technologies won't, if not now, but "someday soon" be used in such a way? I certainly don't think so.
  - talldayo 3y ago
    Humans are really good at subtlety and research. I guess my point is this: pre-LLM era XZ-style exploit: takes 5 years to attain developer trust, then you merge malicious patches and hope nobody sees them post-LLM era XZ-style exploit: takes 5 years to attain developer trust, then you merge patches that were written slightly faster and hope nobody sees them For 99% of people, there isn't a meaningful difference between these situations. I really don't think there are people out there that couldn't write a buffer overflow exploit in 5 years but could groom maintainers for trust in the meantime. The code-writing and prose bit is the easy part. The social engineering and deception is hard enough for a human, and Turing-assured destruction for LLMs.

by af3d 3y ago

talldayo 3y ago

> If an LLM can respond to a request to generate code, what makes you think the attacker couldn't simply add things like "Insert a buffer overflow in the code..." or what have you. What makes you think an attacker couldn't do that without an LLM? I'm having a hard time understanding what changes in this scenario by introducing an AI. Pretty much anyone can write a buffer overflow exploit with enough research. The harder part is getting your patches through code review, which is probably only hindered by having ChatGPT help you. Again... the past few decades have mostly gone off without a hitch, and there have been a lot of monkeys on typewriters hooked up to the internet.

af3d 3y ago
I don't know, LLM's are very good at embedding subtle changes in their responses. They are also good at researching things like "best possible place to hide bugs", etc. Is it really a stretch to imagine that these technologies won't, if not now, but "someday soon" be used in such a way? I certainly don't think so.
- talldayo 3y ago
  Humans are really good at subtlety and research. I guess my point is this: pre-LLM era XZ-style exploit: takes 5 years to attain developer trust, then you merge malicious patches and hope nobody sees them post-LLM era XZ-style exploit: takes 5 years to attain developer trust, then you merge patches that were written slightly faster and hope nobody sees them For 99% of people, there isn't a meaningful difference between these situations. I really don't think there are people out there that couldn't write a buffer overflow exploit in 5 years but could groom maintainers for trust in the meantime. The code-writing and prose bit is the easy part. The social engineering and deception is hard enough for a human, and Turing-assured destruction for LLMs.