4 ms·
As well as this being our regular reminder to be careful what you pull from public repositories and other sources, and to verify your dependency trees, it raise
by dspillett 3y ago
As well as this being our regular reminder to be careful what you pull from public repositories and other sources, and to verify your dependency trees, it raises another question:
If malware is massively prolific in public repos, how much does this affect LLMs and other automation tools that are trained using the contents of such resources? What are the chances that we'll see copilot & friends occasionally emit malware in response to coding questions that generate responses long enough for accidentally malicious parts to hide amongst? Simpler vulnerabilities such as simple injection vectors have often been seen already.
- cqqxo4zV46cp 3y agoThis sounds like alarmist journalist talk. What “malware”, even subtle backdoors, are sneaking into the LLM-generated code used in a piece of software that’s actually worth a damn?
- drsim 3y agoIt’s a valid question to posit.
- deleted 3y ago[deleted]
- egberts1 3y agoTry this "write me a port scanning code in Packrat-style JavaScript language that can run from within a PDF document."
- ben_w 3y agoWe'll only find out by looking. Given that LLMs are popular coding assistants, I suspect there are already many issues similar to `goto fail;`
- dspillett 3y agoThere are certainly people slapping AI generated code into production in small projects without adequately checking them, leading to things like inadequate input validating leading to open XSS and injection vectors. With too little oversight in a more significant project it is only a matter of time before this happens somewhere that results in, for instance, a DoS that affects many people, or a personal data leek. Given the way LLMs are trained, it might be unlike but it is conceivable that if they see deliberate back doors injected into enough of the training data, they'll consider it to be a valid part of a certain class of solution and output the same themselves. It is a big step again from deliberate back-doors to active malware, but not an inconceivable one IMO if large enough chunks of code are being trusted with minimal testing.
- pjc50 3y agoI'd expect it to accidentally invent vulnerabilities of its own as well as pasting existing ones from the input set. AI provides no guarantees at all about correctness.
- dartos 3y agoYou can use grammars to guarantee correct syntax, at least.
- pfarrell 3y agoMakes me think about Dijkstra being horrified to learn that the US was sending astronauts into space using unproven computer code [0]. Eventually, we'll have programmers trusting the AI generated code as much as we trust compiler generated code. Sometimes I think this will be the evolution of programming (like binary to assembler to compiled to interpreted to generated)... Other times I think about how we've grown used to buggier and buggier code and yet we press on. 0: https://pncnmnp.github.io/blogs/translating-dijakstra.html https://pncnmnp.github.io/blogs/translating-dijakstra.html
- simonw 3y agoIt's a risk. Similar to the risk that if you accept PRs from coworkers without reviewing them they might have copied and pasted in some vulnerable code from somewhere. Using LLMs means investing more effort in code review. I think that's a worthwhile trade.
- jprete 3y agoThis is not a worthwhile trade. The person sending the PR should not send LLM code that they can't vouch for, under the expectation that reviewers will find any vulnerabilities. That's just dumping the work onto the reviewers.
- simonw 3y agoIt's a worthwhile trade. I've been able to produce a substantial amount of reliable, working code with the assistance of LLMs over the past year - code I would have not been able to produce otherwise simply due to lack of time. Why review code at all if you think your coworkers are infallible?
- dartos 3y agoI’m fine with LLM code as long as someone actually understands what it’s doing.
- bee_rider 3y agoI think you are both saying that it is OK to use LLMs, but you have to check the output. It looks like there’s been a communication hiccup or something; I think you are saying that the LLM user should treat LLM code as if it is written by an unreliable team-mate who might copy-paste from the internet, and check it. Jprete seems to be talking about just receiving a PR from a person who didn’t do that checking and just directly is using the LLM code. I agree with you, but I think it is worth noting that > This is not a worthwhile trade > It's a worthwhile trade The difference here is not in whether or not the trade is worthwhile; you are just talking about two different trades .
- mgii 3y agoThis is more likely than one would think, given such a large amount of samples as detected in this campaign. But there are at least 2 main barriers of an actual incident: 1. Internal instructions telling the generator to avoid exactly that. We wouldn't want to rely on this alone though. 2. Due to LLMs nature, it's unlikely that such generated malicious code would repeat addresses of actual malicious actors. This still leaves a variety of attack vectors such as bind shell, dos, on-site exfiltration, and more.
- dns_snek 3y agoI'm less worried about backdoors accidentally appearing in LLM output and more worried about backdoors being placed into LLM output by 3 letter agencies. Maybe not today, but certainly in a few years time.
- lb4r 3y agoWouldn't it be easier (since they probably have very skilled programmers working for them) and way, way more effective to just set up a team and create a quality open source project with one or two extremely stealthy backdoors? Or just pay or threaten a struggling company or dev to insert them?
- underscoring 3y agoits all about ease. easier to clone and infect existing ones. what you are describing might be effective but would be orders of magnitude more time consuming. cloning and infecting provides 100x more opportunities because these are already popular repos as to paying or coercing someone, again it costs time and money. far easier to just abuse this loophole
- gitaarik 3y agoHow would you secretly hide something like that in FOSS? And why would that be easier? It's seems to me that it's easier to inject into an existing company than to do all the work yourself. This is what they do with most things as I understand.
- Lt_Riza_Hawkeye 3y agoThe heartbleed vulnerability was hidden in plain sight for the better part of a decade, no?
- gitaarik 3y agoYes, but that was a memory leak, giving access to unauthorized random memory. That is not an intentionally created exploit / backdoor which gives the owner easy access to the victim's system.
- bluGill 3y agoGiven the code that I've seen LLMs write so far I'm not too worried for now. They are very useful to write a lot of boilerplate code quick so I use them, but they also tend to write the wrong code often and so when I use them the code is well reviewed. Of course if this is underhanded code (not to be confused with obfuscated - I won't accept obfuscated code from LLMs) I might miss things.
- instagib 3y agoI just posted about an llm issue referring to hijacking the huggingface conversion bot for safetensors. https://news.ycombinator.com/item?id=39549482 https://news.ycombinator.com/item?id=39549482 “we show how an attacker could compromise the Hugging Face Safetensors conversion space and its associated service bot. These comprise a popular service on the site dedicated to converting insecure machine learning models within their ecosystem into safer versions.”
- zitterbewegung 3y agoDatasets will probably move toward a curated datasets instead of scraping everything from the Internet. Also you could add a tool that would have the purpose of identifying malware and reject the output like using virustotal
- thriftwy 3y agoWhy not just ask LLM whether it thinks the snippet is kooky, before adding it into LLM training set? You don't need tools in the age of AI, just ass an AI pipeline step.
- cchance 3y agoDoubtful like they said, while it's a 100,000 the fact is that thats still a drop in the swimming pool that is github, thats why it's lasting so long, because even though the number seems big when it comes to github its a relatively small number, which seems insane, and if a LLM is training it'll end up getting averaged out for the most part by the vast other swaths of code.