7 ms·
Cybercriminals pose as "helpful" Stack Overflow users to push malware
- entuno 2y agoCertainly not a new thing; but a good reminder to be very careful when copy and pasting code from Stackoverflow. Surprised that they'd not using a load of bots to upvote the malware-laden answers to make them seem more legitimate.
- dgellow 2y agoI guess that would increase scrutiny too?
- entuno 2y agoAny unusual voting pattern would certainly make it more likely that Stackoverflow noticed and investigated. But I wonder how many people are likely to try downloading a library they've never heard of from a post with zero upvotes? Of course, once an LLM scrapes that answer and regurgitates it, then you'd never know.
- ChrisMarshallNY 2y agoI’ve found that LLMs develop hallucinations from questionable answers. Since Swift allows extensions of fundamental types, these hallucinations can be quite convincing.
- vertis 2y agoA truly patient malware author wouldn't even necessarily need to make it obvious. You could upvote something over the course of six months and still have a very negative impact. Hell, you could create a library, not have malware in it, answer the questions, wait six months, add the malware. Oh wait, I'm not a criminal. Forget I said that. Patience is unlikely for individuals, but it becomes a real concern when you come to the point of state-based actors. I regularly use Python libraries in my work that emulate Excel. and they're so often out of date, they're so abandoned/unloved for the most part because the people that write them eventually move on and they don't have a big enough user base. That means that they're ripe for somebody inserting stuff. Worse Excel is usually used by big companies, so it's a target as well.
- ChrisMarshallNY 2y agoUgh, but not particularly new. Users have been pushing their work in answers for years. Not a surprise that some are malicious. In my case, I never bother looking at answers that prescribe a dependency. I find that it’s very important that I completely understand any code that I use from SO (or any other source), and I usually change the code to better fit my specific use case (and it’s usually a pretty small code sample).
- ChrisRR 2y agoI suppose this is the one time where stackoverflow being a cesspool is actually useful. Any attempt to post malware would just be marked as duplicate and closed instantly
- entuno 2y agoI was refraining from making a comment that these would be easy to spot because they were actually being helpful and (apparently) answering the question..
- tux3 2y agoOnly questions are closed as duplicates. Stackoverflow is strict and pedantic to the point of being very unfriendly to newbies, but it doesn't close things at random. If you want to criticize Stackoverflow, there's plenty of solid ground to do that. No need to say things that aren't.
- chx 2y ago> Stackoverflow is strict and pedantic to the point of being very unfriendly to newbies at what point do you stop being a newbie? I have been on SO for 14 years and have found "lately" or the last five or so years to be extraordinarily unfriendly. I am in the middle of solving a complex problem so I posit a question but it's stripped to bare bones to make it easier to answer -- and the answer is almost always is not what I seek but instead "why are you doing this". Aaaaargh. I would need to post War & Peace to make you understand why so I just go and delete the question when this happens which is too often. Example: https://stackoverflow.com/q/77202800/308851 https://stackoverflow.com/q/77202800/308851 lots of comments asking why I am doing this were deleted but downvote remains. I kept this one up for whatever reason although downvotes usually are enough to make me delete a question. My advice: if you can't help then stay away from the question. Alas, this likely won't reach the hopped-on-SO-power idiots but it's worth a try.
- croemer 2y agoMy strategy is to wait and see. Initially downvoted questions can become quite popular. The people doing the downvoting on new questions are a minority. Majority of traffic and votes comes over months from search engines.
- throwaway_serp 2y agoThat becomes really fun when gpt takes knowledge from stackoverflow
- belinder 2y agoYes, I would be a lot more concerned about that than people finding stuff on stackoverflow. I can see copilot or some other gpt "suggest" stuff like this soon
- bheadmaster 2y agoThe root of the problem is allowing pip to execute arbitrary code when installing a package from PyPI (default package index), combined with the complete lack of vetting of PyPI packages. Even something as simple as mistyping a package name can cause malicious code to be executed on your computer. Installing packages (i.e. source code) for programming languages should not execute arbitrary code.
- throwaway290 2y agoSame is true about NPM and post-install scripts. Actually it is even worse because JS is more popular and average JS dependency tree is so much more massive. Any tiny forgotten transitive dependency of a dependency (or dev dependency) 15 layers deep can pull this off. Total leftpadization is not a thing on pypi due to different culture
- mrspuratic 2y ago"curl ... | bash" makes me run away from a project. There is the reductive argument that basically everything you download is arbitrary code, but throwing away the code that is run seems uniquely silly. https://news.ycombinator.com/item?id=21490151 https://news.ycombinator.com/item?id=21490151
- avz 2y agoAgreed. Perhaps the most prominent example of this shameful behavior is this one: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs https://sh.rustup.rs | sh Source: https://www.rust-lang.org/tools/install https://www.rust-lang.org/tools/install
- hulitu 2y agoHaha. This is the most secure part. When you start compiling, rust will happily download random "crates" and include them in your program, because "dependencies".
- chowells 2y ago"curl | bash" is the exact same threat vector as "download this compiled installer". I wonder why that doesn't result in the same hyperbolic responses.
- braiamp 2y agoI along with others, years ago, have said that answers that don't address the question asked should be removed. In this case, malicious actors are using packages that are not related to the problem presented in the question. This will become more prevalent than package squatting.
- xattt 2y agoParticularly if LLMs are being trained on this material.
- entuno 2y agoI wonder if there's any weighting done based on things like upvotes in their training sets, or if the consider all answers equally?
- djhn 2y agoI thought that was the entire ”value” of Twitter, Reddit, SO and other such platforms for LLM training? It’s not like we haven’t got higher quality corpora, on average. They’re just poorly annotated.
- entuno 2y agoYou could probably create a more convincing version of this with a bit of collusion between multiple accounts on StackOverflow. If you have one person asking a deliberate question, someone else answering with a backdoored package that does actually does solve the problem, then a few accounts upvoting and adding comments it would lot a more more convincing than just a random answer with zero upvotes that doesn't actually solve the problem.
- colejohnson66 2y agoStackExchange is really good at detecting voting rings, so this won't work.
- Sebb767 2y agoBut you would also need someone to search for and find that answer in order to actually deliver you payload. Answering random questions has the advantage that the original poster is incentivized to execute your code.
- mbil 2y agoHow about scrape SO to find new, popular questions; have an LLM write a convincing solution; tack on “you could also try pip install notmalware, which is much faster”; automatically post that poisoned answer. Extra credit: have the infected user’s computer also start posting these answers to SO
- arp242 2y agoThis would very likely set up one of the voting ring detection mechanisms. It's not so easy to manipulate voting on Stack Overflow – quite a lot of work goes on behind the scenes to stop this. I'm not saying it's impossible, but it's not easy either and requires significant effort. Combined with that your malicious package is also quite likely to be detected fairly quickly(-ish), I'm going to say that it's probably not that effective of a mechanism.
- skilled 2y agoSlightly unrelated… This reads like yet another major attack vector through LLMs. A threat actor only needs to provide a few mentions of the package on reputable sites and it is almost guaranteed that your backdoored package will be mentioned by the LLM once it is retrained with new data, or even in cases like RAG. I am guessing I am not the first to think of this though. Wouldn’t be surprised if this kind of attack vector is already being set in motion for all kinds of other purposes; product reviews, etc.
- vertis 2y agoIt is an important consideration for live language models. The example recently of a bad Reddit answer popping up in Gemini -- putting glue on the pizza demonstrates how easy it is. While there are definitely ethical issues with the data that they are using. Trainers need to get a handle on this kind of thing because for LLMs to be truly useful they have to consume large parts of the web.
- nottorp 2y agoPerhaps a ... human curated index ... would be more useful then? :) I mean, if you're manually curating what you feed to the LLM you end up doing one of the web directories of old and might as well skip the LLM part... or use it just to gain funding...
- venusenvy47 2y agoDo virus scanners know how to interpret base64 encoding when they scan a file? I assumed they would be able to catch something like this. Or wouldn't the OS catch this .exe file when it is being downloaded?
- darkclouds 2y ago[dead]
- nextworddev 2y agoThis will be unfortunately a canonical problem that LLMs can amplify
- RollAHardSix 2y agoSo much wasted effort. Just create an account on Upwork, play by Upworks rules & get complete unferreted access to any number of servers and websites run by small businesses and marketing consultancies. Sure you might have to do a bit of Wordpress dev to keep the illusion going but...there you go.
- allkindsof 2y agoWould be pretty easy to manipulate a real question and be less obvious by solving the problem, but then the malicious user also embeds a bad package, within their answer. Solving the problem, so the real user would make it the selected working answer, which would help to cascade the malicious package for others that come to solve the same issue. E.g. Here's a simple solution to the problem: ’’’ from setuptools import setup from badtoolkit import setupnotmslicious setupnotmslicious() setup { Name = "custom_log", Version = "0.1" # rest of your setup } ’’’
- apantel 2y agoThis is much bigger than this particular issue. This is the same as the Xz backdoor: supply chain attack. It’s sad, but the fact is you cannot actually trust anyone else’s code unless you audit it yourself. Even something seemingly-well-vetted like Xz can be a vehicle for malware.
- humanlity 2y agoSo, the real question is: where is the real helpful person?
- ashishbijlani 2y agoPlug: I’ve been building Packj [1] to detect malicious PyPI/NPM/Ruby/PHP/etc. dependencies using behavioral analysis. It uses static+dynamic code analysis to scan for indicators of compromise (e.g., spawning of shell, use of SSH keys, network communication, use of decode+eval, etc). It also checks for several metadata attributes to detect bad actors (e.g., typo squatting). 1. https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj
- 6c696e7578 2y agoWasn't there an LLM that was slurping SO? Maybe that can be trained to flag abuse at the same time?
- positr0n 2y agoHaha, my initial thought was: Wasn't there an LLM that was slurping SO? Maybe that will start suggesting malware to users.