15 ms·

50% of new NPM packages are spam

520 points by miohtama 4y ago

miohtama 4y ago
Spam problems can be solved by - Cross-Internet reputation system for accounts - Small fee on submission
- throwaway290 4y ago
  NPM for companies costs enough that it surely covers all the reviews already.
  - ehutch79 4y ago
    It’s not about revenue, it’s about making spam unprofitable. Charging 0.25$usd is enough to make spam not worth it. It also attaches an identity to the posting.
    - JosephRedfern 4y ago
      I think the suggestion was that the revenue generated by NPM's commercial dealings should cover any cost associated with a review process for OSS submissions (which in itself would make such spam repositories ineffective)
      - ehutch79 4y ago
        So, let the spam happen, and remove it after the fact using humans? Or hold all submissions until a human reviews it?
        JosephRedfern 4y ago
        I'm just clarifying my interpretation of OP's comment, not necessarily agreeing with it. Anyway, involving humans (funded by NPM's commercial revenue) doesn't reduce the options to "letting the spam happen and dealing it after the fact" or "holding all submissions until a human reviews [them]". If I was trying to solve this problem, I'd be open to a solution that tried to automatically classify submissions as either legitimate or spammy, with an associated confidence level. If the confidence level fell below a given threshold then I'd involve a human.
        throwaway290 4y ago
        Yes, the first one. Not exclusively with using humans to develop better detection of spam.
- mr90210 4y ago
  I don’t see anybody paying to submit a package to a registry. Even if NPM didn’t support other registries or direct installs from a versioning system, different tools would have been created by the community.
- gregoriol 4y ago
  Small fee to submit open-source packages?
  - pipo234 4y ago
    Ie.: Open source = free as in speech, and cheap as in beer ;-)
    - verdverm 4y ago
      sounds like twitter's new strategy
      - nailer 4y ago
        It's a good strategy. Suddenly spam costs money.
        verdverm 4y ago
        Does spam costing more money stop spam? Does it cost money per account, project, version? If I can make $100 from one victim, is this spam still profitable? What happens to the international developers who cannot easily get a payment method setup? Does a $10/m "identity verification" stop a nation state from using the platform to influence?
        pipo234 4y ago
        Fair argument. The spam fee should be proportional to the amount of spam rather than account or fixed monthly fee. Some kind of micropayment per spam item. Per repo, or per commit / per release in case of NPM? Micropayments on internet have always proven difficult to implement. No silver bullets.
  - blowski 4y ago
    When there's so much spam, there's a cost to everyone.
  - 4y ago
- pipo234 4y ago
  > - Cross-Internet reputation system for accounts Gets rid of anonymous spam. > - Small fee on submission Gets rid of amateur spam. I guess that's 98% of the problem. I think this is a good start. What to do about bogus projects sponsored by wealthy companies? What about abandonware? And how do we remain open and inclusive to newbees?
  - justinclift 4y ago
    > What to do about bogus projects sponsored by wealthy companies? Does this happen in the real world, rather than as a theoretical concern? As a thought, when a problem is pressing then sometimes it's best to start with a reasonable action then course correct over time. Rather than doing nothing waiting for a perfect solution.
    - mnw21cam 4y ago
      > Does this happen in the real world, rather than be a theoretical concern? Heck yes. 99% of the stuff advertised to me in big money advertising campaigns is stuff that I will never want. If that doesn't count as "bogus projects sponsored by wealthy companies" then I don't know what does.
      - mindcandy 4y ago
        In the real world, do wealthy companies want to be named on this list of 3 or 4 groups spamming NPM? That’s a lot different than being seen buying a banner ad.
        pipo234 4y ago
        That's probably where the reputation part would come in - fair enough. Still, a large wealthy company might consider creating an "unaffiliated" front company to act on their behalf. For example, take out a legit open source competitor by having the front publish a mediocre bogus project with very similar name. Or paying a small fee to bundle malware into a legit FOSS. So, similar to Twitter's blue check mark - Yes, asking a "small fee" adds friction, but it's not an obstacle to the wealthy.
- newswasboring 4y ago
  > Small fee on submission This will immediately bias the submissions only coming in from the west. Remember you can make the fee small but sometimes a person can't even pay even if they have the money. I remember having the 1000 or so rupees required for some VPS stuff when I was a teenager and not being able to pay since I didn't have a credit card. I hope we don't ever make money a barrier to open source.
  - duxup 4y ago
    Are people who submit to NPM really that short on cash? I doubt it.
    - cacois 4y ago
      Please thoroughly read the comment you are replying to.
    - Raed667 4y ago
      A lot of counties (like mine) don't have access to global payments. Having a card in Euro or USD requires special paperwork.
      - traveler01 4y ago
        From my experience cryptocurrencies are helping countries overcome that. But still, money wouldn't be a solution for this issue.
        mschuster91 4y ago
        You'd have to get cryptocoins in the first place, and there are countries which ban all kinds of cryptocurrency (China) or place it behind onerous KYC requirements (EU, US).
      - duxup 4y ago
        Do you mind if I ask what country?
- madsbuch 4y ago
  The fee structure is the most interesting in my perspective. It would be interesting to see how an open source platform could combat this with a gas-fee structure using their own token economy. You'd need to get tokens to submit, which you can get by donations by engaging with a community, or by buying them, etc.
- ndespres 4y ago
  This old gem comes to mind in any conversation about how to alleviate spam problems: https://craphound.com/spamsolutions.txt https://craphound.com/spamsolutions.txt
- wruza 4y ago
  Captcha is an alternative to small fee, cause solving it automatically costs money. Real fee will scare away almost all amateur developers and almost all professional developers who don’t already have a business account available.
  - bkanber 4y ago
    I run a big web property with great SEO ranking, and captcha definitely does not deter the spam. A lot of this spam is posted by actual humans.
    - wruza 4y ago
      Clearly, captcha only works in the same budget category. If that spam “business” endures hiring humans, it will easily swallow small fees too.
- marban 4y ago
  Do you have an example for cross-community reviews?
  - acomjean 4y ago
    Essentially this is what academic journals are doing. Every paper should be reviewed manually. Of course that costs some money (although the reviewers aren’t paid).
    - verdverm 4y ago
      And they often barely validate the actual work, many times passing it on to subordinates.
  - wruza 4y ago
    Stackoverflow does that. As a regular user you can contribute by reviewing q&as from a special queue, it’s next to your username+score div. https://www.google.com/search?q=stackoverflow+review+queue&tbm=isch https://www.google.com/search?q=stackoverflow+review+queue&t...
- hnbad 4y ago
  - Abolishing capitalism Wait, I may have been overdoing it with root cause analysis again.
- rk06 4y ago
  A small fee on package creation/account creation would be better.
- pjc50 4y ago
  The fees are in the wrong direction. Submitters of good npm packages should be paid. (The overall lack of quality control on npm is a separate question)
- thenerdhead 4y ago
  Package managers have a hard enough time managing packages. Imagine them having to manage payments...
amrb 4y ago
Was the joke creating JavaScript frameworks is way to get promoted in FANG / MANGA..
- jgerrish 4y ago
  I related this "story" elsewhere, but in another context is good too. So, when I was younger, I used to frequent DEMF, the Detroit Electronic Music Festival. It's now called Movement or whatever. It was a great time. And one of the favorite feel good moments was seeing Grandma Techno there: https://mixmag.net/feature/grandma-techno-shares-her-love https://mixmag.net/feature/grandma-techno-shares-her-love Looking back, this view might have been kind of ageist. It shouldn't be that surprising or weird to have older people there. Elders in the tribe. But at the time it was also just a happy recurring image during the event. We were taught to love this woman through informal network stories and "whispers". Everybody had that friend who pointed her out pridefully, and maybe you became that friend pointing her out to another. I can imagine another world where seeing Grandma Techno dancing among the younglings is creepy. At least make her wear a VR headset when she's within 1000 feet of the festival. It's nothing personal, it's just nature. She has a book now. Whether that's because she wanted that sweet "FAANG" publishing deal or liked contributing to the scene. Well... who cares... both benefit us. Fortunately, we were taught to accept people through informal support networks of friends. Same goes with npm and cargo. Yeah, this story seems disjointed and out-of-place. But it's important. And I'm naive, but I'd rather have an orderly migration to social and technical controls for packaging than drama. And I can still respect the people who want some other solution. Because I'm that fucking old person now.
hulitu 4y ago
> 50% of new NPM packages are spam So all an attacker has to do is to publish an npm package. Wait, this alteady happened.
- blowski 4y ago
  To do what? You also need people to install said package.
- verdverm 4y ago
  spam != malware or rootkit (i.e. attacker in the software sense)
werdnapk 4y ago
So what happens when the spam is detected? Are the packages removed?
- andreimarinescu 4y ago
  I've been monitoring NPMJS for a few months now. Up until Feb 28th they had a very close handle on these packages and removed them in around 8 hours. We're seeing these packages stay online for a lot longer now (and the volume is much higher)
hollowturtle 4y ago
I'm afraid it can get worse. What happens when there will be a proliferation of "looking legit npm packages" thanks to AI, full with ransomware? Currently I can't really figure out a one size fits all solution to that. Any idea?
- wongarsu 4y ago
  One idea that's gaining (marginal) traction in Rust (which really sits in the same boat here) is trusted reviews, where trust is established by a web of trust. You probably have some developers you trust, and they have a different set of people they trust, so you can establish transient trust (that decays as the chain gets longer). The most relevant project for Rust is https://web.crev.dev/rust-reviews/ https://web.crev.dev/rust-reviews/, not sure if anything like this already exists for NPM.
  - Mayzie 4y ago
    Aand we’re back to PGP/GPG.
    - verdverm 4y ago
      https://sigstore.dev https://sigstore.dev (& cosign) seem to be gaining in popularity, ease of setup, and integrations
  - sokoloff 4y ago
    I would find amusement if the solution to the spamming of npm turns out to be a genuinely useful use case for blockchain.
    - lelanthran 4y ago
      > I would find amusement if the solution to the spamming of npm turns out to be a genuinely useful use case for blockchain. I think you can implement a web-of-trust without a blockchain.
sirius87 4y ago
Spammers are possibly trying to take advantage of npmjs.com domain's high Google rank. I found and reported this spam account [1] with links to download movies. They seem to be using npmjs as a free web host with good SEO. [1] https://www.npmjs.com/~aarilzd https://www.npmjs.com/~aarilzd
- Ciantic 4y ago
  If the spammers only want to be indexed, then NPM should disable indexing for major search engines. But still allow it to be indexed other ways, which aren't unearthed on Google search. Other ideas include: do not index new packages before they've garnered enough downloads.
  - leshenka 4y ago
    how do you garner enough downloads without being discoverable by Google?
    - Ciantic 4y ago
      It's a fair question, most JS libraries I've discovered weren't directly accessed with Google -> npmjs.com but instead from the library's own page, GitHub, Hacker News, etc. If I Google a library and end up on npmjs.com I usually just click on a link to the library's repository or home page first. Of course, it would disenfranchise a bit, but what is another option?
    - counttheforks 4y ago
      Not npmjs.org's problem. Most languages their dependency managers don't give away indexed flashy web pages for free either, yet discoverability is usually not a problem.
      - chatmasta 4y ago
        Which languages have dependency managers with a public registry that is not indexed in Google? pypi.org and docs.rs are both indexed in Google, for example. With docs.rs it's even kind of annoying because often the indexed page is for an outdated version of the package. There's really no reason why the same spammer couldn't target those sites too.
tyingq 4y ago
Searching for the string "down_load_ebook" does unearth a lot of packages. https://www.npmjs.com/search?q=down_load_ebook https://www.npmjs.com/search?q=down_load_ebook About 100k spam packages, with no false positives that I can see.
- flanbiscuit 4y ago
  wow 104,395 packages found So far the oldest package release I've seen was only 7 days go, all authored by uniquely generated name with the same format: Random First Name + Random Last Name + Random 4 numbers Interesting that npm lists 5,219 pages of results but errors at anything past page 2000. https://www.npmjs.com/search?q=down_load_ebook&page=2000&perPage=20 https://www.npmjs.com/search?q=down_load_ebook&page=2000&per...
  - not_your_vase 4y ago
    And very informatively the HTTP error code is "418 - I'm a Teapot" at page 2001. (Though the response body does say "out of bound", so it's not all bad. I guess this amount of fun is allowed.)
    - flanbiscuit 4y ago
      ha! I didn't even think to look at the response I guess they want to spare their server some unnecessary work and figured "who is going to look at more than 2000 pages of results?!", or maybe that's some sort of caching limit.
  - _5hxt 4y ago
    And looks like we're up to 108,702 packages a mere 6 hours later.
- tyingq 4y ago
  Some other patterns that don't have quite as much, but still 100% spam results: https://www.npmjs.com/search?q=zip-mp3-a-lbum https://www.npmjs.com/search?q=zip-mp3-a-lbum https://www.npmjs.com/search?q=do-wnload-available https://www.npmjs.com/search?q=do-wnload-available https://www.npmjs.com/search?q=file-alb-um-zip https://www.npmjs.com/search?q=file-alb-um-zip
wruza 4y ago
Just think of it, there is a real developer who decided to do this. Spam is immoral, but doing that to an open source repository is your personal all time low.
- FredPret 4y ago
  So true. It's truly sad that some people can hold tight to their cynicism even as they build up their technical skills
  - criley2 4y ago
    The people who do this are likely not American or Western European, likely not from a wealthy background, likely don't have access to high end tech jobs, and probably can't even make 5% of what a Facebook or Google employee makes. These people might feel spite and anger towards the western world for the extreme lavish excess that developers enjoy. It's not hard to imagine a world where developers can learn some skills but are locked out participating like we do, and thus decide to weaponize those skills against us for whatever profit they can.
    - themitigating 4y ago
      Being jealous isn't a justification for any action
      - criley2 4y ago
        I think you mean to say that you don't respect actors who justify their actions through "jealousy". In reality, jealously is a fine justification for actions and arguably the most used justification for any action in human history. Hard to think of a historical war that wasn't based on "jealousy", in the end. I kind of feel like your comment is like saying "Being poor isn't an excuse for stealing bread", and while completely and totally true, it really works hard to miss the point.
        4y ago
supriyo-biswas 4y ago
Is this spam not easily mitigated by simple Bayesian approaches and collection of link features by visiting them?
- andreimarinescu 4y ago
  That would probably work. Also not allowing fully anonymous accounts and linking publishers to real identities would also work in my mind.
- loic-sharma 4y ago
  Sure, but removing or unlisting a valid package could break projects. The folks maintaining the package ecosystem need to be careful. Let’s say there’s 10 spam uploads per hour and it takes you 1 second to verify a package is spam and remove it. That’s 30 minutes a week just dealing with spam. While I was on the .NET package manager, we had the on-call engineer handle this thankless chore. Could you detect these packages at upload time? Yes, but spammers will change their patterns once the package ecosystem gets too effective at detecting current patterns. Perhaps machine learning could help, but often times package manager teams are small and don’t have expertise in this area. Regardless, package removals require human review.
  - adql 4y ago
    We're talking about packages that don't even come with code > More than half of all new packages that are currently (29 Mar 2023) being submitted to npm are SEO spam. That is - empty packages, with just a single README file that contains links to various malicious websites. Yeah once you cut the obvious they will get smarter but at least some will leave to look for other easier target. Spammers just try to find something that ranks high in SEO and costs them nothing, if repository stops being that most will leave. Most other package repositories don't have that problem to such degree > unlisting a valid package could break project ... and about packages that most likely are NOT used as dep anywhere > Let’s say there’s 10 spam uploads per hour and it takes you 1 second to verify a package is spam and remove it. That’s 30 minutes a week just dealing with spam. While I was on the .NET package manager, we had the on-call engineer handle this thankless chore. No need. Just add flag button where a package can be flagged for a check. Users will do the flagging for that so at least you won't have too many valid packages to verify > Could you detect these packages at upload time? Yes, but spammers will change their patterns once the package ecosystem gets too effective at detecting current patterns. Perhaps machine learning could help, but often times package manager teams are small and don’t have expertise in this area. With AI I'm afraid it might get awfully close to "newbie user just publishing package full of shit code"
bin_bash 4y ago
Remember this is a Microsoft product. They certainly have the resources to resolve this if they want to.
- crop_rotation 4y ago
  A Microsoft product doesn't mean the full capacity of the company will be devoted to resolve it. In a big company almost all products have to fight very hard for additional resources, they are not given resources just because the company as a whole made tons of profits.
  - loic-sharma 4y ago
    Exactly this. Don’t blame the team as they’re doing the best they can with their limited resources. However, calling out spam on HN will help convince Microsoft’s leadership to invest in this problem :)
  - bin_bash 4y ago
    It sounds like you completely missed the last 4 words of my comment.
- delfinom 4y ago
  Microsoft is pretty hands off when it comes to their acquisitions the last decade. And moreso, this is GitHub's product (they acquired it, not the larger MS org), the GitHub group is still fairly independent of Microsoft. I can imagine GitHub doesn't give a shit as they continue to push people to use the GitHub package registry instead.
  - bin_bash 4y ago
    This is no longer true. We're in a post-copilot world now where GitHub is the star of the show for the entire corporation.
verdverm 4y ago
Is this outcome a point against having centralized registries? Why not go straight to the source code host?
- pmontra 4y ago
  This is basically what I was doing in the 80s and 90s, downloading compressed tarballs from ftp sites and compiling them. It takes quite a longer developer time than the package manager approach. That includes the time to learn which sites you can trust (probably none today) and which dependencies to use (usually listed in the README.) Furthermore there would be a big incentive to use very few libraries: this is both good and bad. Good because there won't be silly one function modules, bad because a dozen of small modules can add significant value to a project in a short time. Having to code them or build a bigger all comprising module is much harder.
  - verdverm 4y ago
    The registry-less dependency management is how Go works today, and doesn't have those problems. It's even less developer time than NPM. 1. No need to spend time publishing, just push a commit 2. No need to `npm i` or edit a file, modules can be inferred from imports because they use FQDN
    - pabs3 4y ago
      Indeed, Golang really got this right.
  - throw_m239339 4y ago
    The problem is that all the popular NPM packages have so much dependencies that you cannot just download a zip file and install the package on your local computer in order for the library to work, you'd need a way to track all the dependencies... PHP (composer) or Java (Maven) are less prone to that issue because a composer package can't have 50 versions of the same dependency, unlike NPM. So even if a composer package has 20 dependencies, it's relatively easy to track down and download all of them. NPM dependency tries are often exponentials, a stupid design decision. Version conflicts should be solved upstream, not by the package manager. But that decision allowed NPM to grow as a business which was eventually bought by Microsoft.
hogu 4y ago
Why doesn’t this happen with GitHub? GitHub also has very good domain authority.
- speedgoose 4y ago
  GitHub probably has a team working on the spam problem. Doesn’t look like anyone cares at NPM.
- thenerdhead 4y ago
  Rate limiting, spam filters, easy ways to report, 2FA requirements, etc. Many package managers are on this path too. https://github.blog/2022-07-26-introducing-even-more-security-enhancements-to-npm/ https://github.blog/2022-07-26-introducing-even-more-securit...
yklcs 4y ago
Since this is an npmjs problem, I wonder if a CAPTCHA requiring the uploader to solve a JS programming problem could work. Something hard for spammers to solve just by googling – writing a function, filling in blank code, etc. This would require the uploader to have at least basic (or intermediate, depending on the difficulty) knowledge in JS. Maybe the generated data could be used to fine tune LLMs.
- loic-sharma 4y ago
  Disallowing automated publishing would prevent CI/CD scenarios. The spammers are creating large amounts of one-off accounts on external login providers like Microsoft Account. I’m sure those have some sort of CAPTCHA.
paradite 4y ago
Funny enough, I used to work on a project that requires publishing a new npm package for each major Xcode version (precompiled swift library). I was doing incremental suffixes for some time until npm blocked our releases after a few versions due to suspected spam. Had to do some Roman numerals to walk around it.
- jrochkind1 4y ago
  Why couldn't that just be different package versions with the same package name?
MR4D 4y ago
The time for curated package repositories has come. The Good Housekeeping Seal of Approval is sorely needed across many languages.
tiffanyh 4y ago
For all complaints against Apple, they’ve done a remarkably good job at their scale to keep spam low in their Apple Store.
- datkam 4y ago
  Too bad they force you to use that store though...
scrose 4y ago
It would be great if Sandworm listed these malicious repos in a text file that could be imported into a blocklist in a service like Pihole. I’m not worried about hitting these URLs but definitely worry about the less tech savvy people in my family stumbling across these accidentally
- VWWHFSfQ 4y ago
  how would pihole block these though
  - scrose 4y ago
    It was too early for me when I posted this :) There were two ideas in mind that were conflated: 1) A list for blocking the subpaths of these packages in npm that could be imported. 2) A list for blocking the malicious URLs in the repos themselves. Ie they mentioned that the repos have malicious URLs that navigate you off the page. This is where something like pihole could come in handy.
seqizz 4y ago
Wait until spammers integrate with AI coding.
bambax 4y ago
> ... SEO spam. That is - empty packages, with just a single README file (...) All the identified spam packages are currently live on npmjs.com How is that possible? It seems it would be trivial to filter out spam based just on the observation above, why is it not done? (I'm (obviously) not familiar with the process of submitting an NPM package, so I'm genuinely curious how this works).
- delfinom 4y ago
  I guess the reason is to avoid punishing newbies for one that are doing baby's first NPM tutorials. The other problem is if you make a rule "reject npm packages with only a single file called README". The spam bots will just add another fake file. This is a race to the bottom and requires far more aggressive fighting.
pookha 4y ago
What I find interesting with NPM is that it pushed the boundries of the Unix philosophy (build a tool to do one thing well) and it turns out that this philosphy can suffer from even small amounts of narcissism, machiavellianism, and psychopathy. people were able to compose and reason about large amounts of complexity with these simple modules but the devil was in the details (literally).
porsager 4y ago
Would be great if npm->github->microsoft partnered up with https://socket.dev https://socket.dev to get a crude filter and take down any obvious malicious/spam packages.
- feross 4y ago
  Speaking as CEO at https://socket.dev https://socket.dev, we’d love to partner with GitHub on an initiative like this.
xwdv 4y ago
IMO this isn’t really a problem as long as you have good package discovery mechanisms in place. You won’t slurp up some spam package with little reviews and no ratings if you’re paying attention. Look at all the spam email we get yet never even open.
lesquivemeau 4y ago
I was expecting this article to be a promotion of their audit tool considering a thread about it was flagged as spam less than two weeks ago[1] Turns out it indeed is. Interesting article nonetheless, but it's quite ironic that it's about spam [1] https://news.ycombinator.com/item?id=35233877 https://news.ycombinator.com/item?id=35233877
- Hnrobert42 4y ago
  Hmm. I found this article informative. I suppose it did mention their service, but only toward the end. Even then, it wasn’t like “Buy now for 50% off!!!” So on balance, I am glad they posted.
  - miohtama 4y ago
    I am not any way affiliated with the company and I did the submission. I do believe that informative blog posts by industry insider should be allowed and it is not bad practice to promote your company. Especially on HackerNews where it is relevant for audience (no conflict of interest with YCombinator funded companies?). Otherwise any SaaS ecosystem could become AWS/Google/Microsoft well known names only. Rules should be also equally applied. E.g. Each GitHub blog post promotes GitHub and thus Microsoft.
  - lesquivemeau 4y ago
    I 100% agree with you on that point
  - dmix 4y ago
    There's nothing wrong with content marketing if the content is quality.
- thenerdhead 4y ago
  This is common in the "security" space. i.e. Dunk on an ecosystem, promote your tool that somehow "makes it better", but ultimately doesn't help the problem. Source: I work on a notable package manager where this happens regularly.
- 4y ago
alkonaut 4y ago
Just raise the barriers for contribution. Can't have completely open systems and simultaneously not have them be cesspools of spam and malicious packages. E.g. either get approval from N contributors of existing highly regarded packages, or pay an administrative fee for publishing new packages.
EVa5I7bHFq9mnYK 4y ago
Anything free will be abused for spam. Make it pay a small fee to add an npm package, and the problem will disappear. The fee may be going to pay for moderation, for example. To make payments frictionless and anonymous, accept cryptocurrency.
- dspillett 4y ago
  > Make it pay a small fee to add an npm package, and the problem will disappear. As will many useful packages because people just won't bother no matter how small the small fee is. For some they simply can't (no access to internation payment systems), for others they simply won't want the extra admin (I know I wouldn't, being lazy^H^H^H^Htime-efficient as I am). A free alternative will spring up, many will move to that, and once it becomes significant enough it'll become a spam target, and we are back where we began except things are a bit more fragmented so less convenient for all. > To make payments frictionless and anonymous, accept cryptocurrency. That still blocks some financially (what if someone can ill afford any currency, crypto or otherwise?) and many on “why should I bother” (I don't have any crypto accounts, I have to learn a new system to pay someone so I can give my stuff away for free?). This also breaks the small fee matter. If the fee is genuinely small enough it is very easy for an effective spammer to socially engineer a few bits of cryptocurrency out of an innocent fool.
  - EVa5I7bHFq9mnYK 4y ago
    Anyone smart enough to create an npm package can afford $1. You can pay with Satoshi Wallet instantly and virtually for free, and it's easier to fund a Satoshi Wallet than to open a bank account with a payment card. Geo and age agnostic etc.
    - dspillett 4y ago
      > Anyone smart enough to create an npm package can afford $1 I would disagree with that. If also posit that while it would put off a fair few bad actors some would be quite happy to spend that, especially if they're not spending their own money. But ignoring that… > You can I can, no doubt. But would I? The being able to afford it and caring enough to are two separate issues. I suspect there are many that would make something useful, package it up so others find it convenient, then see some extra admin and think "you know want? Nah". Onboarding friction is not just a thing in B2C & B2B contexts. (And if anyone thinks "but what about the community?": Good point, maybe I'd wait for someone in "the community" to do the admin and pay the dollar!)
dirkc 4y ago
I feel for the poor person that now has to clean up that mess :( I've been that person in the past and it was no fun
nobody9999 4y ago
There's been lots of discussion about blockchains, webs of trust, trusted reviews, small[0] fees and a host of other ideas to address npm package spam. I'll throw out another one: create an automated testing process for uploaded NPMs, such testing to be performed before allowing the new "package" to be visible to others. If the testing process can't find any code or if it really is a real package, but can't be successfully tested, the upload can be rejected with (or without for obvious spam) an email to the "developer" letting them know their code doesn't work and won't be visible to the world until they fix their bugs. The devil is, of course, in the details. I'm sure there are many edge cases and special circumstances that will likely require manual intervention, but I'd expect that such a solution would cover the vast majority of "spam" packages, with the added benefit of not allowing broken code on the site either. Perhaps (likely even) there are other, better ways to handle this issue, but this idea would, presumably, significantly reduce the spam issue without negatively impacting honest/real developers. Just a crazy thought. [0] "Small" is relative, as a bunch of folks have pointed out.
- lenzm 4y ago
  This seems like an arms race doomed to failure. The spammers can just add Hello World to pass the check. Then the check could be upgraded to look for some non-trivial behavior. Then the spammers will work around that. ... all at increasing costs to the package hosts. And now they have to be arbiters on what counts as trivial functionality.
  - nobody9999 4y ago
    >This seems like an arms race doomed to failure. The spammers can just add Hello World to pass the check. Then the check could be upgraded to look for some non-trivial behavior. Then the spammers will work around that. ... all at increasing costs to the package hosts. And now they have to be arbiters on what counts as trivial functionality. IIUC, most of these spam "packages" don't have any code at all, just a README with links to whatever malicious sites they want folks to visit. As such, don't assume that just because someone uploads a spam package actually knows how to code anything, especially since it appears that such spam packages are uploaded not to scam Node devs, but to use the good reputation of npmjs.com to host their spammy content. Getting rid of that stuff is the low-hanging fruit. And I would not be at all surprised if almost all of these these folks couldn't code anything useful or worthwhile in Node or any other language. It's highly unlikely that most of the folks uploading these spam packages are node devs, or devs of any kind. As such, most of these folks wouldn't be able to participate in an "arms race." And while some tiny fraction of those folks might be an enterprising spammer who writes an actual npm package. The problem with that, of course, is that it's quite likely that it's just a small number of folks who are uploading dozens (hundreds?) of these "packages," forcing them to either reuse the code over and over again (which is fairly easy to spot) or to actually develop new code for each package. And that's way too resource intensive for scammers. If they were folks who had skills, decent work ethic and/or an interest in anything other than running their scams, they wouldn't be posting fake (i.e., just an empty package with a README) packages in an attempt to use npmjs.com to host their crap. I mean, I get it. Perhaps you made the assumption that these folks are actually devs? Since they're using the site -- but IIUC, there's no proof that's the case -- at least for the specific empty packages I referenced above. Edit: Clarified my thoughts.
djitz 4y ago
I’ve noticed quite a jump in PHP composer package spam recently, too. A lot of just ever so slightly renamed popular packages. A couple even had quite a few stars on their repo.
ravenstine 4y ago
When I did a coding boot camp, one of our assignments was to push a package to RubyGems. It didn't matter if the package did anything; just make up a name and publish it. I'm pretty sure this kind of thing was a common practice with other boot camps, and applied to NPM as well. I always despised how this effectively trashes the repository and represents a complete waste of digital space, no matter how insignificant, as well as take up names that could go towards code that is actually useful. I wouldn't be surprised if a significant number of spam NPM packages were these boot camp assignments.
- deleted 4y ago
  [deleted]
- Cthulhu_ 4y ago
  I wish they did reviews, but if half of the NPM packages are spam, that's still 172.000 legitimate NPM packages - per WEEK. That's not feasible to review. Are these new packages or version releases of existing packages as well? I think there's a market for a verified nodejs repository, where every package is reviewed, scanned and approved by a human + a heap of security tools. It wouldn't accept all updates of packages, because the volume would be too high. It would have to be a paid for service though, aimed at enterprises.
  - zokier 4y ago
    "RHEL" model for nodejs? Why not, but finding enough people willing to actually pay for it will probably be difficult
    - BiteCode_dev 4y ago
      In Python, Continuum is making bank with exactly that.
    - bornfreddy 4y ago
      You'd be surprised. 3rd party dependencies are huge risk for enterprise.
  - 4y ago
throwitaway222 4y ago
I like how Java did it. It seemed like there was a guy with a key protecting maven. At least originally, I don't know if the gates are easy to get through now. It seems like minimally owning a github repo is too low a bar, but that's what made it work I guess.
wdb 4y ago
Forked packages of ESM-only to transpile them to Commonjs and publish them as a new package is reasonably common. Or fork a Commonjs package that became a ESM-only package and backport changes to the package.
cutler 4y ago
I always wondered about the npm package output which, according to http://www.modulecounts.com http://www.modulecounts.com , is 51k per day. That's insane when you consider its nearest competitor, PyPI is 250 per day.
cutler 4y ago
CPAN had the right idea - CPAN testers - back in the late 90s. Oh, how advanced we are 25 years later.
synergy20 4y ago
maybe separate the repo into a few groups: main - the well known gold standard popular ones staging - the ones that will be moved to main when good enough experimental - whatever you want to push this is kind of like debian repos
pictur 4y ago
I don't think it's right to blame npm for this issue. The service they offer is very clear. The responsibility of the downloaded packages is the user.
transitivebs 4y ago
Also check out https://socket.dev https://socket.dev who are my goto for stuff like this. They wrote a similar article recently: https://socket.dev/blog/npm-registry-spam-john-wick https://socket.dev/blog/npm-registry-spam-john-wick
lazzlazzlazz 4y ago
If NPM was a crypto system, half of Hacker News would be saying this is a problem with crypto. The truth is that this happens in any mostly permissionless system: email, text messaging, Github, YouTube, etc. Most of the content is garbage. The challenge is that we need to find powerful ways to identify what's real/useful/safe without limiting permissionless innovation.
- bilater 4y ago
  perhaps the solution is an npm verified packages badge (blue checkmark vibes lol)? You could get that with a nominal annual fee like 10 bucks (so solo devs aren't priced out) + review like an Apple app review.
not_enoch_wise 4y ago
Wait - 50% of npm packages AREN'T spam??
apatheticonion 4y ago
So many npm packages have taken the best names and do nothing with them. It's so frustrating to publish anything these days

520 points by miohtama 4y ago

miohtama 4y ago

Spam problems can be solved by - Cross-Internet reputation system for accounts - Small fee on submission

throwaway290 4y ago
NPM for companies costs enough that it surely covers all the reviews already.
- ehutch79 4y ago
  It’s not about revenue, it’s about making spam unprofitable. Charging 0.25$usd is enough to make spam not worth it. It also attaches an identity to the posting.
  - JosephRedfern 4y ago
    I think the suggestion was that the revenue generated by NPM's commercial dealings should cover any cost associated with a review process for OSS submissions (which in itself would make such spam repositories ineffective)
    - ehutch79 4y ago
      So, let the spam happen, and remove it after the fact using humans? Or hold all submissions until a human reviews it?
      - JosephRedfern 4y ago
        I'm just clarifying my interpretation of OP's comment, not necessarily agreeing with it. Anyway, involving humans (funded by NPM's commercial revenue) doesn't reduce the options to "letting the spam happen and dealing it after the fact" or "holding all submissions until a human reviews [them]". If I was trying to solve this problem, I'd be open to a solution that tried to automatically classify submissions as either legitimate or spammy, with an associated confidence level. If the confidence level fell below a given threshold then I'd involve a human.
      - throwaway290 4y ago
        Yes, the first one. Not exclusively with using humans to develop better detection of spam.
mr90210 4y ago
I don’t see anybody paying to submit a package to a registry. Even if NPM didn’t support other registries or direct installs from a versioning system, different tools would have been created by the community.
gregoriol 4y ago
Small fee to submit open-source packages?
- pipo234 4y ago
  Ie.: Open source = free as in speech, and cheap as in beer ;-)
  - verdverm 4y ago
    sounds like twitter's new strategy
    - nailer 4y ago
      It's a good strategy. Suddenly spam costs money.
      - verdverm 4y ago
        Does spam costing more money stop spam? Does it cost money per account, project, version? If I can make $100 from one victim, is this spam still profitable? What happens to the international developers who cannot easily get a payment method setup? Does a $10/m "identity verification" stop a nation state from using the platform to influence?
        pipo234 4y ago
        Fair argument. The spam fee should be proportional to the amount of spam rather than account or fixed monthly fee. Some kind of micropayment per spam item. Per repo, or per commit / per release in case of NPM? Micropayments on internet have always proven difficult to implement. No silver bullets.
- blowski 4y ago
  When there's so much spam, there's a cost to everyone.
- 4y ago
pipo234 4y ago
> - Cross-Internet reputation system for accounts Gets rid of anonymous spam. > - Small fee on submission Gets rid of amateur spam. I guess that's 98% of the problem. I think this is a good start. What to do about bogus projects sponsored by wealthy companies? What about abandonware? And how do we remain open and inclusive to newbees?
- justinclift 4y ago
  > What to do about bogus projects sponsored by wealthy companies? Does this happen in the real world, rather than as a theoretical concern? As a thought, when a problem is pressing then sometimes it's best to start with a reasonable action then course correct over time. Rather than doing nothing waiting for a perfect solution.
  - mnw21cam 4y ago
    > Does this happen in the real world, rather than be a theoretical concern? Heck yes. 99% of the stuff advertised to me in big money advertising campaigns is stuff that I will never want. If that doesn't count as "bogus projects sponsored by wealthy companies" then I don't know what does.
    - mindcandy 4y ago
      In the real world, do wealthy companies want to be named on this list of 3 or 4 groups spamming NPM? That’s a lot different than being seen buying a banner ad.
      - pipo234 4y ago
        That's probably where the reputation part would come in - fair enough. Still, a large wealthy company might consider creating an "unaffiliated" front company to act on their behalf. For example, take out a legit open source competitor by having the front publish a mediocre bogus project with very similar name. Or paying a small fee to bundle malware into a legit FOSS. So, similar to Twitter's blue check mark - Yes, asking a "small fee" adds friction, but it's not an obstacle to the wealthy.
newswasboring 4y ago
> Small fee on submission This will immediately bias the submissions only coming in from the west. Remember you can make the fee small but sometimes a person can't even pay even if they have the money. I remember having the 1000 or so rupees required for some VPS stuff when I was a teenager and not being able to pay since I didn't have a credit card. I hope we don't ever make money a barrier to open source.
- duxup 4y ago
  Are people who submit to NPM really that short on cash? I doubt it.
  - cacois 4y ago
    Please thoroughly read the comment you are replying to.
  - Raed667 4y ago
    A lot of counties (like mine) don't have access to global payments. Having a card in Euro or USD requires special paperwork.
    - traveler01 4y ago
      From my experience cryptocurrencies are helping countries overcome that. But still, money wouldn't be a solution for this issue.
      - mschuster91 4y ago
        You'd have to get cryptocoins in the first place, and there are countries which ban all kinds of cryptocurrency (China) or place it behind onerous KYC requirements (EU, US).
    - duxup 4y ago
      Do you mind if I ask what country?
madsbuch 4y ago
The fee structure is the most interesting in my perspective. It would be interesting to see how an open source platform could combat this with a gas-fee structure using their own token economy. You'd need to get tokens to submit, which you can get by donations by engaging with a community, or by buying them, etc.
ndespres 4y ago
This old gem comes to mind in any conversation about how to alleviate spam problems: https://craphound.com/spamsolutions.txt https://craphound.com/spamsolutions.txt
wruza 4y ago
Captcha is an alternative to small fee, cause solving it automatically costs money. Real fee will scare away almost all amateur developers and almost all professional developers who don’t already have a business account available.
- bkanber 4y ago
  I run a big web property with great SEO ranking, and captcha definitely does not deter the spam. A lot of this spam is posted by actual humans.
  - wruza 4y ago
    Clearly, captcha only works in the same budget category. If that spam “business” endures hiring humans, it will easily swallow small fees too.
marban 4y ago
Do you have an example for cross-community reviews?
- acomjean 4y ago
  Essentially this is what academic journals are doing. Every paper should be reviewed manually. Of course that costs some money (although the reviewers aren’t paid).
  - verdverm 4y ago
    And they often barely validate the actual work, many times passing it on to subordinates.
- wruza 4y ago
  Stackoverflow does that. As a regular user you can contribute by reviewing q&as from a special queue, it’s next to your username+score div. https://www.google.com/search?q=stackoverflow+review+queue&tbm=isch https://www.google.com/search?q=stackoverflow+review+queue&t...
hnbad 4y ago
- Abolishing capitalism Wait, I may have been overdoing it with root cause analysis again.
rk06 4y ago
A small fee on package creation/account creation would be better.
pjc50 4y ago
The fees are in the wrong direction. Submitters of good npm packages should be paid. (The overall lack of quality control on npm is a separate question)
thenerdhead 4y ago
Package managers have a hard enough time managing packages. Imagine them having to manage payments...

amrb 4y ago

Was the joke creating JavaScript frameworks is way to get promoted in FANG / MANGA..

jgerrish 4y ago
I related this "story" elsewhere, but in another context is good too. So, when I was younger, I used to frequent DEMF, the Detroit Electronic Music Festival. It's now called Movement or whatever. It was a great time. And one of the favorite feel good moments was seeing Grandma Techno there: https://mixmag.net/feature/grandma-techno-shares-her-love https://mixmag.net/feature/grandma-techno-shares-her-love Looking back, this view might have been kind of ageist. It shouldn't be that surprising or weird to have older people there. Elders in the tribe. But at the time it was also just a happy recurring image during the event. We were taught to love this woman through informal network stories and "whispers". Everybody had that friend who pointed her out pridefully, and maybe you became that friend pointing her out to another. I can imagine another world where seeing Grandma Techno dancing among the younglings is creepy. At least make her wear a VR headset when she's within 1000 feet of the festival. It's nothing personal, it's just nature. She has a book now. Whether that's because she wanted that sweet "FAANG" publishing deal or liked contributing to the scene. Well... who cares... both benefit us. Fortunately, we were taught to accept people through informal support networks of friends. Same goes with npm and cargo. Yeah, this story seems disjointed and out-of-place. But it's important. And I'm naive, but I'd rather have an orderly migration to social and technical controls for packaging than drama. And I can still respect the people who want some other solution. Because I'm that fucking old person now.

hulitu 4y ago

> 50% of new NPM packages are spam So all an attacker has to do is to publish an npm package. Wait, this alteady happened.

blowski 4y ago
To do what? You also need people to install said package.
verdverm 4y ago
spam != malware or rootkit (i.e. attacker in the software sense)

werdnapk 4y ago

So what happens when the spam is detected? Are the packages removed?

andreimarinescu 4y ago
I've been monitoring NPMJS for a few months now. Up until Feb 28th they had a very close handle on these packages and removed them in around 8 hours. We're seeing these packages stay online for a lot longer now (and the volume is much higher)

hollowturtle 4y ago

I'm afraid it can get worse. What happens when there will be a proliferation of "looking legit npm packages" thanks to AI, full with ransomware? Currently I can't really figure out a one size fits all solution to that. Any idea?

wongarsu 4y ago
One idea that's gaining (marginal) traction in Rust (which really sits in the same boat here) is trusted reviews, where trust is established by a web of trust. You probably have some developers you trust, and they have a different set of people they trust, so you can establish transient trust (that decays as the chain gets longer). The most relevant project for Rust is https://web.crev.dev/rust-reviews/ https://web.crev.dev/rust-reviews/, not sure if anything like this already exists for NPM.
- Mayzie 4y ago
  Aand we’re back to PGP/GPG.
  - verdverm 4y ago
    https://sigstore.dev https://sigstore.dev (& cosign) seem to be gaining in popularity, ease of setup, and integrations
- sokoloff 4y ago
  I would find amusement if the solution to the spamming of npm turns out to be a genuinely useful use case for blockchain.
  - lelanthran 4y ago
    > I would find amusement if the solution to the spamming of npm turns out to be a genuinely useful use case for blockchain. I think you can implement a web-of-trust without a blockchain.

sirius87 4y ago

Spammers are possibly trying to take advantage of npmjs.com domain's high Google rank. I found and reported this spam account [1] with links to download movies. They seem to be using npmjs as a free web host with good SEO. [1] https://www.npmjs.com/~aarilzd https://www.npmjs.com/~aarilzd

Ciantic 4y ago
If the spammers only want to be indexed, then NPM should disable indexing for major search engines. But still allow it to be indexed other ways, which aren't unearthed on Google search. Other ideas include: do not index new packages before they've garnered enough downloads.
- leshenka 4y ago
  how do you garner enough downloads without being discoverable by Google?
  - Ciantic 4y ago
    It's a fair question, most JS libraries I've discovered weren't directly accessed with Google -> npmjs.com but instead from the library's own page, GitHub, Hacker News, etc. If I Google a library and end up on npmjs.com I usually just click on a link to the library's repository or home page first. Of course, it would disenfranchise a bit, but what is another option?
  - counttheforks 4y ago
    Not npmjs.org's problem. Most languages their dependency managers don't give away indexed flashy web pages for free either, yet discoverability is usually not a problem.
    - chatmasta 4y ago
      Which languages have dependency managers with a public registry that is not indexed in Google? pypi.org and docs.rs are both indexed in Google, for example. With docs.rs it's even kind of annoying because often the indexed page is for an outdated version of the package. There's really no reason why the same spammer couldn't target those sites too.

tyingq 4y ago

Searching for the string "down_load_ebook" does unearth a lot of packages. https://www.npmjs.com/search?q=down_load_ebook https://www.npmjs.com/search?q=down_load_ebook About 100k spam packages, with no false positives that I can see.

flanbiscuit 4y ago
wow 104,395 packages found So far the oldest package release I've seen was only 7 days go, all authored by uniquely generated name with the same format: Random First Name + Random Last Name + Random 4 numbers Interesting that npm lists 5,219 pages of results but errors at anything past page 2000. https://www.npmjs.com/search?q=down_load_ebook&page=2000&perPage=20 https://www.npmjs.com/search?q=down_load_ebook&page=2000&per...
- not_your_vase 4y ago
  And very informatively the HTTP error code is "418 - I'm a Teapot" at page 2001. (Though the response body does say "out of bound", so it's not all bad. I guess this amount of fun is allowed.)
  - flanbiscuit 4y ago
    ha! I didn't even think to look at the response I guess they want to spare their server some unnecessary work and figured "who is going to look at more than 2000 pages of results?!", or maybe that's some sort of caching limit.
- _5hxt 4y ago
  And looks like we're up to 108,702 packages a mere 6 hours later.
tyingq 4y ago
Some other patterns that don't have quite as much, but still 100% spam results: https://www.npmjs.com/search?q=zip-mp3-a-lbum https://www.npmjs.com/search?q=zip-mp3-a-lbum https://www.npmjs.com/search?q=do-wnload-available https://www.npmjs.com/search?q=do-wnload-available https://www.npmjs.com/search?q=file-alb-um-zip https://www.npmjs.com/search?q=file-alb-um-zip

wruza 4y ago

Just think of it, there is a real developer who decided to do this. Spam is immoral, but doing that to an open source repository is your personal all time low.

FredPret 4y ago
So true. It's truly sad that some people can hold tight to their cynicism even as they build up their technical skills
- criley2 4y ago
  The people who do this are likely not American or Western European, likely not from a wealthy background, likely don't have access to high end tech jobs, and probably can't even make 5% of what a Facebook or Google employee makes. These people might feel spite and anger towards the western world for the extreme lavish excess that developers enjoy. It's not hard to imagine a world where developers can learn some skills but are locked out participating like we do, and thus decide to weaponize those skills against us for whatever profit they can.
  - themitigating 4y ago
    Being jealous isn't a justification for any action
    - criley2 4y ago
      I think you mean to say that you don't respect actors who justify their actions through "jealousy". In reality, jealously is a fine justification for actions and arguably the most used justification for any action in human history. Hard to think of a historical war that wasn't based on "jealousy", in the end. I kind of feel like your comment is like saying "Being poor isn't an excuse for stealing bread", and while completely and totally true, it really works hard to miss the point.
      - 4y ago

supriyo-biswas 4y ago

Is this spam not easily mitigated by simple Bayesian approaches and collection of link features by visiting them?

andreimarinescu 4y ago
That would probably work. Also not allowing fully anonymous accounts and linking publishers to real identities would also work in my mind.
loic-sharma 4y ago
Sure, but removing or unlisting a valid package could break projects. The folks maintaining the package ecosystem need to be careful. Let’s say there’s 10 spam uploads per hour and it takes you 1 second to verify a package is spam and remove it. That’s 30 minutes a week just dealing with spam. While I was on the .NET package manager, we had the on-call engineer handle this thankless chore. Could you detect these packages at upload time? Yes, but spammers will change their patterns once the package ecosystem gets too effective at detecting current patterns. Perhaps machine learning could help, but often times package manager teams are small and don’t have expertise in this area. Regardless, package removals require human review.
- adql 4y ago
  We're talking about packages that don't even come with code > More than half of all new packages that are currently (29 Mar 2023) being submitted to npm are SEO spam. That is - empty packages, with just a single README file that contains links to various malicious websites. Yeah once you cut the obvious they will get smarter but at least some will leave to look for other easier target. Spammers just try to find something that ranks high in SEO and costs them nothing, if repository stops being that most will leave. Most other package repositories don't have that problem to such degree > unlisting a valid package could break project ... and about packages that most likely are NOT used as dep anywhere > Let’s say there’s 10 spam uploads per hour and it takes you 1 second to verify a package is spam and remove it. That’s 30 minutes a week just dealing with spam. While I was on the .NET package manager, we had the on-call engineer handle this thankless chore. No need. Just add flag button where a package can be flagged for a check. Users will do the flagging for that so at least you won't have too many valid packages to verify > Could you detect these packages at upload time? Yes, but spammers will change their patterns once the package ecosystem gets too effective at detecting current patterns. Perhaps machine learning could help, but often times package manager teams are small and don’t have expertise in this area. With AI I'm afraid it might get awfully close to "newbie user just publishing package full of shit code"

bin_bash 4y ago

Remember this is a Microsoft product. They certainly have the resources to resolve this if they want to.

crop_rotation 4y ago
A Microsoft product doesn't mean the full capacity of the company will be devoted to resolve it. In a big company almost all products have to fight very hard for additional resources, they are not given resources just because the company as a whole made tons of profits.
- loic-sharma 4y ago
  Exactly this. Don’t blame the team as they’re doing the best they can with their limited resources. However, calling out spam on HN will help convince Microsoft’s leadership to invest in this problem :)
- bin_bash 4y ago
  It sounds like you completely missed the last 4 words of my comment.
delfinom 4y ago
Microsoft is pretty hands off when it comes to their acquisitions the last decade. And moreso, this is GitHub's product (they acquired it, not the larger MS org), the GitHub group is still fairly independent of Microsoft. I can imagine GitHub doesn't give a shit as they continue to push people to use the GitHub package registry instead.
- bin_bash 4y ago
  This is no longer true. We're in a post-copilot world now where GitHub is the star of the show for the entire corporation.

verdverm 4y ago

Is this outcome a point against having centralized registries? Why not go straight to the source code host?

pmontra 4y ago
This is basically what I was doing in the 80s and 90s, downloading compressed tarballs from ftp sites and compiling them. It takes quite a longer developer time than the package manager approach. That includes the time to learn which sites you can trust (probably none today) and which dependencies to use (usually listed in the README.) Furthermore there would be a big incentive to use very few libraries: this is both good and bad. Good because there won't be silly one function modules, bad because a dozen of small modules can add significant value to a project in a short time. Having to code them or build a bigger all comprising module is much harder.
- verdverm 4y ago
  The registry-less dependency management is how Go works today, and doesn't have those problems. It's even less developer time than NPM. 1. No need to spend time publishing, just push a commit 2. No need to `npm i` or edit a file, modules can be inferred from imports because they use FQDN
  - pabs3 4y ago
    Indeed, Golang really got this right.
- throw_m239339 4y ago
  The problem is that all the popular NPM packages have so much dependencies that you cannot just download a zip file and install the package on your local computer in order for the library to work, you'd need a way to track all the dependencies... PHP (composer) or Java (Maven) are less prone to that issue because a composer package can't have 50 versions of the same dependency, unlike NPM. So even if a composer package has 20 dependencies, it's relatively easy to track down and download all of them. NPM dependency tries are often exponentials, a stupid design decision. Version conflicts should be solved upstream, not by the package manager. But that decision allowed NPM to grow as a business which was eventually bought by Microsoft.

hogu 4y ago

Why doesn’t this happen with GitHub? GitHub also has very good domain authority.

speedgoose 4y ago
GitHub probably has a team working on the spam problem. Doesn’t look like anyone cares at NPM.
thenerdhead 4y ago
Rate limiting, spam filters, easy ways to report, 2FA requirements, etc. Many package managers are on this path too. https://github.blog/2022-07-26-introducing-even-more-security-enhancements-to-npm/ https://github.blog/2022-07-26-introducing-even-more-securit...

yklcs 4y ago

Since this is an npmjs problem, I wonder if a CAPTCHA requiring the uploader to solve a JS programming problem could work. Something hard for spammers to solve just by googling – writing a function, filling in blank code, etc. This would require the uploader to have at least basic (or intermediate, depending on the difficulty) knowledge in JS. Maybe the generated data could be used to fine tune LLMs.

loic-sharma 4y ago
Disallowing automated publishing would prevent CI/CD scenarios. The spammers are creating large amounts of one-off accounts on external login providers like Microsoft Account. I’m sure those have some sort of CAPTCHA.

paradite 4y ago

Funny enough, I used to work on a project that requires publishing a new npm package for each major Xcode version (precompiled swift library). I was doing incremental suffixes for some time until npm blocked our releases after a few versions due to suspected spam. Had to do some Roman numerals to walk around it.

jrochkind1 4y ago
Why couldn't that just be different package versions with the same package name?

MR4D 4y ago

The time for curated package repositories has come. The Good Housekeeping Seal of Approval is sorely needed across many languages.

tiffanyh 4y ago

For all complaints against Apple, they’ve done a remarkably good job at their scale to keep spam low in their Apple Store.

datkam 4y ago
Too bad they force you to use that store though...

scrose 4y ago

It would be great if Sandworm listed these malicious repos in a text file that could be imported into a blocklist in a service like Pihole. I’m not worried about hitting these URLs but definitely worry about the less tech savvy people in my family stumbling across these accidentally

VWWHFSfQ 4y ago
how would pihole block these though
- scrose 4y ago
  It was too early for me when I posted this :) There were two ideas in mind that were conflated: 1) A list for blocking the subpaths of these packages in npm that could be imported. 2) A list for blocking the malicious URLs in the repos themselves. Ie they mentioned that the repos have malicious URLs that navigate you off the page. This is where something like pihole could come in handy.

seqizz 4y ago

Wait until spammers integrate with AI coding.

bambax 4y ago

> ... SEO spam. That is - empty packages, with just a single README file (...) All the identified spam packages are currently live on npmjs.com How is that possible? It seems it would be trivial to filter out spam based just on the observation above, why is it not done? (I'm (obviously) not familiar with the process of submitting an NPM package, so I'm genuinely curious how this works).

delfinom 4y ago
I guess the reason is to avoid punishing newbies for one that are doing baby's first NPM tutorials. The other problem is if you make a rule "reject npm packages with only a single file called README". The spam bots will just add another fake file. This is a race to the bottom and requires far more aggressive fighting.

pookha 4y ago

What I find interesting with NPM is that it pushed the boundries of the Unix philosophy (build a tool to do one thing well) and it turns out that this philosphy can suffer from even small amounts of narcissism, machiavellianism, and psychopathy. people were able to compose and reason about large amounts of complexity with these simple modules but the devil was in the details (literally).

porsager 4y ago

Would be great if npm->github->microsoft partnered up with https://socket.dev https://socket.dev to get a crude filter and take down any obvious malicious/spam packages.

feross 4y ago
Speaking as CEO at https://socket.dev https://socket.dev, we’d love to partner with GitHub on an initiative like this.

xwdv 4y ago

IMO this isn’t really a problem as long as you have good package discovery mechanisms in place. You won’t slurp up some spam package with little reviews and no ratings if you’re paying attention. Look at all the spam email we get yet never even open.

lesquivemeau 4y ago

I was expecting this article to be a promotion of their audit tool considering a thread about it was flagged as spam less than two weeks ago[1] Turns out it indeed is. Interesting article nonetheless, but it's quite ironic that it's about spam [1] https://news.ycombinator.com/item?id=35233877 https://news.ycombinator.com/item?id=35233877

Hnrobert42 4y ago
Hmm. I found this article informative. I suppose it did mention their service, but only toward the end. Even then, it wasn’t like “Buy now for 50% off!!!” So on balance, I am glad they posted.
- miohtama 4y ago
  I am not any way affiliated with the company and I did the submission. I do believe that informative blog posts by industry insider should be allowed and it is not bad practice to promote your company. Especially on HackerNews where it is relevant for audience (no conflict of interest with YCombinator funded companies?). Otherwise any SaaS ecosystem could become AWS/Google/Microsoft well known names only. Rules should be also equally applied. E.g. Each GitHub blog post promotes GitHub and thus Microsoft.
- lesquivemeau 4y ago
  I 100% agree with you on that point
- dmix 4y ago
  There's nothing wrong with content marketing if the content is quality.
thenerdhead 4y ago
This is common in the "security" space. i.e. Dunk on an ecosystem, promote your tool that somehow "makes it better", but ultimately doesn't help the problem. Source: I work on a notable package manager where this happens regularly.
4y ago

alkonaut 4y ago

Just raise the barriers for contribution. Can't have completely open systems and simultaneously not have them be cesspools of spam and malicious packages. E.g. either get approval from N contributors of existing highly regarded packages, or pay an administrative fee for publishing new packages.

EVa5I7bHFq9mnYK 4y ago

Anything free will be abused for spam. Make it pay a small fee to add an npm package, and the problem will disappear. The fee may be going to pay for moderation, for example. To make payments frictionless and anonymous, accept cryptocurrency.

dspillett 4y ago
> Make it pay a small fee to add an npm package, and the problem will disappear. As will many useful packages because people just won't bother no matter how small the small fee is. For some they simply can't (no access to internation payment systems), for others they simply won't want the extra admin (I know I wouldn't, being lazy^H^H^H^Htime-efficient as I am). A free alternative will spring up, many will move to that, and once it becomes significant enough it'll become a spam target, and we are back where we began except things are a bit more fragmented so less convenient for all. > To make payments frictionless and anonymous, accept cryptocurrency. That still blocks some financially (what if someone can ill afford any currency, crypto or otherwise?) and many on “why should I bother” (I don't have any crypto accounts, I have to learn a new system to pay someone so I can give my stuff away for free?). This also breaks the small fee matter. If the fee is genuinely small enough it is very easy for an effective spammer to socially engineer a few bits of cryptocurrency out of an innocent fool.
- EVa5I7bHFq9mnYK 4y ago
  Anyone smart enough to create an npm package can afford $1. You can pay with Satoshi Wallet instantly and virtually for free, and it's easier to fund a Satoshi Wallet than to open a bank account with a payment card. Geo and age agnostic etc.
  - dspillett 4y ago
    > Anyone smart enough to create an npm package can afford $1 I would disagree with that. If also posit that while it would put off a fair few bad actors some would be quite happy to spend that, especially if they're not spending their own money. But ignoring that… > You can I can, no doubt. But would I? The being able to afford it and caring enough to are two separate issues. I suspect there are many that would make something useful, package it up so others find it convenient, then see some extra admin and think "you know want? Nah". Onboarding friction is not just a thing in B2C & B2B contexts. (And if anyone thinks "but what about the community?": Good point, maybe I'd wait for someone in "the community" to do the admin and pay the dollar!)

dirkc 4y ago

I feel for the poor person that now has to clean up that mess :( I've been that person in the past and it was no fun

nobody9999 4y ago

There's been lots of discussion about blockchains, webs of trust, trusted reviews, small[0] fees and a host of other ideas to address npm package spam. I'll throw out another one: create an automated testing process for uploaded NPMs, such testing to be performed before allowing the new "package" to be visible to others. If the testing process can't find any code or if it really is a real package, but can't be successfully tested, the upload can be rejected with (or without for obvious spam) an email to the "developer" letting them know their code doesn't work and won't be visible to the world until they fix their bugs. The devil is, of course, in the details. I'm sure there are many edge cases and special circumstances that will likely require manual intervention, but I'd expect that such a solution would cover the vast majority of "spam" packages, with the added benefit of not allowing broken code on the site either. Perhaps (likely even) there are other, better ways to handle this issue, but this idea would, presumably, significantly reduce the spam issue without negatively impacting honest/real developers. Just a crazy thought. [0] "Small" is relative, as a bunch of folks have pointed out.

lenzm 4y ago
This seems like an arms race doomed to failure. The spammers can just add Hello World to pass the check. Then the check could be upgraded to look for some non-trivial behavior. Then the spammers will work around that. ... all at increasing costs to the package hosts. And now they have to be arbiters on what counts as trivial functionality.
- nobody9999 4y ago
  >This seems like an arms race doomed to failure. The spammers can just add Hello World to pass the check. Then the check could be upgraded to look for some non-trivial behavior. Then the spammers will work around that. ... all at increasing costs to the package hosts. And now they have to be arbiters on what counts as trivial functionality. IIUC, most of these spam "packages" don't have any code at all, just a README with links to whatever malicious sites they want folks to visit. As such, don't assume that just because someone uploads a spam package actually knows how to code anything, especially since it appears that such spam packages are uploaded not to scam Node devs, but to use the good reputation of npmjs.com to host their spammy content. Getting rid of that stuff is the low-hanging fruit. And I would not be at all surprised if almost all of these these folks couldn't code anything useful or worthwhile in Node or any other language. It's highly unlikely that most of the folks uploading these spam packages are node devs, or devs of any kind. As such, most of these folks wouldn't be able to participate in an "arms race." And while some tiny fraction of those folks might be an enterprising spammer who writes an actual npm package. The problem with that, of course, is that it's quite likely that it's just a small number of folks who are uploading dozens (hundreds?) of these "packages," forcing them to either reuse the code over and over again (which is fairly easy to spot) or to actually develop new code for each package. And that's way too resource intensive for scammers. If they were folks who had skills, decent work ethic and/or an interest in anything other than running their scams, they wouldn't be posting fake (i.e., just an empty package with a README) packages in an attempt to use npmjs.com to host their crap. I mean, I get it. Perhaps you made the assumption that these folks are actually devs? Since they're using the site -- but IIUC, there's no proof that's the case -- at least for the specific empty packages I referenced above. Edit: Clarified my thoughts.

djitz 4y ago

I’ve noticed quite a jump in PHP composer package spam recently, too. A lot of just ever so slightly renamed popular packages. A couple even had quite a few stars on their repo.

ravenstine 4y ago

When I did a coding boot camp, one of our assignments was to push a package to RubyGems. It didn't matter if the package did anything; just make up a name and publish it. I'm pretty sure this kind of thing was a common practice with other boot camps, and applied to NPM as well. I always despised how this effectively trashes the repository and represents a complete waste of digital space, no matter how insignificant, as well as take up names that could go towards code that is actually useful. I wouldn't be surprised if a significant number of spam NPM packages were these boot camp assignments.

deleted 4y ago
[deleted]
Cthulhu_ 4y ago
I wish they did reviews, but if half of the NPM packages are spam, that's still 172.000 legitimate NPM packages - per WEEK. That's not feasible to review. Are these new packages or version releases of existing packages as well? I think there's a market for a verified nodejs repository, where every package is reviewed, scanned and approved by a human + a heap of security tools. It wouldn't accept all updates of packages, because the volume would be too high. It would have to be a paid for service though, aimed at enterprises.
- zokier 4y ago
  "RHEL" model for nodejs? Why not, but finding enough people willing to actually pay for it will probably be difficult
  - BiteCode_dev 4y ago
    In Python, Continuum is making bank with exactly that.
  - bornfreddy 4y ago
    You'd be surprised. 3rd party dependencies are huge risk for enterprise.
- 4y ago

throwitaway222 4y ago

I like how Java did it. It seemed like there was a guy with a key protecting maven. At least originally, I don't know if the gates are easy to get through now. It seems like minimally owning a github repo is too low a bar, but that's what made it work I guess.

wdb 4y ago

Forked packages of ESM-only to transpile them to Commonjs and publish them as a new package is reasonably common. Or fork a Commonjs package that became a ESM-only package and backport changes to the package.

cutler 4y ago

I always wondered about the npm package output which, according to http://www.modulecounts.com http://www.modulecounts.com , is 51k per day. That's insane when you consider its nearest competitor, PyPI is 250 per day.

cutler 4y ago

CPAN had the right idea - CPAN testers - back in the late 90s. Oh, how advanced we are 25 years later.

synergy20 4y ago

maybe separate the repo into a few groups: main - the well known gold standard popular ones staging - the ones that will be moved to main when good enough experimental - whatever you want to push this is kind of like debian repos

pictur 4y ago

I don't think it's right to blame npm for this issue. The service they offer is very clear. The responsibility of the downloaded packages is the user.

transitivebs 4y ago

Also check out https://socket.dev https://socket.dev who are my goto for stuff like this. They wrote a similar article recently: https://socket.dev/blog/npm-registry-spam-john-wick https://socket.dev/blog/npm-registry-spam-john-wick

lazzlazzlazz 4y ago

If NPM was a crypto system, half of Hacker News would be saying this is a problem with crypto. The truth is that this happens in any mostly permissionless system: email, text messaging, Github, YouTube, etc. Most of the content is garbage. The challenge is that we need to find powerful ways to identify what's real/useful/safe without limiting permissionless innovation.

bilater 4y ago
perhaps the solution is an npm verified packages badge (blue checkmark vibes lol)? You could get that with a nominal annual fee like 10 bucks (so solo devs aren't priced out) + review like an Apple app review.

not_enoch_wise 4y ago

Wait - 50% of npm packages AREN'T spam??

apatheticonion 4y ago

So many npm packages have taken the best names and do nothing with them. It's so frustrating to publish anything these days