16 ms·

NPM: 429 Too Many Requests

145 points by kerpele 7y ago

darekkay 7y ago
Check their status page [1] to see the progress: > Investigating - We are currently investigating 403 / 429 errors received by some users. [1] https://status.npmjs.org/ https://status.npmjs.org/
- ORioN63 7y ago
  It seems the issue is resolved at the moment. From the same link > Monitoring - Our content delivery partner has informed us they've implemented a fix. We are monitoring. Feb 17, 13:07 UTC Edit: And in the OP GH thread, people are also mentioning it's back up.
dijit 7y ago
I was (incidentally) just looking for a way of running a npm repository. Apparently it is not so simple as running a web server with a manifest (which is the case for basically every other package manager out there). Is there a reason for that? Is npms approach somehow better? The thing with using web technology for distribution is that it’s easily accessible and, crucially, that it’s cachable in-line.
- giaour 7y ago
  NPM started off as a CouchDB app, and you used to be to keep a local mirror of the full repository by running a copy of CouchDB and setting up one-way replication (not sure if this is still possible).
- gbtw 7y ago
  At my last job we hosted both Maven and NPM repo in Sonatype Nexus. I think artifactory could do this too i think.
cpclermont 7y ago
Some users in the issue are reporting it's fixed.
gmemstr 7y ago
This seems to be slowly clearing up. Regardless, can we talk about the conduct in this GitHub thread? I know every community is different but is it common to have memes and jokes posted this quickly and often in a GitHub issue? It makes it really hard to follow and discourages genuinely useful discussion of workarounds or progress.
- ryuukk_ 7y ago
  that's why it is popular, less corporate, more engaging, welcoming with a fun ambiance, lot of young people when you look at rust repo, everyone feels like they are going to die, just like the C repos, full of boring 60yo people
- lm28469 7y ago
  Some people getting in the workforce in the last few years have troubles making the distinction between work and play contexts. It's extremely visible on github, slack, &c. which are more and more looking like discord / reddit (gifs, memes, random jokes in the middle of serious discussions)
  - matsemann 7y ago
    You say it like it's a bad thing? It's possible to be both professional and have some fun (not talking about the linked GH).
    - vsareto 7y ago
      It should get quarantined into a different channel or server. If not, and you don't like a bunch of shitty meme spam, it's difficult to filter out while keeping your coworkers unblocked.
    - scoutt 7y ago
      I think it's fine to joke, but even if I try, I really have to make an effort to not get angry when someone at work or an external replies with something written like this: "yeah, u know, i could do it better, lol". Edit: to add that I agree with the idea there is a problem with people making a distinction between work and play contexts.
deleted 7y ago
[deleted]
Philomath 7y ago
If this is causing production CI/CD pipelines to fail, this might be a bigger issue than it seems. Has this already happened in the past with npm?
- gmemstr 7y ago
  > if It 100% is, unfortunately. I don't recall this happening in recent history, but it has been the case that 3rd party services have broken CI/CD pipelines and production pushes (e.g pip broke a few weeks ago, and their own pipeline for deploying changes was blocked by the bug).
  - VBprogrammer 7y ago
    It's very easy for these kind of dependancies to creep into the build process. If the worst case cost of not being able to create a new build out-weights the cost of rearchitecting your build process then it's something you should seriously consider. On the plus side it also brings additional benefits like faster builds and resilience against packages being unilaterally removed.
- empath75 7y ago
  You should always run a local mirror/cache of artifact repositories if you’re doing builds at any kind of scale at all.
warpech 7y ago
If this happens in future, you can switch an NPM mirror, such as: - https://open-registry.dev/#get-started https://open-registry.dev/#get-started - https://npm.taobao.org/ https://npm.taobao.org/ WARNING: Research who runs the mirror before putting your trust in it. How to turn on: npm config set registry https://npm.open-registry.dev How to turn off: npm config delete registry
- yellow_lead 7y ago
  I would not recommend to use the Taobao registry though. This is operated by a Chinese company. Aside from the cybersecurity concerns, if it's hosted in China you'll be getting bad latency.
  - lqs469 7y ago
    It's just a mirror service address, the NPM package is the same. Worrying too much about cybersecurity is a bit of a storm in a teacup.
    - warpech 7y ago
      Well, how do you know it's "just" a mirror service, and it is not using a zero-day to exploit your system, by installing a root-kit or copying your code to their servers? I agree that it's a valid concern.
      - lqs469 7y ago
        In fact, you can compare the installed dependencies code line by line, Javascript won't be compiled anyway.
      - gpmcadam 7y ago
        If you're concerned about injection into a third-party package, you should be using `package-lock.json` (or equiv) and integrity hashing your dependencies at install time.
      - wp381640 7y ago
mxschumacher 7y ago
In the last company I worked for, we had a pretty standard build pipeline for a web application. I was a bit shocked to see how many different packages repositories we depend on for every build: NPM, Pypi, Alpine packages, docker hub ... a single failure in any of those centralized systems that we don't pay for and builds fail.
Waterluvian 7y ago
I wonder how many requests to npm are an utter waste given that often dependencies don't change due to the lockfile. In Travis there's some (not too obvious) caching mechanisms that in many cases avoid this and speed builds up a ton. I wonder if we would win from a review on popularity of configuring the cache and to educate people further on its use. I'm sure other CI systems have similar capabilities.
imtringued 7y ago
Honestly, don't depend on central repositories for daily availability. Especially if you are doing CI that redownloads everything from scratch. Use something like artifactory to cache the repository you are using: https://www.jfrog.com/confluence/display/RTF/npm+Registry https://www.jfrog.com/confluence/display/RTF/npm+Registry
- merb 7y ago
  there is also sonatype nexus: https://www.sonatype.com/product-nexus-repository https://www.sonatype.com/product-nexus-repository
  - nurettin 7y ago
    nexus is also free to setup on premise
- wojciechpolak 7y ago
  Yup, artifactory helps a lot. The company where I work is using it and we run a lot of npm ci on a daily basis.
- krab 7y ago
  I think that's the issue of cost/reward. The cost is - N developers can't work for X hours - or the company can't release new versions due to CI dependency on the registry. - or the registry removes a package you were using - or the existing package contents changes to something malicious BUT you pay this price very occasionally and if you're a small shop, the cost is often negligible. On the other hand, maintaining your own mirror has very real costs even though they can be small. One time setup, hardware, sometimes license or hosted service fee, security upgrades. When there's a sponsor maintaining the central repository, having very good uptime and offering it for free, the marginal utility of a local mirror is quite small.
  - naniwaduni 7y ago
    There's no reward. It's a risk/cost tradeoff.
sethammons 7y ago
This is one of the reasons why, in Go, my team vendors our dependencies. For any service that seeks stability and the ability to deploy any time, removing networked build dependencies is important.
- ratata 7y ago
  > vendors our dependencies Is that like a local cache?
  - sethammons 7y ago
    Vendoring is checking in your dependencies with the source. You can def consider that a local cache. The next step up is to run your own proxy server (pretty much a package server / mirror). Next is to use a service like artifacory that does similar.
    - erik_seaberg 7y ago
      We should call it a backup. Calling it a cache puts people in mind of size/hit ratio tradeoffs; what's needed is zero tolerance for loss of mission critical code.
dgellow 7y ago
Yearly reminder to vendor your dependencies.
hannob 7y ago
I guess the solution here is for the larger CI providers to host a mirror of popular package repositories locally and provide an easy way to use them. It really doesn't make a lot of sense to re-download tons of packages for every minor commit where you run your CI.
colonwqbang 7y ago
Is it common to rely on a free service like npm for your company's core business? It seems like you would be taking a huge risk by not mirroring anything you need internally.
- dividedbyzero 7y ago
  I believe (based on lots of anecdata) that it's not just common, it's absolutely overwhelmingly often the case at companies of pretty much every size, be it a data scientist using stuff from CRAN for mission-critical modelling or an OS package repo or the like. It appears that few shops have this fully under control.
- dkersten 7y ago
  Yeah, you should definitely keep a mirror of packages your company is using rather than pulling from some external source.
- jeltz 7y ago
  Yes, it is very common. Setting up local caches for package repositories is rarely prioritized high enough to ever get done by IT or the developers. There is almost always something else which is more important to the business.
- hyperpape 7y ago
  Anecdata time: I'm in a 300 person (~50 dev) company serving the enterprise space (we have SOC audits). All our NPM and Maven needs are handled through a local Artifactory instance.
- jcrawfordor 7y ago
  I strongly recommend keeping a local mirror of your dependencies... however, I've spent years maintaining such mirrors for fairly large projects (incl. Artifactory, Nexus, and one-off OSS setups like Docker registry server), and I think it's easy to underestimate how much work it is. Whether you use expensive 'turnkey' solutions like Artifactory or keep things simple, there's just a surprising number of ways for a local mirror to go wrong, especially if you depend on it for any kind of third-party dependency compliance control. Some repository mirrors will also become very large, which means that if you're e.g. running them in a cloud provider the bill can add up. Not really a problem on local hardware but the up-front cost of hardware can be substantial and a lot of startups have little to no in-house IT capability (e.g. the org I work with right now has reached hundreds of employees without having a single system administrator on staff, so as devops person I end up having to do the care and feeding of our recently purchased local hardware as well). In general I think this is an important and often overlooked issue in modern tech businesses - it is amazing how many technology-centric firms like software startups get to appreciable size relying entirely on outside SaaS/PaaS providers with no real in-house IT operation. This reduces up-front and staffing cost but has a way of coming back to bite you when you hit a certain point. A conversation I've been in before, in reasonably large software outfits, is "we want actual real office phones now, but telephony-as-a-service is real expensive and the on-prem products use scary words like VLAN and QoS in their setup documentation". As someone with an IT rather than software background it's a little baffling to me how this happens, I feel like a combo sysadmin/network engineer would be an early hire. But here I am working for a company instead of running one...
chrissnell 7y ago
I dealt with this problem at a previous job where we had a build pipeline and apps that were very dependent on NPM. To fix it, I used nginx to build a set of two-tiered caching servers within our CI Kubernetes cluster. One used a ramdisk to provide a low latency cache for the NPM client fetches. The second was a disk-backed cache to provide persistence to the ramdisk-backed cache. I had a script that would alter the package deps so that they were pulled from the ramdisk-backed service, which used the disk-backed service as it's backend. The disk-backed service uses actual NPM URLs for its backend. The result was lightning-fast fetches and no rate limiting. We needed the two-tiered system because this was Kube and occasionally we would have to rebuild/restart nodes and we didn't want to completely lose the cache when that happened. You can easily extend this system to handle any package artifacts used in your build process: .deb's, .rpm's, etc.
SirensOfTitan 7y ago
I’ve been considering checking node_modules into source control for some time now, has anyone else done that successfully? There would be a variety of benefits: 1. Eliminate redownload of packages on every CI build 2. Reduce the amount of gigantic IO operations from unpacking the tens-of-thousands of files sitting in node_modules. 3. Better security: code checked in can be audited better if not downloaded every single CI build. yarn’s PnP system is promising for the zero-install paradigm, but it doesn’t seem quite ready yet (so many packages don’t seem to get their dependencies right).
- joeyrobert 7y ago
  Helps if you only have one platform you're developing on and deploying to (e.g. x86-64 Linux). If developing on macOS there can be Mac specific binaries installed, depending on the package.
  - jackewiehose 7y ago
    And if you do have more platforms, why not just check in one node_modules-directory for each? This idea to redownload all packages all the time from external sources (and not even having a fallback-plan) seems completely brain-dead to me. Didn't the people learn from leftpad-gate?
    - LeonidasXIV 7y ago
      > And if you do have more platforms, why not just check in one node_modules-directory for each? Now you have to sync it or risk running into unreproducible build failures. Also, if you update the binary dependencies on say, macOS, then you still need some x86-64 Linux to build the dependency. Not saying it is not possible but without a proper process (e.g. a build server being the only place that updates dependencies) this is going to be painful.
  - andrewl-hn 7y ago
    That's why npm has a command `npm install --ignore-scripts`. It download the dependencies, but doesn't run the postinstall scripts (that either download pre-build binaries or run a compiler locally). In early days of node (circa 2011-2013) we used to do the following: 1. run `npm install --ignore-scripts` first. 2. Check the node_modules folder to source control, 3. run `npm install` again - this time without the flag 4. put all extra files generated by install scripts to .gitignore This way the third-party code (at least, the JS-part of that code) was in the repository, and every developer / server got the version of binaries for their architecture. It wasn't a bullet-proof, though, since: 1. The scripts could do different things anyway 2. More importantly: one could upload a new version of library to npm with the same version number. These days, lockfiles and stricter npm publishing rules largely eliminated both issues, and updating dependencies doesn't produce 10k-line diffs in git history anymore.
clownpenis_fart 7y ago
and the js/npm clown show marches on
- wraithy 7y ago
  Looking at your username, I'm unsure if this is a positive or negative thing.
kuon 7y ago
Isn't mirroring still a thing? I am not saying this as the old creepy guy (well a bit), but seriously wondering why a registry like npm doesn't have tons of geographicaly spreaded mirrors. Package can easily be signed and mirrored, that shouldn't be complicated.
lqs469 7y ago
That's why we need a decentralized network.
rickspencer3 7y ago
I came here to see if anyone had insights about what happened to see if I could apply any lessons learned. But all the comments seem to be "get off my lawn" style comments annoyed about people posting jokes in the issue. Unfortunately, I am unable to resist the urge to add to the noise by complaining about people complaining.
29athrowaway 7y ago
Most of the responses to the ticket are completely irrelevant.
tablethnuser 7y ago
I'm always surprised that npm, a for-profit company with a lame business model, is graciously serving redundant package requests millions of times per day to everyone's CI/CD flows. One day this is going to happen for real and it will be because npm org decided to charge for API requests by `npm ci`.
zovin 7y ago
Here's a postmortem comment from a Cloudflare engineer: https://github.com/npm/cli/issues/836#issuecomment-587019096 https://github.com/npm/cli/issues/836#issuecomment-587019096
- deleted 7y ago
  [deleted]
  - rapunkill 7y ago
    How's your day going?
hkchad 7y ago
1) Glad we have a Nexus proxy in front of npm... [a] 2) Cloudflare strikes again, this one company is at the same time making the internet better and worse. I'm constantly blocked by cloudflare for something that should not be, it's extremely frustrating. Then when you complain to them they throw their hands up and say "owner hasn't configured their site for that". Ugh. [a] https://blog.sonatype.com/using-nexus-3-as-your-repository-part-2-npm-packages https://blog.sonatype.com/using-nexus-3-as-your-repository-p...
- hoppla 7y ago
  Cloudflare must value my ability to identify fire hydrants and crosswalks, because I sure get a lot of these challenges...
keanzu 7y ago
Reliance on npm is one of the things covered in Ryan Dahl's "10 Things I Regret About Node.js" talk at JSConf EU 2018 and is fixed in Deno. Includes a built-in package manager for resource fetching, thus no need for NPM. https://en.wikipedia.org/wiki/Deno_(software) https://en.wikipedia.org/wiki/Deno_(software)
nonbirithm 7y ago
Here's an explanation from CloudFlare as to the root cause. [0] > I am the engineering manager for the DDoS protection team and this morning at 11:06 UTC we tweaked a rule that affected one of our signals. The signal relates to the HTTP referer header, and we have a piece of code that looks at invalid referer headers. In this case we tweaked it to include not just "obvious garbage" but "anything that does not conform to the HTTP specification"... i.e. is the referer a URI? If not then it contributes to knowledge about bad traffic. > So... why did this impact npmjs.org? It turns out that a lot of NPM traffic sends the referer as "install" which is invalid according to the HTTP specification. As NPM is also a heavily trafficked site this resulted in the DDoS systems picking this up and treating the traffic as a HTTP flood and determining that a rate-limit should be applied. > When we noticed that NPM was seeing an increase in HTTP 429s (as seen on Twitter) we contacted NPM and started an internal investigation. As soon as we identified the root cause we reverted the change, which was at 13:00 UTC. > We'll note that NPM and 1 other site use the referer for purposes outside the HTTP spec and we'll update our systems to ensure that this does not happen again. Additionally we'll improve our monitoring around changes of this nature so that we can discover impact sooner and roll back automatically. [0] https://github.com/npm/cli/issues/836#issuecomment-587019096 https://github.com/npm/cli/issues/836#issuecomment-587019096

145 points by kerpele 7y ago

Check their status page [1] to see the progress: > Investigating - We are currently investigating 403 / 429 errors received by some users. [1] https://status.npmjs.org/ https://status.npmjs.org/

ORioN63 7y ago
It seems the issue is resolved at the moment. From the same link > Monitoring - Our content delivery partner has informed us they've implemented a fix. We are monitoring. Feb 17, 13:07 UTC Edit: And in the OP GH thread, people are also mentioning it's back up.

dijit 7y ago

I was (incidentally) just looking for a way of running a npm repository. Apparently it is not so simple as running a web server with a manifest (which is the case for basically every other package manager out there). Is there a reason for that? Is npms approach somehow better? The thing with using web technology for distribution is that it’s easily accessible and, crucially, that it’s cachable in-line.

giaour 7y ago
NPM started off as a CouchDB app, and you used to be to keep a local mirror of the full repository by running a copy of CouchDB and setting up one-way replication (not sure if this is still possible).
gbtw 7y ago
At my last job we hosted both Maven and NPM repo in Sonatype Nexus. I think artifactory could do this too i think.

cpclermont 7y ago

Some users in the issue are reporting it's fixed.

gmemstr 7y ago

This seems to be slowly clearing up. Regardless, can we talk about the conduct in this GitHub thread? I know every community is different but is it common to have memes and jokes posted this quickly and often in a GitHub issue? It makes it really hard to follow and discourages genuinely useful discussion of workarounds or progress.

ryuukk_ 7y ago
that's why it is popular, less corporate, more engaging, welcoming with a fun ambiance, lot of young people when you look at rust repo, everyone feels like they are going to die, just like the C repos, full of boring 60yo people
lm28469 7y ago
Some people getting in the workforce in the last few years have troubles making the distinction between work and play contexts. It's extremely visible on github, slack, &c. which are more and more looking like discord / reddit (gifs, memes, random jokes in the middle of serious discussions)
- matsemann 7y ago
  You say it like it's a bad thing? It's possible to be both professional and have some fun (not talking about the linked GH).
  - vsareto 7y ago
    It should get quarantined into a different channel or server. If not, and you don't like a bunch of shitty meme spam, it's difficult to filter out while keeping your coworkers unblocked.
  - scoutt 7y ago
    I think it's fine to joke, but even if I try, I really have to make an effort to not get angry when someone at work or an external replies with something written like this: "yeah, u know, i could do it better, lol". Edit: to add that I agree with the idea there is a problem with people making a distinction between work and play contexts.

deleted 7y ago

[deleted]

Philomath 7y ago

If this is causing production CI/CD pipelines to fail, this might be a bigger issue than it seems. Has this already happened in the past with npm?

gmemstr 7y ago
> if It 100% is, unfortunately. I don't recall this happening in recent history, but it has been the case that 3rd party services have broken CI/CD pipelines and production pushes (e.g pip broke a few weeks ago, and their own pipeline for deploying changes was blocked by the bug).
- VBprogrammer 7y ago
  It's very easy for these kind of dependancies to creep into the build process. If the worst case cost of not being able to create a new build out-weights the cost of rearchitecting your build process then it's something you should seriously consider. On the plus side it also brings additional benefits like faster builds and resilience against packages being unilaterally removed.
empath75 7y ago
You should always run a local mirror/cache of artifact repositories if you’re doing builds at any kind of scale at all.

warpech 7y ago

If this happens in future, you can switch an NPM mirror, such as: - https://open-registry.dev/#get-started https://open-registry.dev/#get-started - https://npm.taobao.org/ https://npm.taobao.org/ WARNING: Research who runs the mirror before putting your trust in it. How to turn on: npm config set registry https://npm.open-registry.dev How to turn off: npm config delete registry

yellow_lead 7y ago
I would not recommend to use the Taobao registry though. This is operated by a Chinese company. Aside from the cybersecurity concerns, if it's hosted in China you'll be getting bad latency.
- lqs469 7y ago
  It's just a mirror service address, the NPM package is the same. Worrying too much about cybersecurity is a bit of a storm in a teacup.
  - warpech 7y ago
    Well, how do you know it's "just" a mirror service, and it is not using a zero-day to exploit your system, by installing a root-kit or copying your code to their servers? I agree that it's a valid concern.
    - lqs469 7y ago
      In fact, you can compare the installed dependencies code line by line, Javascript won't be compiled anyway.
    - gpmcadam 7y ago
      If you're concerned about injection into a third-party package, you should be using `package-lock.json` (or equiv) and integrity hashing your dependencies at install time.
    - wp381640 7y ago

mxschumacher 7y ago

In the last company I worked for, we had a pretty standard build pipeline for a web application. I was a bit shocked to see how many different packages repositories we depend on for every build: NPM, Pypi, Alpine packages, docker hub ... a single failure in any of those centralized systems that we don't pay for and builds fail.

Waterluvian 7y ago

I wonder how many requests to npm are an utter waste given that often dependencies don't change due to the lockfile. In Travis there's some (not too obvious) caching mechanisms that in many cases avoid this and speed builds up a ton. I wonder if we would win from a review on popularity of configuring the cache and to educate people further on its use. I'm sure other CI systems have similar capabilities.

imtringued 7y ago

Honestly, don't depend on central repositories for daily availability. Especially if you are doing CI that redownloads everything from scratch. Use something like artifactory to cache the repository you are using: https://www.jfrog.com/confluence/display/RTF/npm+Registry https://www.jfrog.com/confluence/display/RTF/npm+Registry

merb 7y ago
there is also sonatype nexus: https://www.sonatype.com/product-nexus-repository https://www.sonatype.com/product-nexus-repository
- nurettin 7y ago
  nexus is also free to setup on premise
wojciechpolak 7y ago
Yup, artifactory helps a lot. The company where I work is using it and we run a lot of npm ci on a daily basis.
krab 7y ago
I think that's the issue of cost/reward. The cost is - N developers can't work for X hours - or the company can't release new versions due to CI dependency on the registry. - or the registry removes a package you were using - or the existing package contents changes to something malicious BUT you pay this price very occasionally and if you're a small shop, the cost is often negligible. On the other hand, maintaining your own mirror has very real costs even though they can be small. One time setup, hardware, sometimes license or hosted service fee, security upgrades. When there's a sponsor maintaining the central repository, having very good uptime and offering it for free, the marginal utility of a local mirror is quite small.
- naniwaduni 7y ago
  There's no reward. It's a risk/cost tradeoff.

sethammons 7y ago

This is one of the reasons why, in Go, my team vendors our dependencies. For any service that seeks stability and the ability to deploy any time, removing networked build dependencies is important.

ratata 7y ago
> vendors our dependencies Is that like a local cache?
- sethammons 7y ago
  Vendoring is checking in your dependencies with the source. You can def consider that a local cache. The next step up is to run your own proxy server (pretty much a package server / mirror). Next is to use a service like artifacory that does similar.
  - erik_seaberg 7y ago
    We should call it a backup. Calling it a cache puts people in mind of size/hit ratio tradeoffs; what's needed is zero tolerance for loss of mission critical code.

dgellow 7y ago

Yearly reminder to vendor your dependencies.

hannob 7y ago

I guess the solution here is for the larger CI providers to host a mirror of popular package repositories locally and provide an easy way to use them. It really doesn't make a lot of sense to re-download tons of packages for every minor commit where you run your CI.

colonwqbang 7y ago

Is it common to rely on a free service like npm for your company's core business? It seems like you would be taking a huge risk by not mirroring anything you need internally.

dividedbyzero 7y ago
I believe (based on lots of anecdata) that it's not just common, it's absolutely overwhelmingly often the case at companies of pretty much every size, be it a data scientist using stuff from CRAN for mission-critical modelling or an OS package repo or the like. It appears that few shops have this fully under control.
dkersten 7y ago
Yeah, you should definitely keep a mirror of packages your company is using rather than pulling from some external source.
jeltz 7y ago
Yes, it is very common. Setting up local caches for package repositories is rarely prioritized high enough to ever get done by IT or the developers. There is almost always something else which is more important to the business.
hyperpape 7y ago
Anecdata time: I'm in a 300 person (~50 dev) company serving the enterprise space (we have SOC audits). All our NPM and Maven needs are handled through a local Artifactory instance.
jcrawfordor 7y ago
I strongly recommend keeping a local mirror of your dependencies... however, I've spent years maintaining such mirrors for fairly large projects (incl. Artifactory, Nexus, and one-off OSS setups like Docker registry server), and I think it's easy to underestimate how much work it is. Whether you use expensive 'turnkey' solutions like Artifactory or keep things simple, there's just a surprising number of ways for a local mirror to go wrong, especially if you depend on it for any kind of third-party dependency compliance control. Some repository mirrors will also become very large, which means that if you're e.g. running them in a cloud provider the bill can add up. Not really a problem on local hardware but the up-front cost of hardware can be substantial and a lot of startups have little to no in-house IT capability (e.g. the org I work with right now has reached hundreds of employees without having a single system administrator on staff, so as devops person I end up having to do the care and feeding of our recently purchased local hardware as well). In general I think this is an important and often overlooked issue in modern tech businesses - it is amazing how many technology-centric firms like software startups get to appreciable size relying entirely on outside SaaS/PaaS providers with no real in-house IT operation. This reduces up-front and staffing cost but has a way of coming back to bite you when you hit a certain point. A conversation I've been in before, in reasonably large software outfits, is "we want actual real office phones now, but telephony-as-a-service is real expensive and the on-prem products use scary words like VLAN and QoS in their setup documentation". As someone with an IT rather than software background it's a little baffling to me how this happens, I feel like a combo sysadmin/network engineer would be an early hire. But here I am working for a company instead of running one...

chrissnell 7y ago

I dealt with this problem at a previous job where we had a build pipeline and apps that were very dependent on NPM. To fix it, I used nginx to build a set of two-tiered caching servers within our CI Kubernetes cluster. One used a ramdisk to provide a low latency cache for the NPM client fetches. The second was a disk-backed cache to provide persistence to the ramdisk-backed cache. I had a script that would alter the package deps so that they were pulled from the ramdisk-backed service, which used the disk-backed service as it's backend. The disk-backed service uses actual NPM URLs for its backend. The result was lightning-fast fetches and no rate limiting. We needed the two-tiered system because this was Kube and occasionally we would have to rebuild/restart nodes and we didn't want to completely lose the cache when that happened. You can easily extend this system to handle any package artifacts used in your build process: .deb's, .rpm's, etc.

SirensOfTitan 7y ago

I’ve been considering checking node_modules into source control for some time now, has anyone else done that successfully? There would be a variety of benefits: 1. Eliminate redownload of packages on every CI build 2. Reduce the amount of gigantic IO operations from unpacking the tens-of-thousands of files sitting in node_modules. 3. Better security: code checked in can be audited better if not downloaded every single CI build. yarn’s PnP system is promising for the zero-install paradigm, but it doesn’t seem quite ready yet (so many packages don’t seem to get their dependencies right).

joeyrobert 7y ago
Helps if you only have one platform you're developing on and deploying to (e.g. x86-64 Linux). If developing on macOS there can be Mac specific binaries installed, depending on the package.
- jackewiehose 7y ago
  And if you do have more platforms, why not just check in one node_modules-directory for each? This idea to redownload all packages all the time from external sources (and not even having a fallback-plan) seems completely brain-dead to me. Didn't the people learn from leftpad-gate?
  - LeonidasXIV 7y ago
    > And if you do have more platforms, why not just check in one node_modules-directory for each? Now you have to sync it or risk running into unreproducible build failures. Also, if you update the binary dependencies on say, macOS, then you still need some x86-64 Linux to build the dependency. Not saying it is not possible but without a proper process (e.g. a build server being the only place that updates dependencies) this is going to be painful.
- andrewl-hn 7y ago
  That's why npm has a command `npm install --ignore-scripts`. It download the dependencies, but doesn't run the postinstall scripts (that either download pre-build binaries or run a compiler locally). In early days of node (circa 2011-2013) we used to do the following: 1. run `npm install --ignore-scripts` first. 2. Check the node_modules folder to source control, 3. run `npm install` again - this time without the flag 4. put all extra files generated by install scripts to .gitignore This way the third-party code (at least, the JS-part of that code) was in the repository, and every developer / server got the version of binaries for their architecture. It wasn't a bullet-proof, though, since: 1. The scripts could do different things anyway 2. More importantly: one could upload a new version of library to npm with the same version number. These days, lockfiles and stricter npm publishing rules largely eliminated both issues, and updating dependencies doesn't produce 10k-line diffs in git history anymore.

clownpenis_fart 7y ago

and the js/npm clown show marches on

wraithy 7y ago
Looking at your username, I'm unsure if this is a positive or negative thing.

kuon 7y ago

Isn't mirroring still a thing? I am not saying this as the old creepy guy (well a bit), but seriously wondering why a registry like npm doesn't have tons of geographicaly spreaded mirrors. Package can easily be signed and mirrored, that shouldn't be complicated.

lqs469 7y ago

That's why we need a decentralized network.

rickspencer3 7y ago

I came here to see if anyone had insights about what happened to see if I could apply any lessons learned. But all the comments seem to be "get off my lawn" style comments annoyed about people posting jokes in the issue. Unfortunately, I am unable to resist the urge to add to the noise by complaining about people complaining.

29athrowaway 7y ago

Most of the responses to the ticket are completely irrelevant.

tablethnuser 7y ago

I'm always surprised that npm, a for-profit company with a lame business model, is graciously serving redundant package requests millions of times per day to everyone's CI/CD flows. One day this is going to happen for real and it will be because npm org decided to charge for API requests by `npm ci`.

zovin 7y ago

Here's a postmortem comment from a Cloudflare engineer: https://github.com/npm/cli/issues/836#issuecomment-587019096 https://github.com/npm/cli/issues/836#issuecomment-587019096

deleted 7y ago
[deleted]
- rapunkill 7y ago
  How's your day going?

hkchad 7y ago

1) Glad we have a Nexus proxy in front of npm... [a] 2) Cloudflare strikes again, this one company is at the same time making the internet better and worse. I'm constantly blocked by cloudflare for something that should not be, it's extremely frustrating. Then when you complain to them they throw their hands up and say "owner hasn't configured their site for that". Ugh. [a] https://blog.sonatype.com/using-nexus-3-as-your-repository-part-2-npm-packages https://blog.sonatype.com/using-nexus-3-as-your-repository-p...

hoppla 7y ago
Cloudflare must value my ability to identify fire hydrants and crosswalks, because I sure get a lot of these challenges...

keanzu 7y ago

Reliance on npm is one of the things covered in Ryan Dahl's "10 Things I Regret About Node.js" talk at JSConf EU 2018 and is fixed in Deno. Includes a built-in package manager for resource fetching, thus no need for NPM. https://en.wikipedia.org/wiki/Deno_(software) https://en.wikipedia.org/wiki/Deno_(software)

nonbirithm 7y ago

Here's an explanation from CloudFlare as to the root cause. [0] > I am the engineering manager for the DDoS protection team and this morning at 11:06 UTC we tweaked a rule that affected one of our signals. The signal relates to the HTTP referer header, and we have a piece of code that looks at invalid referer headers. In this case we tweaked it to include not just "obvious garbage" but "anything that does not conform to the HTTP specification"... i.e. is the referer a URI? If not then it contributes to knowledge about bad traffic. > So... why did this impact npmjs.org? It turns out that a lot of NPM traffic sends the referer as "install" which is invalid according to the HTTP specification. As NPM is also a heavily trafficked site this resulted in the DDoS systems picking this up and treating the traffic as a HTTP flood and determining that a rate-limit should be applied. > When we noticed that NPM was seeing an increase in HTTP 429s (as seen on Twitter) we contacted NPM and started an internal investigation. As soon as we identified the root cause we reverted the change, which was at 13:00 UTC. > We'll note that NPM and 1 other site use the referer for purposes outside the HTTP spec and we'll update our systems to ensure that this does not happen again. Additionally we'll improve our monitoring around changes of this nature so that we can discover impact sooner and roll back automatically. [0] https://github.com/npm/cli/issues/836#issuecomment-587019096 https://github.com/npm/cli/issues/836#issuecomment-587019096