3 ms·

> In fact, as of this writing, the JPC depends on 103 NPM packages It's really scary that you need so many packages to build a web app using Node.js.

by KAdot 11y ago

danbruc 11y ago
When I read 20 seconds to compile the list of the 10 most downloaded packages during the past 14 days, my first reaction was that this seems ridicules slow. How many downloads could that query have to accumulate? A search then revealed that they have surpassed a billion downloads per month. Who on earth installs a billion packages and that every month?
- NathanKP 11y ago
  Most companies using Node probably also have some form of CI. For example the company I work at probably generates tens of thousands of package downloads a day because we have CI running docker container builds for automated testing on every commit, and each container build downloads packages from scratch for an entirely fresh build. Once a container is built and passes tests it is reused as many times as needed for deploying out on edge hosts so there are no more additional package downloads after that, but still the continual CI builds throughout the day as people commit code generate a lot of downloads.
  - optimusclimb 11y ago
    FWIW, in that situation, a company should be using their own npm repo, not hitting the public NPM. Artifactory provides one such solution. It blows me away that so many companies so late in the game still do things like deploy from github, or do CI/CD hitting things like PyPI or NPM constantly. Yes, these services are generally reliable. If you have a critical release to push, should github being down be an acceptable reason to not be able to deploy it?
  - danbruc 11y ago
    Why would you download dependencies on each build instead of once you integrate a new dependency into the project and then just keep it under source control?
    - NathanKP 11y ago
      If you have a thorough test suite then it is possible to use a package version pattern such as ~1.1.0, or even ^1.1.0 to allow the package to upgrade automatically when the maintainer releases a new version. Obviously being able to do this is highly dependent on having really thorough test coverage to make sure that automatic package upgrades don't break stuff, but this would be one primary reason for downloading dependencies fresh for each build. But even if you had locked down the package versions it still wouldn't be a good idea to commit the package into your source control. Part of the benefit of hitting NPM to download the package is that package maintainers will often explicitly deprecate a package version that should now be considered outdated, or perhaps which had a security vulnerability. This will show up as a very visible warning message when running npm install. This has alerted me multiple times to issues with child modules I had added to my project, or even grandchild modules that were included by other child modules.
- aswan 11y ago
  I think you're mixing up two different things discussed in the article. There is one remark about how many packages are used by Jut, but then the case study is about NPM using Jut to analyze traffic at the NPM registry. The billions of downloads per month is global traffic. For instance see https://twitter.com/seldo/status/631289016101441537 https://twitter.com/seldo/status/631289016101441537
  - danbruc 11y ago
    Nope, I totally realized that the parent comment was just talking about the number of Jut dependencies while I looked for the aggregate number of monthly downloads from NPM. Both numbers are unexpectedly large.
betenoire 11y ago
considering how many "packages" are implicitly provided by a traditional LAMP stack, it's not _that_ surprising. You want cookies? package. Sessions? package. promises? package*. make http requests? package. templating? package. hashing algorithm? package. You get the picture. If I had to replace every PHP function I use with a package, I'd end up with just as many packages.
- KAdot 11y ago
  That's the problem. In Python, Ruby or even PHP you usually just pick a function / module from the standard library, in Node.js you need to choose one from ~10 quite similar and quite popular third-party packages.
  - betenoire 11y ago
    that's true. in my memory, it was that way with php 15 years ago too, thinking of phpclasses.org, hotscripts.com etc. Always sorting through the trash...
  - kaoD 11y ago
    > In Python, Ruby or even PHP you usually just pick a function / module from the standard library Which is usually rotten for the sake of not breaking things, and forces you to use a third party library anyways (at least it's my experience with Python and exactly the reason why PHP is a mess). IMHO Node.js not being "batteries included" is for the best.

> In fact, as of this writing, the JPC depends on 103 NPM packages It's really scary that you need so many packages to build a web app using Node.js.

by KAdot 11y ago

> In fact, as of this writing, the JPC depends on 103 NPM packages It's really scary that you need so many packages to build a web app using Node.js.

danbruc 11y ago

When I read 20 seconds to compile the list of the 10 most downloaded packages during the past 14 days, my first reaction was that this seems ridicules slow. How many downloads could that query have to accumulate? A search then revealed that they have surpassed a billion downloads per month. Who on earth installs a billion packages and that every month?

NathanKP 11y ago
Most companies using Node probably also have some form of CI. For example the company I work at probably generates tens of thousands of package downloads a day because we have CI running docker container builds for automated testing on every commit, and each container build downloads packages from scratch for an entirely fresh build. Once a container is built and passes tests it is reused as many times as needed for deploying out on edge hosts so there are no more additional package downloads after that, but still the continual CI builds throughout the day as people commit code generate a lot of downloads.
- optimusclimb 11y ago
  FWIW, in that situation, a company should be using their own npm repo, not hitting the public NPM. Artifactory provides one such solution. It blows me away that so many companies so late in the game still do things like deploy from github, or do CI/CD hitting things like PyPI or NPM constantly. Yes, these services are generally reliable. If you have a critical release to push, should github being down be an acceptable reason to not be able to deploy it?
- danbruc 11y ago
  Why would you download dependencies on each build instead of once you integrate a new dependency into the project and then just keep it under source control?
  - NathanKP 11y ago
    If you have a thorough test suite then it is possible to use a package version pattern such as ~1.1.0, or even ^1.1.0 to allow the package to upgrade automatically when the maintainer releases a new version. Obviously being able to do this is highly dependent on having really thorough test coverage to make sure that automatic package upgrades don't break stuff, but this would be one primary reason for downloading dependencies fresh for each build. But even if you had locked down the package versions it still wouldn't be a good idea to commit the package into your source control. Part of the benefit of hitting NPM to download the package is that package maintainers will often explicitly deprecate a package version that should now be considered outdated, or perhaps which had a security vulnerability. This will show up as a very visible warning message when running npm install. This has alerted me multiple times to issues with child modules I had added to my project, or even grandchild modules that were included by other child modules.
aswan 11y ago
I think you're mixing up two different things discussed in the article. There is one remark about how many packages are used by Jut, but then the case study is about NPM using Jut to analyze traffic at the NPM registry. The billions of downloads per month is global traffic. For instance see https://twitter.com/seldo/status/631289016101441537 https://twitter.com/seldo/status/631289016101441537
- danbruc 11y ago
  Nope, I totally realized that the parent comment was just talking about the number of Jut dependencies while I looked for the aggregate number of monthly downloads from NPM. Both numbers are unexpectedly large.

betenoire 11y ago

considering how many "packages" are implicitly provided by a traditional LAMP stack, it's not _that_ surprising. You want cookies? package. Sessions? package. promises? package*. make http requests? package. templating? package. hashing algorithm? package. You get the picture. If I had to replace every PHP function I use with a package, I'd end up with just as many packages.

KAdot 11y ago
That's the problem. In Python, Ruby or even PHP you usually just pick a function / module from the standard library, in Node.js you need to choose one from ~10 quite similar and quite popular third-party packages.
- betenoire 11y ago
  that's true. in my memory, it was that way with php 15 years ago too, thinking of phpclasses.org, hotscripts.com etc. Always sorting through the trash...
- kaoD 11y ago
  > In Python, Ruby or even PHP you usually just pick a function / module from the standard library Which is usually rotten for the sake of not breaking things, and forces you to use a third party library anyways (at least it's my experience with Python and exactly the reason why PHP is a mess). IMHO Node.js not being "batteries included" is for the best.