6 ms·

> NPM being such an important pillar in the software supply chain while having an unviable business model and largely being funded by VC money was never a good

by mbesto 7y ago

> NPM being such an important pillar in the software supply chain while having an unviable business model and largely being funded by VC money was never a good position to be in. Why does NPM need to be funded as a commercial entity at all? What other open source library has a private company running its package manager? This one still boggles my mind.

fierarul 7y ago
Maven Central is no hosted by Apache for example.
timrod 7y ago
For programming languages, there are several examples of commercially run package managers: - the Java/Kotlin/Scala ecosystem is based around maven central, which is run by Sonatype, Inc. - Go modules are hosted by Google. Previously, most libraries were hosted on Github - Rust's crate index is on Github - The Docker/Moby registry is run by Docker, Inc. (though that might be a stretch for "package manager" :))
- zeeboo 7y ago
  Technically the Go module _proxy_ is hosted by Google. Even if the proxy went away, you'd still be able to get access to all of the packages as they're still hosted elsewhere. It just wouldn't be as fast.
- bad_user 7y ago
  Maven Central has mirrors and alternatives and you can trivially host your own repository, all you'd need is a plain web server serving a bunch of static files. Some libraries aren't hosted on Maven Central actually, so it's not uncommon to see instructions for adding extra resolvers to your build config. The Java ecosystem isn't as dependent on Maven Central as the JavaScript ecosystem is on npmjs.com
  - brunoborges 7y ago
    Almost every library out there is on Maven Central. Even Oracle JDBC drivers are now (finally) on Maven Central. If MC goes away as it exists today, the Java Ecosystem will take a huge hit as almost every open source project would stop building in CICD environments from the get-go.
    - thu2111 7y ago
      If it vanished instantly then yes, but a huge number of packages on Central are mirrors from jcenter. There are not only theoretical competitors to Maven Central but an actual widely used one (jcenter/bintray), which is easier to use anyway. There's also jitpack too. So people could migrate pretty quickly to alternatives.
      - sverhagen 7y ago
        You're not technically wrong, but I bet that 90% of the projects or at least the examples have some sort of intentional or unintentional reliance on Maven Central, that would break the build if it weren't there. Even a lot of companies that set up internal repositories don't realize that they hit still are gonna hit Maven Central initially, before everything is bootstrapped, depending on how things are configured. It would reflect just really poorly on the Maven ecosystem (or any ecosystem alike, I have to assume, but know less about) if the canonical repository would just... "poof".
        thu2111 7y ago
        It's actually really common in the JVM ecosystem to run private mirrors of Central.
- monadic2 7y ago
  I wasn’t aware that I was a commercial entity because I use github!
  - iterator5 7y ago
    I think the point is that you are using a commercial entity to host your code. There is a bill for the code you have hosted, and you aren't the one paying for it.
    - monadic2 7y ago
      If that was the point why did they write “commercially run”? That is explicitly about management, not hosting or single points of failure.
  - dmix 7y ago
    It's never a problem until it is all at once and you realize they hold all the keys.
- notRobot 7y ago
  Please don't use code blocks for regular text and quotes. Really hard to read on mobile and narrow viewports.
- kibwen 7y ago
  > Rust's crate index is on Github Note that the crates.io index is just a single git repo that holds JSON metadata about each crate: https://github.com/rust-lang/crates.io-index https://github.com/rust-lang/crates.io-index . The actual code found on crates.io is hosted on S3. The index is an important part of the system, but there's nothing tying it to Github specifically.
  - avolcano 7y ago
    Who's funding the S3 bill? Took a peek around crates.io but didn't see anything about it.
    - sondr3 7y ago
      Amazon is! https://aws.amazon.com/jp/blogs/opensource/aws-sponsorship-of-the-rust-project/ https://aws.amazon.com/jp/blogs/opensource/aws-sponsorship-o...
      - avolcano 7y ago
        Ah, neat! Works out well for them; I assume it's relatively cheap to just serve an S3 bucket on their CDN (though I guess bandwidth costs may rise rather dramatically if Rust ever reaches Node levels of popularity), while not taking on any other operational expenses of actually running a registry.
        cloudboy1 7y ago
        CDN bandwidth by nature of the reach of Cloudfront will mostly be offload onto the local peering fabrics. It basically costs nothing in per mbit billing - e.g. linx https://www.linx.net/wp-content/uploads/2017/10/Fees_Schedule_20183.pdf https://www.linx.net/wp-content/uploads/2017/10/Fees_Schedul.... 100gb hand. That's not to say there aren't other major costs in running a resilent edge network which go someway to justify 0.0Y/GB pricing where Y varies from 1-9 depending on location for non sponsored projects. tl;dr - it won't ever be a problem
- steveklabnik 7y ago
  crates.io is not a commercial enterprise.
greggman3 7y ago
Confused why you think a service servings millions or billions of requests a day wouldn't require money to run. Do you think some grant magically appear out of thin air to pay for the servers, storage, bandwidth, and maintenance?
- ivanbakel 7y ago
  Big leap from "servers cost money" to "a package manager requires a commercial entity". How are other language ecosystems and package managers operating, many without private companies attached, when they too are serving millions of requests a day?
  - yawaramin 7y ago
    Well the big ones are hosted by commercial enterprises, and the smaller ones don't have the scale of JavaScript so they don't require that level of support.
  - fastball 7y ago
    Which package manager has the scale of JS and isn't hosted by a commercial entity?
    - shakna 7y ago
      apt, pacman, pip, and others. You can have large scale systems that aren't dependent on a commercial entity.
      - fastball 7y ago
        I mean, the scale of pacman is miniscule compared to NPM. PyPI is hosted by AWS and Google. Not sure how APT works in the linux community can really work for anyone else, but definitely worth a shot for sure.
- hobofan 7y ago
  Almost any other package repository is funded by donations from companies using them or a grant from an infrastructure provider.
deleted 7y ago
[deleted]
nl 7y ago
PyPy runs on donated infrastructure that costs over 800K/month in hosting costs[1]. Not many non-commercial entities can afford that. [1] https://twitter.com/dstufft/status/1236331765846990848 https://twitter.com/dstufft/status/1236331765846990848
- hnra 7y ago
  PyPI, not PyPy. For those confused over how PyPy could cost that much.
  - nl 7y ago
    Yes you are correct of course - my mistyping!
  - antonvs 7y ago
    Compiling Python code isn't easy!
- chatmasta 7y ago
  Looks like they’re paying for bandwidth, aka “cloud pricing.” They could probably reduce those numbers significantly, but if it’s funded by a grant why bother?
- deleted 7y ago
  [deleted]
- jefurii 7y ago
  Good reminder to use pip cache.

by mbesto 7y ago

fierarul 7y ago

Maven Central is no hosted by Apache for example.

timrod 7y ago

For programming languages, there are several examples of commercially run package managers: - the Java/Kotlin/Scala ecosystem is based around maven central, which is run by Sonatype, Inc. - Go modules are hosted by Google. Previously, most libraries were hosted on Github - Rust's crate index is on Github - The Docker/Moby registry is run by Docker, Inc. (though that might be a stretch for "package manager" :))

zeeboo 7y ago
Technically the Go module _proxy_ is hosted by Google. Even if the proxy went away, you'd still be able to get access to all of the packages as they're still hosted elsewhere. It just wouldn't be as fast.
bad_user 7y ago
Maven Central has mirrors and alternatives and you can trivially host your own repository, all you'd need is a plain web server serving a bunch of static files. Some libraries aren't hosted on Maven Central actually, so it's not uncommon to see instructions for adding extra resolvers to your build config. The Java ecosystem isn't as dependent on Maven Central as the JavaScript ecosystem is on npmjs.com
- brunoborges 7y ago
  Almost every library out there is on Maven Central. Even Oracle JDBC drivers are now (finally) on Maven Central. If MC goes away as it exists today, the Java Ecosystem will take a huge hit as almost every open source project would stop building in CICD environments from the get-go.
  - thu2111 7y ago
    If it vanished instantly then yes, but a huge number of packages on Central are mirrors from jcenter. There are not only theoretical competitors to Maven Central but an actual widely used one (jcenter/bintray), which is easier to use anyway. There's also jitpack too. So people could migrate pretty quickly to alternatives.
    - sverhagen 7y ago
      You're not technically wrong, but I bet that 90% of the projects or at least the examples have some sort of intentional or unintentional reliance on Maven Central, that would break the build if it weren't there. Even a lot of companies that set up internal repositories don't realize that they hit still are gonna hit Maven Central initially, before everything is bootstrapped, depending on how things are configured. It would reflect just really poorly on the Maven ecosystem (or any ecosystem alike, I have to assume, but know less about) if the canonical repository would just... "poof".
      - thu2111 7y ago
        It's actually really common in the JVM ecosystem to run private mirrors of Central.
monadic2 7y ago
I wasn’t aware that I was a commercial entity because I use github!
- iterator5 7y ago
  I think the point is that you are using a commercial entity to host your code. There is a bill for the code you have hosted, and you aren't the one paying for it.
  - monadic2 7y ago
    If that was the point why did they write “commercially run”? That is explicitly about management, not hosting or single points of failure.
- dmix 7y ago
  It's never a problem until it is all at once and you realize they hold all the keys.
notRobot 7y ago
Please don't use code blocks for regular text and quotes. Really hard to read on mobile and narrow viewports.
kibwen 7y ago
> Rust's crate index is on Github Note that the crates.io index is just a single git repo that holds JSON metadata about each crate: https://github.com/rust-lang/crates.io-index https://github.com/rust-lang/crates.io-index . The actual code found on crates.io is hosted on S3. The index is an important part of the system, but there's nothing tying it to Github specifically.
- avolcano 7y ago
  Who's funding the S3 bill? Took a peek around crates.io but didn't see anything about it.
  - sondr3 7y ago
    Amazon is! https://aws.amazon.com/jp/blogs/opensource/aws-sponsorship-of-the-rust-project/ https://aws.amazon.com/jp/blogs/opensource/aws-sponsorship-o...
    - avolcano 7y ago
      Ah, neat! Works out well for them; I assume it's relatively cheap to just serve an S3 bucket on their CDN (though I guess bandwidth costs may rise rather dramatically if Rust ever reaches Node levels of popularity), while not taking on any other operational expenses of actually running a registry.
      - cloudboy1 7y ago
        CDN bandwidth by nature of the reach of Cloudfront will mostly be offload onto the local peering fabrics. It basically costs nothing in per mbit billing - e.g. linx https://www.linx.net/wp-content/uploads/2017/10/Fees_Schedule_20183.pdf https://www.linx.net/wp-content/uploads/2017/10/Fees_Schedul.... 100gb hand. That's not to say there aren't other major costs in running a resilent edge network which go someway to justify 0.0Y/GB pricing where Y varies from 1-9 depending on location for non sponsored projects. tl;dr - it won't ever be a problem
steveklabnik 7y ago
crates.io is not a commercial enterprise.

greggman3 7y ago

Confused why you think a service servings millions or billions of requests a day wouldn't require money to run. Do you think some grant magically appear out of thin air to pay for the servers, storage, bandwidth, and maintenance?

ivanbakel 7y ago
Big leap from "servers cost money" to "a package manager requires a commercial entity". How are other language ecosystems and package managers operating, many without private companies attached, when they too are serving millions of requests a day?
- yawaramin 7y ago
  Well the big ones are hosted by commercial enterprises, and the smaller ones don't have the scale of JavaScript so they don't require that level of support.
- fastball 7y ago
  Which package manager has the scale of JS and isn't hosted by a commercial entity?
  - shakna 7y ago
    apt, pacman, pip, and others. You can have large scale systems that aren't dependent on a commercial entity.
    - fastball 7y ago
      I mean, the scale of pacman is miniscule compared to NPM. PyPI is hosted by AWS and Google. Not sure how APT works in the linux community can really work for anyone else, but definitely worth a shot for sure.
hobofan 7y ago
Almost any other package repository is funded by donations from companies using them or a grant from an infrastructure provider.

deleted 7y ago

[deleted]

nl 7y ago

PyPy runs on donated infrastructure that costs over 800K/month in hosting costs[1]. Not many non-commercial entities can afford that. [1] https://twitter.com/dstufft/status/1236331765846990848 https://twitter.com/dstufft/status/1236331765846990848

hnra 7y ago
PyPI, not PyPy. For those confused over how PyPy could cost that much.
- nl 7y ago
  Yes you are correct of course - my mistyping!
- antonvs 7y ago
  Compiling Python code isn't easy!
chatmasta 7y ago
Looks like they’re paying for bandwidth, aka “cloud pricing.” They could probably reduce those numbers significantly, but if it’s funded by a grant why bother?
deleted 7y ago
[deleted]
jefurii 7y ago
Good reminder to use pip cache.