5 ms·

Or the URL-based model used by Go.

by Aissen 6y ago

SideburnsOfDoom 6y ago
What does the url model bring? Packages are typically considered immutable once published. If I have a particular package e.g. "FooLib.Bar v 1.2.3" then this zip file should _always_ contain the same bits. If I need to change those bits, e.g. to fix a bug then I need to ship e.g. "FooLib.Bar v 1.2.4" Also packages aren't always small. So it makes sense to cache a copy locally. On dev machine "package cache" and in an org's "internal feed" and only check upstream if it's not there. So I shouldn't need to go to the source url to get it. Ideally, I just ask "who has "FooLib.Bar v 1.2.3" for me?" It also means that tampering can be detected with a hash. But the "check upstream" model is now vulnerable to fake new versions.
- mythz 6y ago
  Using a FQDN is less likely to be unknowingly hijacked when it's a domain they control and use daily. URL references also contain the version number, typically an immutable Git tag reference. They also benefit from just needing to download the source code that's referenced and not the entire package. With Deno you can also optionally host versioned 3rd Party packages on their deno.land/x CDN. URL references are also cached locally on first use and can be used offline thereafter.
- michaelt 6y ago
  For better or for worse, many projects auto-update their dependencies these days. They do this to address the shortfalls of modern conventions like small packages, continuous release cycles and dependencies nested several layers deep. So if you were using the internal package FooLib.Bar v 1.2.3 and an attacker posts FooLib.Bar v 1.2.4 to a global repository, anyone using auto-updating will update to it.
  - SideburnsOfDoom 6y ago
    I don't disagree, but both of these (fully or partly automated updates, and attackers) are fairly recent developments to the model. Of course, mitigation is needed. Supply chain attacks are a hot topic after SolarWinds. But identifying a package version solely by a url doesn't seem like the right abstraction to me. IMHO, the metadata is more structured: Name (text), version (SemVer) and also maybe now fields to verify and mitigate these attacks: content hash, source feed, etc. Even if I run an internal feed that transparently proxies and caches the public one, as well as hosting my company's internal artefacts, the rules now might need to be different between packages? for e.g. between Newtonsoft.Json (new versions always originate on the public feed, never locally) and "SDCo.GeneralUtils" (new versions always originate on the local feed, never upstream)
XorNot 6y ago
At this point I really wish we'd just go with a proper cryptography model, with a discovery overlay to provide names. What I want as a developer is to establish my trust relationship to developers of libraries I depend on. `npm install <somepackage>` should first check a record of signing keys in my source code repo, then check a user-level record of signing keys I've trusted before, and then - and only then - add a tentative trust relationship if this is brand new. `npm release` or whatever (npm is just an example - every system could benefit from this) - would then actually give me the list of new trust relationships needed, so I can go and do some validation that these are the packages I think they are.
- brabel 6y ago
  Maven Central has used PGP for adding "trust" to library authors since the 90's! https://central.sonatype.org/pages/requirements.html#sign-files-with-gpgpgp https://central.sonatype.org/pages/requirements.html#sign-fi... If only people creating new package managers would bother to spend an hour or two learning prior art. With npm, you can only add "trust" to npm itself LOL: https://docs.npmjs.com/about-pgp-signatures-for-packages-in-the-public-registry https://docs.npmjs.com/about-pgp-signatures-for-packages-in-... What a joke.
  - bayindirh 6y ago
    > If only people creating new package managers would bother to spend an hour or two learning prior art. No, no... We should move fast and break things. We can implement this in a week because the old dinosaurs are too close minded to implement these things. Today, the hardware is cheap and network is reliable. No need for any safeguards or security features. /s
- wh33zle 6y ago
  You might be interested in `crev`: https://github.com/crev-dev https://github.com/crev-dev
  - monoideism 6y ago
    Despite being very interested in Rust, this is the first I've heard of crev. It's a very cool project. That said, it's interesting to me that several people are trying to get the project to drop the Web of Trust, and focus on code reviews. I'm the exact opposite - the code reviews are an interesting, experimental approach, but I'm interested in the project because of the cryptographic Web of Trust. Any use of dev-originated code signing in a package ecosystem is great. For this reason, I'd love for this to get major pickup from Rust, and beyond. Finally, I am a bit wary because the project is starting to look moribund. It's important for projects like these to know that the maintainer is in it for the long haul, even if there's initially very little adoption. When the project founder writes that they're in a "fight for survival", it makes me think they may abandon the project if it doesn't get significant adoption.
Philip-J-Fry 6y ago
Using a URL isn't what makes Go's dependency management that good. It's just convenience that the import is a URL. The key thing with Go is that all dependencies have a checksum (go.sum file) and that should be committed to the repo. So even if the domain gets hijacked and a malicious package is served up, then the checksum will fail and it will refuse to build. People should be using internal module proxies anyway for Go. You can just store the module files in a directory, a git repo or a web service and serve up an internal cache.

Or the URL-based model used by Go.

by Aissen 6y ago

Or the URL-based model used by Go.

What does the url model bring? Packages are typically considered immutable once published. If I have a particular package e.g. "FooLib.Bar v 1.2.3" then this zip file should _always_ contain the same bits. If I need to change those bits, e.g. to fix a bug then I need to ship e.g. "FooLib.Bar v 1.2.4" Also packages aren't always small. So it makes sense to cache a copy locally. On dev machine "package cache" and in an org's "internal feed" and only check upstream if it's not there. So I shouldn't need to go to the source url to get it. Ideally, I just ask "who has "FooLib.Bar v 1.2.3" for me?" It also means that tampering can be detected with a hash. But the "check upstream" model is now vulnerable to fake new versions.

mythz 6y ago
Using a FQDN is less likely to be unknowingly hijacked when it's a domain they control and use daily. URL references also contain the version number, typically an immutable Git tag reference. They also benefit from just needing to download the source code that's referenced and not the entire package. With Deno you can also optionally host versioned 3rd Party packages on their deno.land/x CDN. URL references are also cached locally on first use and can be used offline thereafter.
michaelt 6y ago
For better or for worse, many projects auto-update their dependencies these days. They do this to address the shortfalls of modern conventions like small packages, continuous release cycles and dependencies nested several layers deep. So if you were using the internal package FooLib.Bar v 1.2.3 and an attacker posts FooLib.Bar v 1.2.4 to a global repository, anyone using auto-updating will update to it.
- SideburnsOfDoom 6y ago
  I don't disagree, but both of these (fully or partly automated updates, and attackers) are fairly recent developments to the model. Of course, mitigation is needed. Supply chain attacks are a hot topic after SolarWinds. But identifying a package version solely by a url doesn't seem like the right abstraction to me. IMHO, the metadata is more structured: Name (text), version (SemVer) and also maybe now fields to verify and mitigate these attacks: content hash, source feed, etc. Even if I run an internal feed that transparently proxies and caches the public one, as well as hosting my company's internal artefacts, the rules now might need to be different between packages? for e.g. between Newtonsoft.Json (new versions always originate on the public feed, never locally) and "SDCo.GeneralUtils" (new versions always originate on the local feed, never upstream)

XorNot 6y ago

At this point I really wish we'd just go with a proper cryptography model, with a discovery overlay to provide names. What I want as a developer is to establish my trust relationship to developers of libraries I depend on. `npm install <somepackage>` should first check a record of signing keys in my source code repo, then check a user-level record of signing keys I've trusted before, and then - and only then - add a tentative trust relationship if this is brand new. `npm release` or whatever (npm is just an example - every system could benefit from this) - would then actually give me the list of new trust relationships needed, so I can go and do some validation that these are the packages I think they are.

brabel 6y ago
Maven Central has used PGP for adding "trust" to library authors since the 90's! https://central.sonatype.org/pages/requirements.html#sign-files-with-gpgpgp https://central.sonatype.org/pages/requirements.html#sign-fi... If only people creating new package managers would bother to spend an hour or two learning prior art. With npm, you can only add "trust" to npm itself LOL: https://docs.npmjs.com/about-pgp-signatures-for-packages-in-the-public-registry https://docs.npmjs.com/about-pgp-signatures-for-packages-in-... What a joke.
- bayindirh 6y ago
  > If only people creating new package managers would bother to spend an hour or two learning prior art. No, no... We should move fast and break things. We can implement this in a week because the old dinosaurs are too close minded to implement these things. Today, the hardware is cheap and network is reliable. No need for any safeguards or security features. /s
wh33zle 6y ago
You might be interested in `crev`: https://github.com/crev-dev https://github.com/crev-dev
- monoideism 6y ago
  Despite being very interested in Rust, this is the first I've heard of crev. It's a very cool project. That said, it's interesting to me that several people are trying to get the project to drop the Web of Trust, and focus on code reviews. I'm the exact opposite - the code reviews are an interesting, experimental approach, but I'm interested in the project because of the cryptographic Web of Trust. Any use of dev-originated code signing in a package ecosystem is great. For this reason, I'd love for this to get major pickup from Rust, and beyond. Finally, I am a bit wary because the project is starting to look moribund. It's important for projects like these to know that the maintainer is in it for the long haul, even if there's initially very little adoption. When the project founder writes that they're in a "fight for survival", it makes me think they may abandon the project if it doesn't get significant adoption.

Philip-J-Fry 6y ago

Using a URL isn't what makes Go's dependency management that good. It's just convenience that the import is a URL. The key thing with Go is that all dependencies have a checksum (go.sum file) and that should be committed to the repo. So even if the domain gets hijacked and a malicious package is served up, then the checksum will fail and it will refuse to build. People should be using internal module proxies anyway for Go. You can just store the module files in a directory, a git repo or a web service and serve up an internal cache.