4 ms·

URLs for packages makes a lot of sense. It works well in the land of Go. It also conveniently eliminates the need for the language to have a global packages d

URLs for packages makes a lot of sense. It works well in the land of Go. It also conveniently eliminates the need for the language to have a global packages database. Upload your package to example.com/your-thing and it's released! (You can, of course, still offer a cache and search engine if you want to.)

alpaca128 3y ago
Cargo does support URLs to git repos for dependencies. But crates.io is the official platform and almost every search I do on it returns at least one generically named entry with an empty repository that someone snatched away and never used.
swatcoder 3y ago
No, URL's don't make sense because your application shouldn't care where on the internet your dependency happened to be hosted when you integrated it. It's location has nothing to do with what it is. By the time you're going to production, your vetted and locked dependency should be living in your own cache/mirror/vendored-repo/whatever so that you know exactly what code you built your project around and know exactly what the availability will be when you build/instantiate your project. Your project shouldn't need to care whether GitHub fell out of fashion and the project moved to GitLab, and definitely shouldn't be relying on GitHub being available when you need to build, test, deploy, or scale. That's a completely unnecessary failure point for you to introduce. Systems that use URL-identified packages can work around some of this, but just reinforce terrible habits.
- jrockway 3y ago
  It worked for the rest of the Internet.
  - ethbr1 3y ago
    There is something to be said for separating {unique piece of content} and {hosted location}. E.g. doi's https://en.m.wikipedia.org/wiki/Digital_object_identifier https://en.m.wikipedia.org/wiki/Digital_object_identifier
- kevincox 3y ago
  URLs form a nice global namespacing system. But yes, I agree that it should be possible to actually get the source from anywhere. Basically the URL of a package name should be primarily the ID, not the locator (even if it is used for location by default).
- andiareso 3y ago
  Isn’t that why GOPROXY exists though? Not sure why you would need an internet connection. URLs don’t necessarily equate to the internet. Our internal and external packages are all locally hosted and work regardless of the internet being available.
- awesome_dude 3y ago
  > By the time you're going to production, your vetted and locked dependency should be living in your own cache/mirror/vendored-repo/whatever so that you know exactly what code you built your project around and know exactly what the availability will be when you build/instantiate your project. In the Go world this would be "vendored" dependencies, that is, the dependencies are within your source tree, and your CICD can build to its hearts content with no care in the world about the internet because it has the deps. The URL is useful for determining which version of a specific project is being used - "Oh we switched to the one hosted on gitlab because the github one went stale" The advantage of using gitlab, or github, or whatever public code repository is that you get to piggy back off their naming policies which ensure uniqueness. But, at the same time, there's no reason that the repo being referred to cannot be in house (bob.local) or private. Having said all of that, the Go module system is a massive improvement on what they did have originally (nothing) and the 3rd party attempts to solve the problem (dep, glide, and the prototype for modules, vgo), but it's not without its edge cases.
- taberiand 3y ago
  URLs are well structured and unique, with a sensible default - sourcing the file from the internet - and ubiquitous processes for easily mapping the URL to an alternative location. I.e., when you're going to run the production build, the URLs are mapped to fetch from the vetted cache and not the internet. I don't see any downsides to allowing them as a source, or making them the default approach
  - ants_everywhere 3y ago
    > and ubiquitous processes for easily mapping the URL to an alternative location. This seems strange to me because the whole point of a Uniform Resource Locator is to specify where a resource can be located. It's a bit like saying "My project depends on the binder on shelf 7 in Room 42, sixth binder from the left. Except when I go into production, then use...." Don't tell me what binder it's in, tell me what it is. I can see a case made for URIs, which is basically what Java did.
    - devman0 3y ago
      This was a big annoyance for me back in the day when I was dealing with XML namespaces. URLs never made sense for that use case and too many tools tried to pull XSDs from the literal URL which was always generally out of date, some projects switch to URIs like tag uris or URNs and it was much better, imo.
      - drzaiusx11 3y ago
        From my experience, URNs really should be used more often for these sorts of things. One thing that AWS got right almost from the get go
    - drzaiusx11 3y ago
      Fully qualified domain names (java/maven) aren't URIs. The latter are far more transient. Maybe a form of permalink could work, but that likely places too great a burden on package maintainers. I don't see that working out honestly.
- jowea 3y ago
  Isn't that just delegating the problem? URL dependencies do not replace what crates.io does, and a modern language will still want something like it. You'd just end up with most every dependency being declared as crates.io/foo.
- kitd 3y ago
  You can use the `replace` option in the Go mod file to redirect your dependency references elsewhere if you need to.
estebank 3y ago
It doesn't help with the failure mode of dependencies disappearing, which forces people that care about it to vendor, which in turn brings its own set of issues.
jeroenhd 3y ago
You can, though. From a random Cargo package I have downloaded to my computer: [dependencies] uniffi = { git = "https://github.com/mozilla/uniffi-rs" } You can also specify revision/branch/etc. Alternatively, you can do: [registries] maven = { index = "https://rust.maven.org/git/index" } [dependencies] some-package = { index = "maven", version = "1.1" } Obviously Maven doesn't host any Rust crates (yet?), this is just a theoretical example. Very few projects bother to host their own registry, partially because crates.io doesn't allow packages that load dependencies from other indices (for obvious security reasons). The registry definition can also be done globally through environment variables: CARGO_REGISTRIES_MAVEN="https://rust.maven.org/git/index https://rust.maven.org/git/index". Furthermore, the default registry can be set in a global config file. In theory, all you need to do is publish a crate is to `git push upstream master`, and your package will become available on https://github.com/username/crate-name https://github.com/username/crate-name (or example.com/your-package if you choose to host your git repo on there). Personally, I don't like using other people's URL packages, because your website can disappear any moment for any reason. Maybe you decide to call it quits, maybe you get hit by a car, whatever the reason, my build is broken all of the sudden. The probability of crates.io going down is a lot lower than the probability of packages-of-some-random-guy-in-nebraska.ddns.net disappearing

alpaca128 3y ago

Cargo does support URLs to git repos for dependencies. But crates.io is the official platform and almost every search I do on it returns at least one generically named entry with an empty repository that someone snatched away and never used.

swatcoder 3y ago

No, URL's don't make sense because your application shouldn't care where on the internet your dependency happened to be hosted when you integrated it. It's location has nothing to do with what it is. By the time you're going to production, your vetted and locked dependency should be living in your own cache/mirror/vendored-repo/whatever so that you know exactly what code you built your project around and know exactly what the availability will be when you build/instantiate your project. Your project shouldn't need to care whether GitHub fell out of fashion and the project moved to GitLab, and definitely shouldn't be relying on GitHub being available when you need to build, test, deploy, or scale. That's a completely unnecessary failure point for you to introduce. Systems that use URL-identified packages can work around some of this, but just reinforce terrible habits.

jrockway 3y ago
It worked for the rest of the Internet.
- ethbr1 3y ago
  There is something to be said for separating {unique piece of content} and {hosted location}. E.g. doi's https://en.m.wikipedia.org/wiki/Digital_object_identifier https://en.m.wikipedia.org/wiki/Digital_object_identifier
kevincox 3y ago
URLs form a nice global namespacing system. But yes, I agree that it should be possible to actually get the source from anywhere. Basically the URL of a package name should be primarily the ID, not the locator (even if it is used for location by default).
andiareso 3y ago
Isn’t that why GOPROXY exists though? Not sure why you would need an internet connection. URLs don’t necessarily equate to the internet. Our internal and external packages are all locally hosted and work regardless of the internet being available.
awesome_dude 3y ago
> By the time you're going to production, your vetted and locked dependency should be living in your own cache/mirror/vendored-repo/whatever so that you know exactly what code you built your project around and know exactly what the availability will be when you build/instantiate your project. In the Go world this would be "vendored" dependencies, that is, the dependencies are within your source tree, and your CICD can build to its hearts content with no care in the world about the internet because it has the deps. The URL is useful for determining which version of a specific project is being used - "Oh we switched to the one hosted on gitlab because the github one went stale" The advantage of using gitlab, or github, or whatever public code repository is that you get to piggy back off their naming policies which ensure uniqueness. But, at the same time, there's no reason that the repo being referred to cannot be in house (bob.local) or private. Having said all of that, the Go module system is a massive improvement on what they did have originally (nothing) and the 3rd party attempts to solve the problem (dep, glide, and the prototype for modules, vgo), but it's not without its edge cases.
taberiand 3y ago
URLs are well structured and unique, with a sensible default - sourcing the file from the internet - and ubiquitous processes for easily mapping the URL to an alternative location. I.e., when you're going to run the production build, the URLs are mapped to fetch from the vetted cache and not the internet. I don't see any downsides to allowing them as a source, or making them the default approach
- ants_everywhere 3y ago
  > and ubiquitous processes for easily mapping the URL to an alternative location. This seems strange to me because the whole point of a Uniform Resource Locator is to specify where a resource can be located. It's a bit like saying "My project depends on the binder on shelf 7 in Room 42, sixth binder from the left. Except when I go into production, then use...." Don't tell me what binder it's in, tell me what it is. I can see a case made for URIs, which is basically what Java did.
  - devman0 3y ago
    This was a big annoyance for me back in the day when I was dealing with XML namespaces. URLs never made sense for that use case and too many tools tried to pull XSDs from the literal URL which was always generally out of date, some projects switch to URIs like tag uris or URNs and it was much better, imo.
    - drzaiusx11 3y ago
      From my experience, URNs really should be used more often for these sorts of things. One thing that AWS got right almost from the get go
  - drzaiusx11 3y ago
    Fully qualified domain names (java/maven) aren't URIs. The latter are far more transient. Maybe a form of permalink could work, but that likely places too great a burden on package maintainers. I don't see that working out honestly.
jowea 3y ago
Isn't that just delegating the problem? URL dependencies do not replace what crates.io does, and a modern language will still want something like it. You'd just end up with most every dependency being declared as crates.io/foo.
kitd 3y ago
You can use the `replace` option in the Go mod file to redirect your dependency references elsewhere if you need to.

estebank 3y ago

It doesn't help with the failure mode of dependencies disappearing, which forces people that care about it to vendor, which in turn brings its own set of issues.

jeroenhd 3y ago

You can, though. From a random Cargo package I have downloaded to my computer: [dependencies] uniffi = { git = "https://github.com/mozilla/uniffi-rs" } You can also specify revision/branch/etc. Alternatively, you can do: [registries] maven = { index = "https://rust.maven.org/git/index" } [dependencies] some-package = { index = "maven", version = "1.1" } Obviously Maven doesn't host any Rust crates (yet?), this is just a theoretical example. Very few projects bother to host their own registry, partially because crates.io doesn't allow packages that load dependencies from other indices (for obvious security reasons). The registry definition can also be done globally through environment variables: CARGO_REGISTRIES_MAVEN="https://rust.maven.org/git/index https://rust.maven.org/git/index". Furthermore, the default registry can be set in a global config file. In theory, all you need to do is publish a crate is to `git push upstream master`, and your package will become available on https://github.com/username/crate-name https://github.com/username/crate-name (or example.com/your-package if you choose to host your git repo on there). Personally, I don't like using other people's URL packages, because your website can disappear any moment for any reason. Maybe you decide to call it quits, maybe you get hit by a car, whatever the reason, my build is broken all of the sudden. The probability of crates.io going down is a lot lower than the probability of packages-of-some-random-guy-in-nebraska.ddns.net disappearing