4 ms·

This is why I like the Go approach: just use an existing global namespacing system and delegate to it all ownership disputes etc: DNS + actual web servers servi

This is why I like the Go approach: just use an existing global namespacing system and delegate to it all ownership disputes etc: DNS + actual web servers serving traffic (not necessarily just got repos, since the "vanity imports" offer an indirection based in html meta tags)

steveklabnik 6y ago
Yes, there are benefits to that approach. We rejected it because we want builds to be reproducible, and not to have that property depend on external entities.
- ithkuil 6y ago
  That's orthogonal with the naming scheme. In fact the Go team eventually implemented a way to decouple the storage of a copy of the package from the name of the package, namely the goproxy. How they got there was admittedly a bit clumsy, but it proves that it's possible
  - steveklabnik 6y ago
    Sort of; you can still delete github repos and the proxy doesn't keep a copy forever, right?
    - ithkuil 6y ago
      That's an implementarion choice of the proxy. You can run a private proxy that inplementa whatever policy you want. If crates wants a policy that stores crates forever then just store crates forever. What I'm talking about just affects how new crates and new versions of existing crates get "uploaded". Imagine you call your crate foo.com/bar. In order to push your crate the crates server needs to verify that you actually own the foo.com/bar URL. There are many ways to do that. A simple one is to just have the crates server pull code from that URL instead of pushing it. If you want to push (e.g. to work around reachability issues such as firewalls) it's just the matter of devising one or more ways to prove that you "own" that handle. For example by serving some challenge response in a html meta tag
      - steveklabnik 6y ago
        Right, but then you're back to the exact same issues we're talking about. The choices made by that proxy is the same discussion we're having here, and also means you can't totally rely on the external authority for identity, because you have to manage the mutability in their service. There are many ways to do it, and that in fact is the exact issue at stake here. All of the existing services it would make sense to delegate to are more mutable than we wanted for our system.
        ithkuil 6y ago
        Ok the are many angles to this discussion. What I wanted to highlight was: 1. I like a package repository that has "namespaces". In fact that's what I like about Go approach to package repository from day zero, despite other rough edges. 2. Classic approach to "namespacing" like maven's is susceptible to name squatting or just "split authority": who can claim the "org.apache" org? First comes, first served? If that doesn't match who owns the apache.org "brand" on the wider internet, it's not technically a problem of the package repository but surely it's a practical one. 3. Name squatting problems exist for DNS too; it's a know problem, with an existing administration and dispute resolution body, a marketplace and whatnot. If you cannot get hold of foo.com you'll settle on getfoo.com or (some abomination like that) and that will be your brand as far as online presence is concerned. It's only natural for artifacts you want to associate to that owner to just piggy back on that more prominent identity rather than have to fight it independently for getfoo.com and perhaps land on gotfoo.com because getfoo.com has been taken by somebody else. 4. Now, if you decided that you want namespacing and given the observed consequences of allowing your users to compete for namespace names independently of their main online presence, I claim there is a technical approach that can achieve your stated goal of reproducible builds. The goproxy is almost what you want. The main different is that the go team aims at a best effort service and thus want to call it a caching mirror. They could have chosen to offer a more strict retention guarantee. Go packages are indeed immutable. The sumdb transparent log is there to enforce that invariant. It's really just about somebody willing to take the burden of storing the package bits forever, which is what I understood you choose to do with crates.
- andrewrothman 6y ago
  Steve, thanks for all your work on Rust! I'd like to understand more about the arguments for and against a URL-based module naming / resolution approach. Could you point me to any past discussions on this for further reading?
  - steveklabnik 6y ago
    You're welcome! I am not 100% sure exactly what you're asking for, so if you could say a teeny bit more than that, that would help me understand what part of the space you're trying to explore. It would also be handy to know what problem you are trying to solve, there are several here.
    - andrewrothman 6y ago
      I should be clear my interest is theoretical. I haven't run into any practical issues with "crates.io" and it's been working great for me. I think the way Go and ES Modules (Deno / browsers) represent module naming and resolution by URLs has some nice benefits: * decentralized module hosting * could be extended beyond HTTP / Git (ie. import modules via "ipfs://") * re-uses existing web infrastructure for namespace ownership (ie. via DNS) * possibility of hosting backend and frontend modules for multiple languages in a single HTTP registry However, I do see some important tradeoffs to that approach: * modules may become unresolvable from their original URL (link rot) * cannot guarantee immutability at the original URL * semver support becomes a concern of the registry, and some registries may not support it (ie. the logic to resolve "https://example.com/my-module/1.x.x/lib.rs") * more difficult to discover (browse and search) all available modules I see some proposed solutions to the tradeoffs: * proxies and dep lockfiles of file hashes to help prevent link rot and immutability * possibility for open source registry implementations with built-in semver support * central index of modules which can be searched / browsed, if modules are submitted to it However, those solutions don't guarantee fixes for every tradeoff. I'd be very interested in learning more about the costs / benefits of the various approaches and the Rust team's discussions on modules. I really appreciate the team's thorough approach to design decisions, as it's manifested into a really cleanly-designed language. Thanks!

steveklabnik 6y ago

Yes, there are benefits to that approach. We rejected it because we want builds to be reproducible, and not to have that property depend on external entities.

ithkuil 6y ago
That's orthogonal with the naming scheme. In fact the Go team eventually implemented a way to decouple the storage of a copy of the package from the name of the package, namely the goproxy. How they got there was admittedly a bit clumsy, but it proves that it's possible
- steveklabnik 6y ago
  Sort of; you can still delete github repos and the proxy doesn't keep a copy forever, right?
  - ithkuil 6y ago
    That's an implementarion choice of the proxy. You can run a private proxy that inplementa whatever policy you want. If crates wants a policy that stores crates forever then just store crates forever. What I'm talking about just affects how new crates and new versions of existing crates get "uploaded". Imagine you call your crate foo.com/bar. In order to push your crate the crates server needs to verify that you actually own the foo.com/bar URL. There are many ways to do that. A simple one is to just have the crates server pull code from that URL instead of pushing it. If you want to push (e.g. to work around reachability issues such as firewalls) it's just the matter of devising one or more ways to prove that you "own" that handle. For example by serving some challenge response in a html meta tag
    - steveklabnik 6y ago
      Right, but then you're back to the exact same issues we're talking about. The choices made by that proxy is the same discussion we're having here, and also means you can't totally rely on the external authority for identity, because you have to manage the mutability in their service. There are many ways to do it, and that in fact is the exact issue at stake here. All of the existing services it would make sense to delegate to are more mutable than we wanted for our system.
      - ithkuil 6y ago
        Ok the are many angles to this discussion. What I wanted to highlight was: 1. I like a package repository that has "namespaces". In fact that's what I like about Go approach to package repository from day zero, despite other rough edges. 2. Classic approach to "namespacing" like maven's is susceptible to name squatting or just "split authority": who can claim the "org.apache" org? First comes, first served? If that doesn't match who owns the apache.org "brand" on the wider internet, it's not technically a problem of the package repository but surely it's a practical one. 3. Name squatting problems exist for DNS too; it's a know problem, with an existing administration and dispute resolution body, a marketplace and whatnot. If you cannot get hold of foo.com you'll settle on getfoo.com or (some abomination like that) and that will be your brand as far as online presence is concerned. It's only natural for artifacts you want to associate to that owner to just piggy back on that more prominent identity rather than have to fight it independently for getfoo.com and perhaps land on gotfoo.com because getfoo.com has been taken by somebody else. 4. Now, if you decided that you want namespacing and given the observed consequences of allowing your users to compete for namespace names independently of their main online presence, I claim there is a technical approach that can achieve your stated goal of reproducible builds. The goproxy is almost what you want. The main different is that the go team aims at a best effort service and thus want to call it a caching mirror. They could have chosen to offer a more strict retention guarantee. Go packages are indeed immutable. The sumdb transparent log is there to enforce that invariant. It's really just about somebody willing to take the burden of storing the package bits forever, which is what I understood you choose to do with crates.
andrewrothman 6y ago
Steve, thanks for all your work on Rust! I'd like to understand more about the arguments for and against a URL-based module naming / resolution approach. Could you point me to any past discussions on this for further reading?
- steveklabnik 6y ago
  You're welcome! I am not 100% sure exactly what you're asking for, so if you could say a teeny bit more than that, that would help me understand what part of the space you're trying to explore. It would also be handy to know what problem you are trying to solve, there are several here.
  - andrewrothman 6y ago
    I should be clear my interest is theoretical. I haven't run into any practical issues with "crates.io" and it's been working great for me. I think the way Go and ES Modules (Deno / browsers) represent module naming and resolution by URLs has some nice benefits: * decentralized module hosting * could be extended beyond HTTP / Git (ie. import modules via "ipfs://") * re-uses existing web infrastructure for namespace ownership (ie. via DNS) * possibility of hosting backend and frontend modules for multiple languages in a single HTTP registry However, I do see some important tradeoffs to that approach: * modules may become unresolvable from their original URL (link rot) * cannot guarantee immutability at the original URL * semver support becomes a concern of the registry, and some registries may not support it (ie. the logic to resolve "https://example.com/my-module/1.x.x/lib.rs") * more difficult to discover (browse and search) all available modules I see some proposed solutions to the tradeoffs: * proxies and dep lockfiles of file hashes to help prevent link rot and immutability * possibility for open source registry implementations with built-in semver support * central index of modules which can be searched / browsed, if modules are submitted to it However, those solutions don't guarantee fixes for every tradeoff. I'd be very interested in learning more about the costs / benefits of the various approaches and the Rust team's discussions on modules. I really appreciate the team's thorough approach to design decisions, as it's manifested into a really cleanly-designed language. Thanks!