3 ms·

Crate.io - an improv(ed|ing) Python Package index

26 points by jnoller 15y ago

masklinn 15y ago
The name bothers me: the Rust team has chosen "crates" to designate Rust libraries (technically compilation and distribution units for Rust, crates can yield libraries or binaries): http://doc.rust-lang.org/doc/rust.html#crates-and-source-files http://doc.rust-lang.org/doc/rust.html#crates-and-source-fil..., and I don't remember ever seeing this kind of lingo around Python modules and distribution units.
- donaldstufft 15y ago
  Purely coincidence. To be honest I've never seen Rust before, when I was trying to name it I just hit a thesaurus up until I came up with a word that had something to do with "packaging" (Shipping Crates!) that was available that I liked.
  - masklinn 15y ago
    > Purely coincidence. I'm sure it is, and I didn't intend to suggest malfeasance, I'm sorry if it came out wrong.
po 15y ago
Wow, bootstrap really does allow for a much nicer looking front page doesn't it? This seems like a great step forward. I was recently looking to set up a local chishop server to act as a mirror.. it seems like the cacheable url's thing means you're thinking to enable people to set this up locally and have it pass requests that 404 through to the crate.io server? I would love to do that. Is that workable now? My one suggestion would be to make a simpler to type url than 'simple.crate.io' since people may have to type it in on the command line from time to time. Perhaps 'http://crate.io/s/ http://crate.io/s/?
- donaldstufft 15y ago
  This is a good idea! I used the subdomain because I plan on hosting the "simple" interface in EC2 (Packages are hosted on S3/CloudFront) in multiple availability zones while my main site is hosted on Gondor.io. However I could enable a short url that redirected to simple.crate.io to allow easier command line access. Thanks!
  - po 15y ago
    Is the expectation that someone should be able to do what I described? Run a local copy that proxies 404's up to the main crate.io server and then caches the result locally to disk? This would be a very nice way to deal with build-time dependencies. You can get similar behavior today by setting multiple mirrors in pip but this would allow the grabbing packages on-demand instead of having to mirror everything.
    - donaldstufft 15y ago
      pip supports a download cache (off by default) that stores packages as their urlname. It works similar to how you describe. It will see it needs the package at url xyz.com/package.tar.gz, it'll check it's download cache and if it doesn't exist there it will download the file from the url and save it to the cache. The cacheable urls stem from the fact that currently on PyPI you can delete an existing file you've uploaded, and reupload a file of the same name. Pip (and any other cache like varnish etc) isn't notified of the fact that what used to be at url xyz.com/packages.tar.gz is gone and now something else is at xyz.com/packages.tar.gz. This can cause issues with installing packages, especially on remote deployment infrastructures. With crate I place the sha256 of the file in the url, so if someone deletes + uploads the same name, it should still (unless you get (un)lucky and get a collision somehow) have a completely different url, so pip and other caches will see it as a different file. As a side effect it would allow a simple caching proxy to be installed at someones office, or wherever that easily does the behavior you describe as well. It could function similarly to pip but be shared amongst multiple people.

26 points by jnoller 15y ago

The name bothers me: the Rust team has chosen "crates" to designate Rust libraries (technically compilation and distribution units for Rust, crates can yield libraries or binaries): http://doc.rust-lang.org/doc/rust.html#crates-and-source-files http://doc.rust-lang.org/doc/rust.html#crates-and-source-fil..., and I don't remember ever seeing this kind of lingo around Python modules and distribution units.

donaldstufft 15y ago
Purely coincidence. To be honest I've never seen Rust before, when I was trying to name it I just hit a thesaurus up until I came up with a word that had something to do with "packaging" (Shipping Crates!) that was available that I liked.
- masklinn 15y ago
  > Purely coincidence. I'm sure it is, and I didn't intend to suggest malfeasance, I'm sorry if it came out wrong.

po 15y ago

Wow, bootstrap really does allow for a much nicer looking front page doesn't it? This seems like a great step forward. I was recently looking to set up a local chishop server to act as a mirror.. it seems like the cacheable url's thing means you're thinking to enable people to set this up locally and have it pass requests that 404 through to the crate.io server? I would love to do that. Is that workable now? My one suggestion would be to make a simpler to type url than 'simple.crate.io' since people may have to type it in on the command line from time to time. Perhaps 'http://crate.io/s/ http://crate.io/s/?

donaldstufft 15y ago
This is a good idea! I used the subdomain because I plan on hosting the "simple" interface in EC2 (Packages are hosted on S3/CloudFront) in multiple availability zones while my main site is hosted on Gondor.io. However I could enable a short url that redirected to simple.crate.io to allow easier command line access. Thanks!
- po 15y ago
  Is the expectation that someone should be able to do what I described? Run a local copy that proxies 404's up to the main crate.io server and then caches the result locally to disk? This would be a very nice way to deal with build-time dependencies. You can get similar behavior today by setting multiple mirrors in pip but this would allow the grabbing packages on-demand instead of having to mirror everything.
  - donaldstufft 15y ago
    pip supports a download cache (off by default) that stores packages as their urlname. It works similar to how you describe. It will see it needs the package at url xyz.com/package.tar.gz, it'll check it's download cache and if it doesn't exist there it will download the file from the url and save it to the cache. The cacheable urls stem from the fact that currently on PyPI you can delete an existing file you've uploaded, and reupload a file of the same name. Pip (and any other cache like varnish etc) isn't notified of the fact that what used to be at url xyz.com/packages.tar.gz is gone and now something else is at xyz.com/packages.tar.gz. This can cause issues with installing packages, especially on remote deployment infrastructures. With crate I place the sha256 of the file in the url, so if someone deletes + uploads the same name, it should still (unless you get (un)lucky and get a collision somehow) have a completely different url, so pip and other caches will see it as a different file. As a side effect it would allow a simple caching proxy to be installed at someones office, or wherever that easily does the behavior you describe as well. It could function similarly to pip but be shared amongst multiple people.