6 ms·

This is a good idea! I used the subdomain because I plan on hosting the "simple" interface in EC2 (Packages are hosted on S3/CloudFront) in multiple availabilit

This is a good idea! I used the subdomain because I plan on hosting the "simple" interface in EC2 (Packages are hosted on S3/CloudFront) in multiple availability zones while my main site is hosted on Gondor.io. However I could enable a short url that redirected to simple.crate.io to allow easier command line access. Thanks!

po 15y ago
Is the expectation that someone should be able to do what I described? Run a local copy that proxies 404's up to the main crate.io server and then caches the result locally to disk? This would be a very nice way to deal with build-time dependencies. You can get similar behavior today by setting multiple mirrors in pip but this would allow the grabbing packages on-demand instead of having to mirror everything.
- donaldstufft 15y ago
  pip supports a download cache (off by default) that stores packages as their urlname. It works similar to how you describe. It will see it needs the package at url xyz.com/package.tar.gz, it'll check it's download cache and if it doesn't exist there it will download the file from the url and save it to the cache. The cacheable urls stem from the fact that currently on PyPI you can delete an existing file you've uploaded, and reupload a file of the same name. Pip (and any other cache like varnish etc) isn't notified of the fact that what used to be at url xyz.com/packages.tar.gz is gone and now something else is at xyz.com/packages.tar.gz. This can cause issues with installing packages, especially on remote deployment infrastructures. With crate I place the sha256 of the file in the url, so if someone deletes + uploads the same name, it should still (unless you get (un)lucky and get a collision somehow) have a completely different url, so pip and other caches will see it as a different file. As a side effect it would allow a simple caching proxy to be installed at someones office, or wherever that easily does the behavior you describe as well. It could function similarly to pip but be shared amongst multiple people.

po 15y ago

Is the expectation that someone should be able to do what I described? Run a local copy that proxies 404's up to the main crate.io server and then caches the result locally to disk? This would be a very nice way to deal with build-time dependencies. You can get similar behavior today by setting multiple mirrors in pip but this would allow the grabbing packages on-demand instead of having to mirror everything.

donaldstufft 15y ago
pip supports a download cache (off by default) that stores packages as their urlname. It works similar to how you describe. It will see it needs the package at url xyz.com/package.tar.gz, it'll check it's download cache and if it doesn't exist there it will download the file from the url and save it to the cache. The cacheable urls stem from the fact that currently on PyPI you can delete an existing file you've uploaded, and reupload a file of the same name. Pip (and any other cache like varnish etc) isn't notified of the fact that what used to be at url xyz.com/packages.tar.gz is gone and now something else is at xyz.com/packages.tar.gz. This can cause issues with installing packages, especially on remote deployment infrastructures. With crate I place the sha256 of the file in the url, so if someone deletes + uploads the same name, it should still (unless you get (un)lucky and get a collision somehow) have a completely different url, so pip and other caches will see it as a different file. As a side effect it would allow a simple caching proxy to be installed at someones office, or wherever that easily does the behavior you describe as well. It could function similarly to pip but be shared amongst multiple people.