3 ms·

Sorry, there are size limits on the number of files in a repositories that we'll index. That is probably why your repository wasn't indexed. Out of curiosity, w

• orf 4y ago
  Sequences of code from Python package releases from pypi, as an experiment I’m working on. They compress quite nicely as the deltas between releases are fairly small. I thought it would be nice to be able to search through them, but I guess the file limit was reached. That’s a shame.
  • joshuawarner32 4y ago
    Oooh that sounds really useful, would you mind dropping a link?
    • bspammer 4y ago
      Sorry for the cyber-stalking orf, but I'm guessing it's this: https://github.com/orf/pypi-data https://github.com/orf/pypi-data
      • orf 4y ago
        It's not - it's something that builds upon that. It's not really ready for prime time yet, but it will be here: https://github.com/orf/pypi-code-repo https://github.com/orf/pypi-code-repo if it works, which I really hope it does, we should have a repo containing every file (sub-25mb) published to pypi. it's not really useful to clone this, but the git packfile + index seems to be pretty ideal for storing + querying this data, reducing 14tb of compressed packages down to like 100gb or so. this should enable large scale exploration of the contents of pypi from only your laptop, which is useful if you want to look at how the python language is evolving (how many packages use f-strings over time?). the rust tool i've built to do the heavy lifting is here: https://github.com/orf/pypi-import-test https://github.com/orf/pypi-import-test. i'm learning a lot about git internals, it's quite fun