5 ms·

Thanks for the feedback! That's coming, we've been prioritizing scaling the index and ingest process and haven't had a chance got add that yet. There are a bunc

by 100k 4y ago

Thanks for the feedback! That's coming, we've been prioritizing scaling the index and ingest process and haven't had a chance got add that yet. There are a bunch of value-add features like this I am looking forward to knocking out soon.

kowlo 4y ago
Well done
BaculumMeumEst 4y ago
thanks so much, you guys are doing amazing work
orf 4y ago
Out of interest, if I have a repo with many millions of files that compress quite nicely down to about a 1.4gb packfile, is it better for the ingestion and/or indexer if I break this down into many smaller pushes or one large push? Because I pushed such a repo yesterday and it’s still not been indexed.
- 100k 4y ago
  Sorry, there are size limits on the number of files in a repositories that we'll index. That is probably why your repository wasn't indexed. Out of curiosity, what are you storing in this repository?
  - orf 4y ago
    Sequences of code from Python package releases from pypi, as an experiment I’m working on. They compress quite nicely as the deltas between releases are fairly small. I thought it would be nice to be able to search through them, but I guess the file limit was reached. That’s a shame.
    - joshuawarner32 4y ago
      Oooh that sounds really useful, would you mind dropping a link?
      - bspammer 4y ago
        Sorry for the cyber-stalking orf, but I'm guessing it's this: https://github.com/orf/pypi-data https://github.com/orf/pypi-data
        orf 4y ago
        It's not - it's something that builds upon that. It's not really ready for prime time yet, but it will be here: https://github.com/orf/pypi-code-repo https://github.com/orf/pypi-code-repo if it works, which I really hope it does, we should have a repo containing every file (sub-25mb) published to pypi. it's not really useful to clone this, but the git packfile + index seems to be pretty ideal for storing + querying this data, reducing 14tb of compressed packages down to like 100gb or so. this should enable large scale exploration of the contents of pypi from only your laptop, which is useful if you want to look at how the python language is evolving (how many packages use f-strings over time?). the rust tool i've built to do the heavy lifting is here: https://github.com/orf/pypi-import-test https://github.com/orf/pypi-import-test. i'm learning a lot about git internals, it's quite fun
jpgvm 4y ago
Is it possible for you to clarify the bigram weight function? I'm assuming it's inverse frequency but that would a be great tidbit to better understand why the covering ngrams work.

by 100k 4y ago

kowlo 4y ago

Well done

BaculumMeumEst 4y ago

thanks so much, you guys are doing amazing work

orf 4y ago

Out of interest, if I have a repo with many millions of files that compress quite nicely down to about a 1.4gb packfile, is it better for the ingestion and/or indexer if I break this down into many smaller pushes or one large push? Because I pushed such a repo yesterday and it’s still not been indexed.

100k 4y ago
Sorry, there are size limits on the number of files in a repositories that we'll index. That is probably why your repository wasn't indexed. Out of curiosity, what are you storing in this repository?
- orf 4y ago
  Sequences of code from Python package releases from pypi, as an experiment I’m working on. They compress quite nicely as the deltas between releases are fairly small. I thought it would be nice to be able to search through them, but I guess the file limit was reached. That’s a shame.
  - joshuawarner32 4y ago
    Oooh that sounds really useful, would you mind dropping a link?
    - bspammer 4y ago
      Sorry for the cyber-stalking orf, but I'm guessing it's this: https://github.com/orf/pypi-data https://github.com/orf/pypi-data
      - orf 4y ago
        It's not - it's something that builds upon that. It's not really ready for prime time yet, but it will be here: https://github.com/orf/pypi-code-repo https://github.com/orf/pypi-code-repo if it works, which I really hope it does, we should have a repo containing every file (sub-25mb) published to pypi. it's not really useful to clone this, but the git packfile + index seems to be pretty ideal for storing + querying this data, reducing 14tb of compressed packages down to like 100gb or so. this should enable large scale exploration of the contents of pypi from only your laptop, which is useful if you want to look at how the python language is evolving (how many packages use f-strings over time?). the rust tool i've built to do the heavy lifting is here: https://github.com/orf/pypi-import-test https://github.com/orf/pypi-import-test. i'm learning a lot about git internals, it's quite fun

jpgvm 4y ago

Is it possible for you to clarify the bigram weight function? I'm assuming it's inverse frequency but that would a be great tidbit to better understand why the covering ngrams work.