3 ms·

Show HN: Downloading a folder from a repo using rust

10 points by sonderotis 1y ago

luismedel 1y ago
So, $ git-down -d bootstrap-dist https://github.com/twbs/bootstrap.git:master dist is *way* better than $ git clone --depth 1 https://github.com/twbs/boostrap.git $ cd bootstrap $ mv ./dist ~/stuff/boostrap-latest because: "C'mon, you don't have the time to be doing all that." And then we wonder how the fuck we end with malware in our systems.
- panki27 1y ago
  All that's missing is `curl ... | sudo bash` as install instruction in the README
panki27 1y ago
$ git clone --no-checkout $URL/repo.git $ cd repo/ $ git sparse-checkout init $ git sparse-checkout set subdirectory_i_want $ git checkout main
- cakoose 1y ago
  Now I'm curious -- is that here a way to do this that avoids downloading any more than strictly necessary? The command above downloads the whole repo history. You could do a depth=1 to skip the history, but it still downloads the he latest version of the entire repo tree.
  - vient 1y ago
    git-archive downloads only strictly necessary files but is not universally supported https://git-scm.com/docs/git-archive https://git-scm.com/docs/git-archive
  - craftkiller 1y ago
    You could do a blobless or treeless clone https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/ https://github.blog/open-source/git/get-up-to-speed-with-par... Combined with --depth=1 and the --no-checkout / --sparse-checkout flow that the GP already described. I just tested on the emacs repo, left column is disk usage of just the `.git` folder inside: Shallow clones (depth=1): 124K: Treeless clone depth=1 with no-checkout 308K: Blobless clone depth=1 with no-checkout 12M: Treeless clone depth=1 sparse checkout of "doc" folder 12M: Blobless clone depth=1 sparse checkout of "doc" folder 53M: Treeless clone depth=1 non-sparse full checkout 53M: Blobless clone depth=1 non-sparse full checkout 53M: Regular clone with depth=1 Non-shallow clones: 54M: Treeless clone with no-checkout 124M: Blobless clone with no-checkout 65M: Treeless clone sparse checkout of "doc" folder 135M: Blobless clone sparse checkout of "doc" folder 107M: Treeless clone with non-sparse full checkout 177M: Blobless clone with non-sparse full checkout 653M: Full regular git clone with no flags Great tech talk covering some of the newer lesser-known git features: https://www.youtube.com/watch?v=aolI_Rz0ZqY https://www.youtube.com/watch?v=aolI_Rz0ZqY
- archargelod 1y ago
  The rust app just calls a few git commands too[1] Could've been a shell script[2] [1] https://github.com/zikani03/git-down/blob/cb2763020edc81e4649af56d9e405bd7ff8421a0/src/git.rs#L61C1-L70C36 https://github.com/zikani03/git-down/blob/cb2763020edc81e464... [2] https://textbin.net/ja17q8vga4 https://textbin.net/ja17q8vga4
vient 1y ago
Same can be done using git and tar mkdir -p <out_dir> && git archive --remote=<remote> --format=tar.gz <branch> <files...> | tar -xzC <out_dir> Strangely, github does not support this, so tested with bitbucket.
koakuma-chan 1y ago
I have a similar problem with hugging face. I do git clone and it doesn't download models. I know it's supposed to use LFS, but I don't know how to make it work, I tried everything. I had to install their disgusting Python CLI to download a model.
- 0fflineuser 1y ago
  Don't you just need to install git-lfs https://git-lfs.com/ https://git-lfs.com/ and then run `git lfs pull` ?
  - koakuma-chan 1y ago
    Oh I guess I didn't run `git lfs pull` lol thanks
pornel 1y ago
There's hardly any Rust in there. It's shelling out to the git command. This could have been a couple lines of bash. Actually doing this in Rust with lower-level libraries like gix would have been interesting.
- zikani_03 1y ago
  I've found that using Git libraries directly is usually slow and less ergonomic. I've got another tool (written in Go) here: https://github.com/zikani03/git-monorepo https://github.com/zikani03/git-monorepo which uses a Git library and is a bit slow. But I'm willing to take up that challenge and test out gix, I've regained my interest in Rust so the timing is good :)
tsylba 1y ago
You guys using convoluted git commands when a single line of subversion works: svn checkout https://github.com/tensorflow/tensorflow/trunk/tensorflow/examples https://github.com/tensorflow/tensorflow/trunk/tensorflow/ex...
- deleted 1y ago
  [deleted]
- zenmac 1y ago
  Link is broken
- craftkiller 1y ago
  Yes, subversion wins for this ONE specific use case. It's such a shame subversion is miserable to use in every other use case.
pharrington 1y ago
Since this is a "Show HN," did you mean to post using your https://news.ycombinator.com/user?id=zikani_03 https://news.ycombinator.com/user?id=zikani_03 account?
- zikani_03 1y ago
  I'm not the one who posted, thanks to the OP for sharing though :)

10 points by sonderotis 1y ago

So, $ git-down -d bootstrap-dist https://github.com/twbs/bootstrap.git:master dist is *way* better than $ git clone --depth 1 https://github.com/twbs/boostrap.git $ cd bootstrap $ mv ./dist ~/stuff/boostrap-latest because: "C'mon, you don't have the time to be doing all that." And then we wonder how the fuck we end with malware in our systems.

panki27 1y ago
All that's missing is `curl ... | sudo bash` as install instruction in the README

panki27 1y ago

$ git clone --no-checkout $URL/repo.git $ cd repo/ $ git sparse-checkout init $ git sparse-checkout set subdirectory_i_want $ git checkout main

cakoose 1y ago
Now I'm curious -- is that here a way to do this that avoids downloading any more than strictly necessary? The command above downloads the whole repo history. You could do a depth=1 to skip the history, but it still downloads the he latest version of the entire repo tree.
- vient 1y ago
  git-archive downloads only strictly necessary files but is not universally supported https://git-scm.com/docs/git-archive https://git-scm.com/docs/git-archive
- craftkiller 1y ago
  You could do a blobless or treeless clone https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/ https://github.blog/open-source/git/get-up-to-speed-with-par... Combined with --depth=1 and the --no-checkout / --sparse-checkout flow that the GP already described. I just tested on the emacs repo, left column is disk usage of just the `.git` folder inside: Shallow clones (depth=1): 124K: Treeless clone depth=1 with no-checkout 308K: Blobless clone depth=1 with no-checkout 12M: Treeless clone depth=1 sparse checkout of "doc" folder 12M: Blobless clone depth=1 sparse checkout of "doc" folder 53M: Treeless clone depth=1 non-sparse full checkout 53M: Blobless clone depth=1 non-sparse full checkout 53M: Regular clone with depth=1 Non-shallow clones: 54M: Treeless clone with no-checkout 124M: Blobless clone with no-checkout 65M: Treeless clone sparse checkout of "doc" folder 135M: Blobless clone sparse checkout of "doc" folder 107M: Treeless clone with non-sparse full checkout 177M: Blobless clone with non-sparse full checkout 653M: Full regular git clone with no flags Great tech talk covering some of the newer lesser-known git features: https://www.youtube.com/watch?v=aolI_Rz0ZqY https://www.youtube.com/watch?v=aolI_Rz0ZqY
archargelod 1y ago
The rust app just calls a few git commands too[1] Could've been a shell script[2] [1] https://github.com/zikani03/git-down/blob/cb2763020edc81e4649af56d9e405bd7ff8421a0/src/git.rs#L61C1-L70C36 https://github.com/zikani03/git-down/blob/cb2763020edc81e464... [2] https://textbin.net/ja17q8vga4 https://textbin.net/ja17q8vga4

vient 1y ago

Same can be done using git and tar mkdir -p <out_dir> && git archive --remote=<remote> --format=tar.gz <branch> <files...> | tar -xzC <out_dir> Strangely, github does not support this, so tested with bitbucket.

koakuma-chan 1y ago

I have a similar problem with hugging face. I do git clone and it doesn't download models. I know it's supposed to use LFS, but I don't know how to make it work, I tried everything. I had to install their disgusting Python CLI to download a model.

0fflineuser 1y ago
Don't you just need to install git-lfs https://git-lfs.com/ https://git-lfs.com/ and then run `git lfs pull` ?
- koakuma-chan 1y ago
  Oh I guess I didn't run `git lfs pull` lol thanks

pornel 1y ago

There's hardly any Rust in there. It's shelling out to the git command. This could have been a couple lines of bash. Actually doing this in Rust with lower-level libraries like gix would have been interesting.

zikani_03 1y ago
I've found that using Git libraries directly is usually slow and less ergonomic. I've got another tool (written in Go) here: https://github.com/zikani03/git-monorepo https://github.com/zikani03/git-monorepo which uses a Git library and is a bit slow. But I'm willing to take up that challenge and test out gix, I've regained my interest in Rust so the timing is good :)

tsylba 1y ago

You guys using convoluted git commands when a single line of subversion works: svn checkout https://github.com/tensorflow/tensorflow/trunk/tensorflow/examples https://github.com/tensorflow/tensorflow/trunk/tensorflow/ex...

deleted 1y ago
[deleted]
zenmac 1y ago
Link is broken
craftkiller 1y ago
Yes, subversion wins for this ONE specific use case. It's such a shame subversion is miserable to use in every other use case.

pharrington 1y ago

Since this is a "Show HN," did you mean to post using your https://news.ycombinator.com/user?id=zikani_03 https://news.ycombinator.com/user?id=zikani_03 account?

zikani_03 1y ago
I'm not the one who posted, thanks to the OP for sharing though :)