3 ms·
> https://py-code.org/download https://py-code.org/download This is a perfect job for task-spooler! :D To mirror your pypi data, I sshed into my server and di
by codetrotter 3y ago
> https://py-code.org/download https://py-code.org/download
This is a perfect job for task-spooler! :D
To mirror your pypi data, I sshed into my server and did this:
mkdir -p ~/src/github.com/pypi-data
cd !$
wget https://raw.githubusercontent.com/pypi-data/data/main/links/repositories.txt
xargs -L1 tsp git clone --bare < repositories.txt
And then I closed the ssh connection to my server, knowing that my server will proceed to mirror all of those repositories of yours one by one :D
- deleted 3y ago[deleted]
- mrwizrd 3y agoYou have very serendipitous timing, I was needing to solve a similar task just the other day and had a hell of a time getting it worked out. Thanks for taking the time to post it! :) Straightforward, but seeing a "worked example" helps a ton.
- henrydark 3y agoI didn't know about task spooler. Is it better than using xargs with a parallel pool? xargs -L1 -P20 git clone --bare < repositories.txt
- codetrotter 3y agoYes, for me it is better because if you do it your way you have to keep your ssh connection open until all of the git clones have been done, which in this case takes several hours. (Or you could also run your way in tmux or screen.) With task-spooler, it puts all of the commands (in this case, the individual git clone commands for each of the repos) in a queue and it runs the commands independently of my ssh session, so I can quickly add a bunch of jobs like this to the queue and immediately disconnect my ssh session.
- allarm 3y agoI’ve recently worked on something similar that involved cloning multiple repositories and found gnu parallel to be ideal for that task. Parallel gives more control than xargs.