5 ms·
From the article: "The new gcloud storage CLI offers significant performance improvements over the existing gsutil ..." It makes sense to see this comparison
by rsync 4y ago
From the article:
"The new gcloud storage CLI offers significant performance improvements over the existing gsutil ..."
It makes sense to see this comparison because 'gsutil' was relatively poor - we[1] made a decision to deprecate the 'gsutil' binary[2] in our environment in favor of rclone[3] because rclone is better in every way.
We made the same decision for 's3cmd'.
All of that to say:
The biggest question to ask about this is "how does it compare to rclone". I suspect the answer is that authentication and token handling (and things like that) are much better as rclone has a pretty clunky auth workflow for google cloud resources. That is not the case for Amazon resources which behave just as you'd expect them to, with regard to API tokens, and make it hard to justify using any tool other than rclone.
[1] rsync.net
[2] Yes, a binary - we "freeze" python into binary exe because we don't allow interpreters in our environment.
[3] https://rclone.org/ https://rclone.org/
- xen2xen1 4y agoRclone is clunky because it works with everything, even things like OneDrive on Linux, which is not supported officially. Love rclone. Also just a great copy / move utility.
- kldx 4y agoHow do you freeze python tools into binaries?
- kleton 4y agopar- I assume this is the open sourced version https://github.com/google/subpar https://github.com/google/subpar
- rsync 4y ago"How do you freeze python tools into binaries?" I think there are several workflows for doing this - I know in the past we used something called 'cx_Freeze' but I think the person in charge of this is now using 'py2exe' ? I think the 'borg' project packages up their single file .exe distribution with 'pyinstaller' but I may not fully understand that part of their process...
- icelancer 4y agoI use py2exe with pretty good success in deployments at my company. It's bloated and such but sometimes it's just the easiest path forward.
- judge2020 4y agoPyinstaller is the most actively maintained and in-use, but nuitka is on the rise as well. There are less-active but still serviceable ones like py2exe as well https://pyinstaller.org/en/stable/usage.html https://pyinstaller.org/en/stable/usage.html https://www.nuitka.net/ https://www.nuitka.net/
- faizshah 4y agoI use s5cmd, it’s a fast parallel s3 cli client: https://github.com/peak/s5cmd https://github.com/peak/s5cmd Haven’t heard of rclone before so can’t compare yet but s5cmd is supposed to work against GCS and other s3 compatible apis.
- neuronexmachina 4y agoI'd also be curious about how rclone's parallelization compares to gcloud storage's in practice: > Second, it utilizes a new parallelization strategy that treats task management as a graph problem, which allows more work to be done in parallel with far less overhead. > ... In gcloud storage, all operations happen in parallel. As an example, parallel composite uploads are enabled automatically based on bucket configuration. This is a vast improvement compared to gsutil, which requires the -m (parallel operations) flag to improve the performance for uploads and downloads.
- xani_ 4y agohow faster is rclone compared to s3cmd ? We're using s3cmd to sync some stuff for backup purpose (many many small-ish) files and performance has been generally mediocre
- tebbers 4y agoWe often have to sync hundreds of thousands of files to S3 and we use this which often hits 300 files a second on a 1Gbps link: https://github.com/mishudark/s3-parallel-put https://github.com/mishudark/s3-parallel-put
- TheReveller 4y agoHow are gcp and aws different in terms of auth required by rclone?
- flanked-evergl 4y agoI have also had persistent issues with gsutil and have used rclone instead, rclone is clunky but it has been an ever increasing part of our toolkit and I don't see that changing soon.