5 ms·
Gcloud storage: Faster data transfers for Cloud Storage
- keltex 4y agoLooks like they are missing the useful "rsync" command from gsutil (Synchronize content of two buckets/directories).
- rsync 4y agoFrom the article: "The new gcloud storage CLI offers significant performance improvements over the existing gsutil ..." It makes sense to see this comparison because 'gsutil' was relatively poor - we[1] made a decision to deprecate the 'gsutil' binary[2] in our environment in favor of rclone[3] because rclone is better in every way. We made the same decision for 's3cmd'. All of that to say: The biggest question to ask about this is "how does it compare to rclone". I suspect the answer is that authentication and token handling (and things like that) are much better as rclone has a pretty clunky auth workflow for google cloud resources. That is not the case for Amazon resources which behave just as you'd expect them to, with regard to API tokens, and make it hard to justify using any tool other than rclone. [1] rsync.net [2] Yes, a binary - we "freeze" python into binary exe because we don't allow interpreters in our environment. [3] https://rclone.org/ https://rclone.org/
- xen2xen1 4y agoRclone is clunky because it works with everything, even things like OneDrive on Linux, which is not supported officially. Love rclone. Also just a great copy / move utility.
- kldx 4y agoHow do you freeze python tools into binaries?
- kleton 4y agopar- I assume this is the open sourced version https://github.com/google/subpar https://github.com/google/subpar
- rsync 4y ago"How do you freeze python tools into binaries?" I think there are several workflows for doing this - I know in the past we used something called 'cx_Freeze' but I think the person in charge of this is now using 'py2exe' ? I think the 'borg' project packages up their single file .exe distribution with 'pyinstaller' but I may not fully understand that part of their process...
- icelancer 4y agoI use py2exe with pretty good success in deployments at my company. It's bloated and such but sometimes it's just the easiest path forward.
- judge2020 4y agoPyinstaller is the most actively maintained and in-use, but nuitka is on the rise as well. There are less-active but still serviceable ones like py2exe as well https://pyinstaller.org/en/stable/usage.html https://pyinstaller.org/en/stable/usage.html https://www.nuitka.net/ https://www.nuitka.net/
- faizshah 4y agoI use s5cmd, it’s a fast parallel s3 cli client: https://github.com/peak/s5cmd https://github.com/peak/s5cmd Haven’t heard of rclone before so can’t compare yet but s5cmd is supposed to work against GCS and other s3 compatible apis.
- neuronexmachina 4y agoI'd also be curious about how rclone's parallelization compares to gcloud storage's in practice: > Second, it utilizes a new parallelization strategy that treats task management as a graph problem, which allows more work to be done in parallel with far less overhead. > ... In gcloud storage, all operations happen in parallel. As an example, parallel composite uploads are enabled automatically based on bucket configuration. This is a vast improvement compared to gsutil, which requires the -m (parallel operations) flag to improve the performance for uploads and downloads.
- xani_ 4y agohow faster is rclone compared to s3cmd ? We're using s3cmd to sync some stuff for backup purpose (many many small-ish) files and performance has been generally mediocre
- tebbers 4y agoWe often have to sync hundreds of thousands of files to S3 and we use this which often hits 300 files a second on a 1Gbps link: https://github.com/mishudark/s3-parallel-put https://github.com/mishudark/s3-parallel-put
- TheReveller 4y agoHow are gcp and aws different in terms of auth required by rclone?
- flanked-evergl 4y agoI have also had persistent issues with gsutil and have used rclone instead, rclone is clunky but it has been an ever increasing part of our toolkit and I don't see that changing soon.
- unilynx 4y agogsutil hardly ever finished when trying to download a couple-hundred google workspace takeout, preferring to consume all available memory instead - so that's a pretty easy thing to beat. does it outperform rclone?
- seabrookmx 4y agoThis was inevitable it seems. If you look under the hood gsutil is a wrapper around a vendored version of boto2. boto3 is a big departure and never got support for Google Cloud Storage. Also the fact that it was separate from the rest of the gcloud CLI was a weird quirk. This fixes both of those issues.
- ZeroCool2u 4y ago"These tests have been performed on Google Cloud Platform using n2d-standard-16 (8 vCPUs, 32 GB memory) and 1x375GB NVME in RAID0 in us-east4." Am I missing something? Is there some configuration of RAID0 that allows you to use a single drive? Or is this just a confusing way of saying they have one logical drive available, because the two attached NVME drives are configured in RAID0?
- gtirloni 4y agoYou can create a RAID-0 volume on top of a single drive. But it seems this was a typo in the article.
- Latty 4y agoThis could just be an ambiguous wording thing, i.e: it's one 375GB effective volume, backed by NVME in RAID0. I dunno how it works, but depending on how it's sold (e.g: you choose the size you need, then the redundancy/performance you want, rather than having to define how that is constructed at a physical level), it may make sense to talk about it that way.
- mooman219 4y agoI actually worked on gsutil a while ago! I added in flight compression -J/j. Glad to see gsutil is on its way out.
- titanomachy 4y agoHow did you choose the letter "j"? were all the more obvious ones just taken?
- rwiggins 4y agoMaybe the inspiration was `-j` for `tar`, which enables bzip2 compression?
- hinkley 4y agoNow we just have to explain why -j means bzip2 for 'tar'...
- mooman219 4y agoThe original intention for most of the flags was to keep them aligned with rsync. The feature only compressed the data in flight, as opposed to -Z/z which stored the compressed version. This didn't directly align to anything in rsync, so I picked an unused letter in rsync (-J/j). In retrospect, it might have had a better home as some option on -Z/z, or if I had the tool figure out if you were bottlenecked on bandwidth, spare compute, and your data compressed well, and apply it automatically.
- hinkley 4y agoI don't know how long this flag has been there, but -z does mean compression in-flight in rsync. > -z, --compress compress file data during the transfer
- mooman219 4y agoYou're totally right! If anything the gsutil's -Z/z doesn't align with rsync's -Z/z, and gsutil's -J/j is a better match for what rsync's -Z/z does. I added -J/j well after -Z/z was there so it's something that we have to life with unfortunately.
- nomadiccoder 4y agoAlso a lot more convenient to use with WIF
- svarovski 4y ago[dead]
- alpb 4y agoDoes anyone know if the "gcloud storage" vs "gsutil" chart in the article uses "gsutil -m" (parallel mode of the tool)?
- deleted 4y ago[deleted]
- davecamp1717 4y agoWow, feel good to see good news from GCP
- carbocation 4y agoI initially misunderstood the article's explanation of the gsutil shim: namely, I thought it was "turned on" by default, but that is not the case. One either needs to apply the shim, or to start using `gcloud storage` instead of `gsutil`.
- fragmede 4y agoConnect it up with Skyplane and you’ll really be speeding along https://medium.com/@paras_jain/skyplane-110x-faster-data-transfers-on-any-cloud-8f0165c1d711 https://medium.com/@paras_jain/skyplane-110x-faster-data-tra...
- capableweb 4y agoI'm interested to see more benchmarks. The posted numbers in that blogpost seems too good to be true, is tooling really so far behind that a single tool could surpass the transfer time by that much? The architecture goes through some of the methodology: https://skyplane.org/en/latest/architecture.html https://skyplane.org/en/latest/architecture.html which is what I'm currently reading through and trying to understand.
- fragmede 4y agoIt “cheats” by setting up a VM in GCP and another in AWS and does a direct transfer between the two instead of having to upload and download on whatever local connection you’ve got.
- wmf 4y agoThey're comparing against AWS DataSync which I assume also does not go through your local computer either.
- xchip 4y ago94% faster than what? and "up to"? We already know the "up to" trick...
- rippercushions 4y agoFaster than the old gsutil utility. The speed increase is real and easily measurable, but will depend on your files (lots of small ones vs a few huge ones etc).
- nullbyte808 4y agoWould be nice getting these speeds on Google drive!
- Dowwie 4y agoIt's in the article but the tldr is that they split the file into 32 chunks, upload the chunks in parallel, and reassemble when uploading is complete.
- jethkl 4y agoThis triggered a memory from my distant (12+ years ago) past: Globus Grid FTP had this capability, and it was awesome for bulk data transfer [1]. [1] https://docs.globus.org/globus-connect-server/v4/man/globus-gridftp-server/ https://docs.globus.org/globus-connect-server/v4/man/globus-...
- anshumankmr 4y agoI have not worked with gcloud storage command as I am only familiar with gsutil, but is the syntax the same? Also, is there a comparison with Terraform?
- awestroke 4y agoHow long until it's discontinued?
- adammarples 4y agoGcloud has been around a long time and will probably be there as long as GCP is