30 ms·

S5cmd: Parallel S3 and local filesystem execution tool

62 points by polyrand 1y ago

nodesocket 1y ago
I posted a link to s5cmd in the other post about s3mini. Really want to try it out migrating a few TBs of s3 data.
therealmarv 1y ago
Very interesting. And this is an amazing graph for small file uploading/download speed improvement. I have the feeling that all cloud drives are really not optimised for many small files like smaller than 1Mbytes in average. https://raw.githubusercontent.com/peak/s5cmd/master/doc/benchmark2.png https://raw.githubusercontent.com/peak/s5cmd/master/doc/benc... I've implemented at work once a rudimentary parallel uploading of many small files to S3 in Python and with boto3 (was not allowed to use a third party library or tool at that time) because it's soooo slow to upload many small files to S3. It really takes ages and even if you just upload 8 small files in parallel it makes a huge difference.
- random_savv 1y ago
  Most S3 systems are terrible with small files (~50KB). 1MB can still be ok though
Galanwe 1y ago
> For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s) I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3. My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 file is capped at 1.6GB/s (at least per ec2 instance). At least I have never managed myself nor seen any tool capable of going beyond that. My understanding is then that this benchmark's claims of 4.3GB/s are across multiple files, but then it would be rather meaningless, as it's free concurrency basically.
- deleted 1y ago
  [deleted]
- jeffbee 1y ago
  I think that's the point of this tool, that pipelining and parallelism are necessary to get high performance out of S3.
- deleted 1y ago
  [deleted]
- nckslvrmn 1y ago
  I cant verify 40Gbps because I have never had access to a pipe that fast, but when I ran this tool on an AWS instance with a 20Gbps connection, it saturated that easily and maintained that speed for the duration of the transfer.
  - Galanwe 1y ago
    I just spawned a r6a.16xlarge with a 25gbps NIC, created a 10GB file which I uploaded to an S3 bucket in the same region, through a local S3 VPC endpoint. Downloading that 10GB file to /dev/shm with s5cmd took 24s, all while spawning 20 or so threads which were all idling for IO. The same test using a Python tool (http://github.com/NewbiZ/s3pd http://github.com/NewbiZ/s3pd) with the same amount of workers took 10s. Cranking up the worker count of the latter library until there is no more speedup, I can reach 6s with 80 workers. That is, 10/6 = 1.6GB/s, which seems to confirm my previous comment. What am I doing wrong ?
    - Galanwe 1y ago
      Okay I found the trick, buried in the benchmark setup of s5cmd. The claimed numbers are _not_ reached with S3, but rather from a custom server emulating the S3 API, hosted on the client machine. I think this is very misleading, since these benchmark numbers are not reachable in any real life scenario. It also shows that there is very little point in using s5cmd compared to other tools, since beyond 1.6GB/s the throttling will be from S3, not from the client, so any tool able to saturate 1.6GB/s will be enough.
hotpocket777 1y ago
[flagged]
- nckslvrmn 1y ago
  absolutely, though in my experience this is truly the fastest piece of software I have ever used (to transfer files over a network).
- beastman82 1y ago
  this tool crushes the aws cli by an order of magnitude. So in this case it's warranted.
rsync 1y ago
If you don't want to install, or maintain, s5cmd yourself: ssh user@rsync.net s5cmd Password: NAME: s5cmd - Blazing fast S3 and local filesystem execution tool USAGE: s5cmd [global options] command [command options] [arguments...] If you move data between clouds, you don't need to use any of your own bandwidth ...
- rdg42 1y ago
  s5cmd is not listed here https://www.rsync.net/resources/howto/remote_commands.html https://www.rsync.net/resources/howto/remote_commands.html ? :-)
  - vasco 1y ago
    It either was added in the meantime or it already was there > Backup Tools > attic, borg (usage notes), rclone, unison, s5cmd and git are installed (on our server side). > You would most likely call these commands by running these tools locally and connecting to rsync.net with them.
StackTopherFlow 1y ago
Awesome work. I love seeing a community project step up and make a better solution than a multi trillion dollar company.
deleted 1y ago
[deleted]
quodlibetor 1y ago
I recently wrote a similar tool focused more on optimizing the case of exploring millions or billions of objects when you know a few aspects of the path: https://github.com/quodlibetor/s3glob https://github.com/quodlibetor/s3glob It supports glob patterns like so, and will do smart filtering at every stage possible: */2025-0[45]-*/user*/*/object.txt I haven't done real benchmarks, but it's parallel enough to hit s3 parallel request limits/file system open file limits when downloading.*
BlackLotus89 1y ago
If you want parallel upload/download over s3 and other protocols (GDrive, ftp, WebDAV, ...) you can use rclone. For s3 mounts I would use geesefs. Have to later take a look at s5cmd as well...
smpretzer 1y ago
I have used s5cmd in a professional setting and it works wonderfully. I have never attempted to test performance to confirm their claims, but as an out of the box client, it is (anecdotally) significantly faster than anything else I have tried. My only headache was that I was invoking it from python, and it does not have bindings, so I had to write a custom wrapper to call out to it. I am not sure of the difficulty of adding native support for Python, but I assume its not worth the squeeze and just calling out to a subprocess will work for most user's needs.
remram 1y ago
What is meant by "filesystem execution" here? Is it just an S3 file transfer tool?

62 points by polyrand 1y ago

I posted a link to s5cmd in the other post about s3mini. Really want to try it out migrating a few TBs of s3 data.

Very interesting. And this is an amazing graph for small file uploading/download speed improvement. I have the feeling that all cloud drives are really not optimised for many small files like smaller than 1Mbytes in average. https://raw.githubusercontent.com/peak/s5cmd/master/doc/benchmark2.png https://raw.githubusercontent.com/peak/s5cmd/master/doc/benc... I've implemented at work once a rudimentary parallel uploading of many small files to S3 in Python and with boto3 (was not allowed to use a third party library or tool at that time) because it's soooo slow to upload many small files to S3. It really takes ages and even if you just upload 8 small files in parallel it makes a huge difference.

random_savv 1y ago
Most S3 systems are terrible with small files (~50KB). 1MB can still be ok though

Galanwe 1y ago

> For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s) I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3. My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 file is capped at 1.6GB/s (at least per ec2 instance). At least I have never managed myself nor seen any tool capable of going beyond that. My understanding is then that this benchmark's claims of 4.3GB/s are across multiple files, but then it would be rather meaningless, as it's free concurrency basically.

deleted 1y ago
[deleted]
jeffbee 1y ago
I think that's the point of this tool, that pipelining and parallelism are necessary to get high performance out of S3.
deleted 1y ago
[deleted]
nckslvrmn 1y ago
I cant verify 40Gbps because I have never had access to a pipe that fast, but when I ran this tool on an AWS instance with a 20Gbps connection, it saturated that easily and maintained that speed for the duration of the transfer.
- Galanwe 1y ago
  I just spawned a r6a.16xlarge with a 25gbps NIC, created a 10GB file which I uploaded to an S3 bucket in the same region, through a local S3 VPC endpoint. Downloading that 10GB file to /dev/shm with s5cmd took 24s, all while spawning 20 or so threads which were all idling for IO. The same test using a Python tool (http://github.com/NewbiZ/s3pd http://github.com/NewbiZ/s3pd) with the same amount of workers took 10s. Cranking up the worker count of the latter library until there is no more speedup, I can reach 6s with 80 workers. That is, 10/6 = 1.6GB/s, which seems to confirm my previous comment. What am I doing wrong ?
  - Galanwe 1y ago
    Okay I found the trick, buried in the benchmark setup of s5cmd. The claimed numbers are _not_ reached with S3, but rather from a custom server emulating the S3 API, hosted on the client machine. I think this is very misleading, since these benchmark numbers are not reachable in any real life scenario. It also shows that there is very little point in using s5cmd compared to other tools, since beyond 1.6GB/s the throttling will be from S3, not from the client, so any tool able to saturate 1.6GB/s will be enough.

hotpocket777 1y ago

[flagged]

nckslvrmn 1y ago
absolutely, though in my experience this is truly the fastest piece of software I have ever used (to transfer files over a network).
beastman82 1y ago
this tool crushes the aws cli by an order of magnitude. So in this case it's warranted.

rsync 1y ago

If you don't want to install, or maintain, s5cmd yourself: ssh user@rsync.net s5cmd Password: NAME: s5cmd - Blazing fast S3 and local filesystem execution tool USAGE: s5cmd [global options] command [command options] [arguments...] If you move data between clouds, you don't need to use any of your own bandwidth ...

rdg42 1y ago
s5cmd is not listed here https://www.rsync.net/resources/howto/remote_commands.html https://www.rsync.net/resources/howto/remote_commands.html ? :-)
- vasco 1y ago
  It either was added in the meantime or it already was there > Backup Tools > attic, borg (usage notes), rclone, unison, s5cmd and git are installed (on our server side). > You would most likely call these commands by running these tools locally and connecting to rsync.net with them.

StackTopherFlow 1y ago

Awesome work. I love seeing a community project step up and make a better solution than a multi trillion dollar company.

deleted 1y ago

[deleted]

quodlibetor 1y ago

I recently wrote a similar tool focused more on optimizing the case of exploring millions or billions of objects when you know a few aspects of the path: https://github.com/quodlibetor/s3glob https://github.com/quodlibetor/s3glob It supports glob patterns like so, and will do smart filtering at every stage possible: */2025-0[45]-*/user*/*/object.txt I haven't done real benchmarks, but it's parallel enough to hit s3 parallel request limits/file system open file limits when downloading.*

BlackLotus89 1y ago

If you want parallel upload/download over s3 and other protocols (GDrive, ftp, WebDAV, ...) you can use rclone. For s3 mounts I would use geesefs. Have to later take a look at s5cmd as well...

smpretzer 1y ago

I have used s5cmd in a professional setting and it works wonderfully. I have never attempted to test performance to confirm their claims, but as an out of the box client, it is (anecdotally) significantly faster than anything else I have tried. My only headache was that I was invoking it from python, and it does not have bindings, so I had to write a custom wrapper to call out to it. I am not sure of the difficulty of adding native support for Python, but I assume its not worth the squeeze and just calling out to a subprocess will work for most user's needs.

remram 1y ago

What is meant by "filesystem execution" here? Is it just an S3 file transfer tool?