3 ms·

One thing I noticed when I went digging through the s3fs source trying to answer that myself is that it's uploader is single threaded. Part of what makes AWS's

One thing I noticed when I went digging through the s3fs source trying to answer that myself is that it's uploader is single threaded. Part of what makes AWS's cli reasonably fast is that it uses several threads to make sure to use as much bandwidth as possible. If I limit AWS's tool to one thread on a box with plenty of bandwidth, it turns my upload down from around 135 MiB/s to around 20 MiB/s Not sure if that's all of it, or even the majority of the slowness.

gaul 6y ago
Could you expand on this comment? s3fs uses multiple threads for uploading and populating readdir metadata via S3fsMultiCurl::MultiPerform. Earlier versions used curl_multi_perform which may have hidden some of the parallelism from you.
- banana_giraffe 6y ago
  Huh, right you are. I did a cursory search, but whatever I saw clearly was wrong. That said, something is going on that I can't explain: # Default settings for aws $ time aws s3 cp /dev/shm/1GB.bin s3://test-kihaqtowex/a upload: ../../dev/shm/1GB.bin to s3://test-kihaqtowex/a real 0m10.312s user 0m5.909s sys 0m4.204s $ aws configure set default.s3.max_concurrent_requests 4 $ time aws s3 cp /dev/shm/1GB.bin s3://test-kihaqtowex/b upload: ../../dev/shm/1GB.bin to s3://test-kihaqtowex/b real 0m26.732s user 0m4.989s sys 0m2.741s $ time cp /dev/shm/1GB.bin s3fs_mount_-kihaqtowex/c real 0m26.368s user 0m0.006s sys 0m0.699s (I did multiple runs, though not hundreds so I can't say with certainty this would hold) I swear the difference was worse in the past, so maybe things have improved. Still not worth the added slowness to me for my use cases.
  - gaul 6y ago
    Thanks for testing! You may improve performance via increasing -o parallel_count (default="5"). Newer versions of s3fs have improved performance but please report any cliffs at https://github.com/s3fs-fuse/s3fs-fuse/issues https://github.com/s3fs-fuse/s3fs-fuse/issues .

gaul 6y ago

Could you expand on this comment? s3fs uses multiple threads for uploading and populating readdir metadata via S3fsMultiCurl::MultiPerform. Earlier versions used curl_multi_perform which may have hidden some of the parallelism from you.

banana_giraffe 6y ago
Huh, right you are. I did a cursory search, but whatever I saw clearly was wrong. That said, something is going on that I can't explain: # Default settings for aws $ time aws s3 cp /dev/shm/1GB.bin s3://test-kihaqtowex/a upload: ../../dev/shm/1GB.bin to s3://test-kihaqtowex/a real 0m10.312s user 0m5.909s sys 0m4.204s $ aws configure set default.s3.max_concurrent_requests 4 $ time aws s3 cp /dev/shm/1GB.bin s3://test-kihaqtowex/b upload: ../../dev/shm/1GB.bin to s3://test-kihaqtowex/b real 0m26.732s user 0m4.989s sys 0m2.741s $ time cp /dev/shm/1GB.bin s3fs_mount_-kihaqtowex/c real 0m26.368s user 0m0.006s sys 0m0.699s (I did multiple runs, though not hundreds so I can't say with certainty this would hold) I swear the difference was worse in the past, so maybe things have improved. Still not worth the added slowness to me for my use cases.
- gaul 6y ago
  Thanks for testing! You may improve performance via increasing -o parallel_count (default="5"). Newer versions of s3fs have improved performance but please report any cliffs at https://github.com/s3fs-fuse/s3fs-fuse/issues https://github.com/s3fs-fuse/s3fs-fuse/issues .