4 ms·

The more I learn about S3 the more I feel the same awe I feel when looking at the Pyramids or Ankor Wat. > You can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET

The more I learn about S3 the more I feel the same awe I feel when looking at the Pyramids or Ankor Wat. > You can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket.

dheera 6y ago
Side question -- why is s3fs so slow? I get only about 1 put per 10 seconds with s3fs. One would hope that you could use something like s3fs as a makeshift S3 API, but its performance is really horrible.
- rovr138 6y ago
  I can't answer this directly, but I thought I'd chime in with a possible alternative, or at least something else to test. I'm using rclone with S3[1] (and others). There are commands[2] where you can use to just sync/copy/ files but it also has a mount option[3]. It also has caching[4] and other things that might help. [1] https://rclone.org/s3/ https://rclone.org/s3/ [2] https://rclone.org/commands/ https://rclone.org/commands/ [3] https://rclone.org/commands/rclone_mount/ https://rclone.org/commands/rclone_mount/ [4] https://rclone.org/cache/ https://rclone.org/cache/
- banana_giraffe 6y ago
  One thing I noticed when I went digging through the s3fs source trying to answer that myself is that it's uploader is single threaded. Part of what makes AWS's cli reasonably fast is that it uses several threads to make sure to use as much bandwidth as possible. If I limit AWS's tool to one thread on a box with plenty of bandwidth, it turns my upload down from around 135 MiB/s to around 20 MiB/s Not sure if that's all of it, or even the majority of the slowness.
  - gaul 6y ago
    Could you expand on this comment? s3fs uses multiple threads for uploading and populating readdir metadata via S3fsMultiCurl::MultiPerform. Earlier versions used curl_multi_perform which may have hidden some of the parallelism from you.
    - banana_giraffe 6y ago
      Huh, right you are. I did a cursory search, but whatever I saw clearly was wrong. That said, something is going on that I can't explain: # Default settings for aws $ time aws s3 cp /dev/shm/1GB.bin s3://test-kihaqtowex/a upload: ../../dev/shm/1GB.bin to s3://test-kihaqtowex/a real 0m10.312s user 0m5.909s sys 0m4.204s $ aws configure set default.s3.max_concurrent_requests 4 $ time aws s3 cp /dev/shm/1GB.bin s3://test-kihaqtowex/b upload: ../../dev/shm/1GB.bin to s3://test-kihaqtowex/b real 0m26.732s user 0m4.989s sys 0m2.741s $ time cp /dev/shm/1GB.bin s3fs_mount_-kihaqtowex/c real 0m26.368s user 0m0.006s sys 0m0.699s (I did multiple runs, though not hundreds so I can't say with certainty this would hold) I swear the difference was worse in the past, so maybe things have improved. Still not worth the added slowness to me for my use cases.
      - gaul 6y ago
        Thanks for testing! You may improve performance via increasing -o parallel_count (default="5"). Newer versions of s3fs have improved performance but please report any cliffs at https://github.com/s3fs-fuse/s3fs-fuse/issues https://github.com/s3fs-fuse/s3fs-fuse/issues .
- gaul 6y ago
  This is not expected behavior. An older version had pessimized locking which preventing concurrently adding a file to a directory and listing the directory simultaneously which is the most similar symptom. I recommend upgrading to the latest version, running with debug flags `-f -d -o curldbg`, and observing what's happening. Please open an issue at https://github.com/s3fs-fuse/s3fs-fuse/issues https://github.com/s3fs-fuse/s3fs-fuse/issues if your symptoms persist.
- deleted 6y ago
  [deleted]
web007 6y ago
One fun fact that I learned recently: a prefix is not strictly path-delimited. I would think of /foo/bar and /foo/baz and /bar/baz as having two prefixes, but it could be anywhere from one to three, depending on how S3 has partitioned your data.
- randallsquared 6y ago
  It shouldn't be one if you're sending 3500+ requests a second. It may take some time for partitioning to happen, though.
  - CrossWired 6y ago
    This is a major issue when turning batch data into individual files, especially with dynamic prefixes. You can and will find this edge pretty quick using EMR or Glue which tries to run as fast as possible. The only answer is....slow down your writes or just try again, a bit frustrating.
- rossmohax 6y ago
  I asked around and it seems, that `prefix` here is exact value for `prefix` or `key` query in the API. So you can make 5500 ListObject rps for prefix=/a/b and simultaneously make 5500 GetObject rps for key=/a/b/file1 and not be ratelimited. In other word ratelimiting key is `${HTTP_METHOD}(${QURERY_PARAM_KEY}|${QUERY_PARAM_PREFIX})`
swyx 6y ago
11 nines of reliability. I don't know anything else promising that (unclear if they actually reach it)
- darkr 6y ago
  11 nines of _durability_. Uptime SLA for s3 is 99.9% AFAICR
  - mrep 6y ago
    And those 11 nines of durability are what it is designed for, they don't offer any sort of SLA for that.
  - swyx 6y ago
    thank you for the correction, I'm still pretty new to the backend world so I play too fast n loose with this terminology.
- ignoramous 6y ago
  Route53 promises a 100% availability. https://www.slideshare.net/AmazonWebServices/under-the-hood-of-amazon-route-53-arc408r1-aws-reinvent-2018 https://www.slideshare.net/AmazonWebServices/under-the-hood-... https://aws.amazon.com/route53/sla/ https://aws.amazon.com/route53/sla/

dheera 6y ago

Side question -- why is s3fs so slow? I get only about 1 put per 10 seconds with s3fs. One would hope that you could use something like s3fs as a makeshift S3 API, but its performance is really horrible.

rovr138 6y ago
I can't answer this directly, but I thought I'd chime in with a possible alternative, or at least something else to test. I'm using rclone with S3[1] (and others). There are commands[2] where you can use to just sync/copy/ files but it also has a mount option[3]. It also has caching[4] and other things that might help. [1] https://rclone.org/s3/ https://rclone.org/s3/ [2] https://rclone.org/commands/ https://rclone.org/commands/ [3] https://rclone.org/commands/rclone_mount/ https://rclone.org/commands/rclone_mount/ [4] https://rclone.org/cache/ https://rclone.org/cache/
banana_giraffe 6y ago
One thing I noticed when I went digging through the s3fs source trying to answer that myself is that it's uploader is single threaded. Part of what makes AWS's cli reasonably fast is that it uses several threads to make sure to use as much bandwidth as possible. If I limit AWS's tool to one thread on a box with plenty of bandwidth, it turns my upload down from around 135 MiB/s to around 20 MiB/s Not sure if that's all of it, or even the majority of the slowness.
- gaul 6y ago
  Could you expand on this comment? s3fs uses multiple threads for uploading and populating readdir metadata via S3fsMultiCurl::MultiPerform. Earlier versions used curl_multi_perform which may have hidden some of the parallelism from you.
  - banana_giraffe 6y ago
    Huh, right you are. I did a cursory search, but whatever I saw clearly was wrong. That said, something is going on that I can't explain: # Default settings for aws $ time aws s3 cp /dev/shm/1GB.bin s3://test-kihaqtowex/a upload: ../../dev/shm/1GB.bin to s3://test-kihaqtowex/a real 0m10.312s user 0m5.909s sys 0m4.204s $ aws configure set default.s3.max_concurrent_requests 4 $ time aws s3 cp /dev/shm/1GB.bin s3://test-kihaqtowex/b upload: ../../dev/shm/1GB.bin to s3://test-kihaqtowex/b real 0m26.732s user 0m4.989s sys 0m2.741s $ time cp /dev/shm/1GB.bin s3fs_mount_-kihaqtowex/c real 0m26.368s user 0m0.006s sys 0m0.699s (I did multiple runs, though not hundreds so I can't say with certainty this would hold) I swear the difference was worse in the past, so maybe things have improved. Still not worth the added slowness to me for my use cases.
    - gaul 6y ago
      Thanks for testing! You may improve performance via increasing -o parallel_count (default="5"). Newer versions of s3fs have improved performance but please report any cliffs at https://github.com/s3fs-fuse/s3fs-fuse/issues https://github.com/s3fs-fuse/s3fs-fuse/issues .
gaul 6y ago
This is not expected behavior. An older version had pessimized locking which preventing concurrently adding a file to a directory and listing the directory simultaneously which is the most similar symptom. I recommend upgrading to the latest version, running with debug flags `-f -d -o curldbg`, and observing what's happening. Please open an issue at https://github.com/s3fs-fuse/s3fs-fuse/issues https://github.com/s3fs-fuse/s3fs-fuse/issues if your symptoms persist.
deleted 6y ago
[deleted]

web007 6y ago

One fun fact that I learned recently: a prefix is not strictly path-delimited. I would think of /foo/bar and /foo/baz and /bar/baz as having two prefixes, but it could be anywhere from one to three, depending on how S3 has partitioned your data.

randallsquared 6y ago
It shouldn't be one if you're sending 3500+ requests a second. It may take some time for partitioning to happen, though.
- CrossWired 6y ago
  This is a major issue when turning batch data into individual files, especially with dynamic prefixes. You can and will find this edge pretty quick using EMR or Glue which tries to run as fast as possible. The only answer is....slow down your writes or just try again, a bit frustrating.
rossmohax 6y ago
I asked around and it seems, that `prefix` here is exact value for `prefix` or `key` query in the API. So you can make 5500 ListObject rps for prefix=/a/b and simultaneously make 5500 GetObject rps for key=/a/b/file1 and not be ratelimited. In other word ratelimiting key is `${HTTP_METHOD}(${QURERY_PARAM_KEY}|${QUERY_PARAM_PREFIX})`

swyx 6y ago

11 nines of reliability. I don't know anything else promising that (unclear if they actually reach it)

darkr 6y ago
11 nines of _durability_. Uptime SLA for s3 is 99.9% AFAICR
- mrep 6y ago
  And those 11 nines of durability are what it is designed for, they don't offer any sort of SLA for that.
- swyx 6y ago
  thank you for the correction, I'm still pretty new to the backend world so I play too fast n loose with this terminology.
ignoramous 6y ago
Route53 promises a 100% availability. https://www.slideshare.net/AmazonWebServices/under-the-hood-of-amazon-route-53-arc408r1-aws-reinvent-2018 https://www.slideshare.net/AmazonWebServices/under-the-hood-... https://aws.amazon.com/route53/sla/ https://aws.amazon.com/route53/sla/