3 ms·
One fun fact that I learned recently: a prefix is not strictly path-delimited. I would think of /foo/bar and /foo/baz and /bar/baz as having two prefixes, but i
by web007 6y ago
One fun fact that I learned recently: a prefix is not strictly path-delimited. I would think of /foo/bar and /foo/baz and /bar/baz as having two prefixes, but it could be anywhere from one to three, depending on how S3 has partitioned your data.
- randallsquared 6y agoIt shouldn't be one if you're sending 3500+ requests a second. It may take some time for partitioning to happen, though.
- CrossWired 6y agoThis is a major issue when turning batch data into individual files, especially with dynamic prefixes. You can and will find this edge pretty quick using EMR or Glue which tries to run as fast as possible. The only answer is....slow down your writes or just try again, a bit frustrating.
- rossmohax 6y agoI asked around and it seems, that `prefix` here is exact value for `prefix` or `key` query in the API. So you can make 5500 ListObject rps for prefix=/a/b and simultaneously make 5500 GetObject rps for key=/a/b/file1 and not be ratelimited. In other word ratelimiting key is `${HTTP_METHOD}(${QURERY_PARAM_KEY}|${QUERY_PARAM_PREFIX})`