3 ms·
This used to be the case, but it's not true any more. For example, previously Amazon S3 performance guidelines recommended randomizing prefix naming with hashe
by sterwill 5y ago
This used to be the case, but it's not true any more.
For example, previously Amazon S3 performance guidelines recommended randomizing prefix naming with hashed characters to optimize performance for frequent data retrievals. You no longer have to randomize prefix naming for performance, and can use sequential date-based naming for your prefixes.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi...
- henrydark 5y agoThe doc is talking about how the full prefix is now part of the sharding, in contrast to previous times when only the few characters mattered. But if you put many files with the exact same prefix - even sequential dates - then you hit the threshold the doc says, about 5000 ops/sec.
- sterwill 5y agoI re-read your comment and you do say "same" prefixes (I think I read it as "shared"), which AWS hasn't changed the behavior of (AFAIK). You're right that objects with the exact same prefix are still routed to the same shard (set?) and have that throughput limit. P.S.: In a few projects I've built prefixes using timestamps, but not at the very beginning, and worried that they weren't getting sharded out. The change I linked to fixes that problem.