4 ms·
Yeah S3 has similar performance issues where accessing objects with the same prefixes has lower throughput because they get sharded onto the same server. It's v
by ummonk 5y ago
Yeah S3 has similar performance issues where accessing objects with the same prefixes has lower throughput because they get sharded onto the same server. It's very counterintuitive when you're used to how performance works on single computers where you want to optimize for cache-locality.
- nielsole 5y agoUgh really? Why would they not hash the whole filename for shard assignment?
- bowmessage 5y agoBecause the file name includes the "directory" prefix, i.e. each file's name stores the `/entire/bucket/dir/tree`, which can get large.
- manigandham 5y agoThe size of the input doesn't affect the hashing.
- bowmessage 5y agoIt affects the runtime of the hashing.
- mmoskal 5y agoIn these systems you typically can list blobs based on prefix. If you shard blobs sharing prefix to different nodes that becomes difficult.
- sterwill 5y agoThis used to be the case, but it's not true any more. For example, previously Amazon S3 performance guidelines recommended randomizing prefix naming with hashed characters to optimize performance for frequent data retrievals. You no longer have to randomize prefix naming for performance, and can use sequential date-based naming for your prefixes. https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi...
- henrydark 5y agoThe doc is talking about how the full prefix is now part of the sharding, in contrast to previous times when only the few characters mattered. But if you put many files with the exact same prefix - even sequential dates - then you hit the threshold the doc says, about 5000 ops/sec.
- sterwill 5y agoI re-read your comment and you do say "same" prefixes (I think I read it as "shared"), which AWS hasn't changed the behavior of (AFAIK). You're right that objects with the exact same prefix are still routed to the same shard (set?) and have that throughput limit. P.S.: In a few projects I've built prefixes using timestamps, but not at the very beginning, and worried that they weren't getting sharded out. The change I linked to fixes that problem.