5 ms·
Here's another one for you: distribute your s3 paths/names Because of the way s3 is designed, the place files are stored on the physical infrastructure is depe
by thomseddon 13y ago
Here's another one for you: distribute your s3 paths/names
Because of the way s3 is designed, the place files are stored on the physical infrastructure is dependant on the prefix of the key name. I'm not exactly sure how much of the key name is used, but for example if you prefixed all you images with imges/....jpg it's highly likely they will all be stored on the same physical hardware.
I know of at least two companies for whom this has caused large problems for, one of them is netflix. Imagine all the videos in a single bucket with key names "/video/breaking_bad_s1_e1.mp4" (a crude example I know), all requests hit the same physical hardware and under high load the hardware just can't keep up and this exact issue has apparently been the cause of more than one Netflix outage.
The solution is simple, ensure your files have a random prefix ({uuid}.breaking_bad_s1_e1.mp4) and they will be spread around the datacentre :)
- jeffbarr 13y agoThis is definitely a good tip. Here's a blog post with more info on the topic: http://aws.typepad.com/aws/2012/03/amazon-s3-performance-tips-tricks-seattle-hiring-event.html http://aws.typepad.com/aws/2012/03/amazon-s3-performance-tip...
- moe 13y agoThis seems a rather odd implementation detail to burden the user with. Why don't you just use a hash of the filename for the partition key, so all files are distributed randomly regardless of their name? Is there an advantage for amazon or for the user in having S3 files cluster up by filename by default?
- penguindev 13y agomost likely so dir listings are a lot faster. (they are sorted by key, and can use delimiters for subdir emulation..)
- dllthomas 13y ago... but for that they just need to store the filenames sorted somewhere. Content could be located based on a hash of the filename, still.
- sitkack 13y agoIf this is true, it is umm, naively suboptimal (I have harsher words in my head).
- dbarlett 13y agoRelevant sessions from re:Invent: Building Scalable Applications on Amazon Simple Storage Service (2012) https://www.youtube.com/watch?v=YYnVRYbUR6A https://www.youtube.com/watch?v=YYnVRYbUR6A Maximizing Amazon S3 Performance (2013) https://www.youtube.com/watch?v=uXHw0Xae2ww https://www.youtube.com/watch?v=uXHw0Xae2ww
- deleted 13y ago[deleted]
- jurassic 13y agoRandomness isn't necessary to distribute the path names. Somewhere I read that reversing sequential ids is sufficient. So, for instance, if you have some sequentially numbered files you want to store: 123.jpg 124.jpg 125.jpg 126.jpg You save them at: 321.jpg 421.jpg 521.jpg 621.jpg
- donavanm 13y agoS3 keys use the leading ~16 bytes for partitioning, most significant first, as I recall. Prefixing with guids is ideal, epoch timestamps are pathologically bad, a "random" char should be enough for 99% of users.