51 ms·
Ok pretty obvious, but if you don't know what you are storing inside your bucket, how are you accessing your objects in the first place ? If your use-case is s
by Eikon 5y ago
Ok pretty obvious, but if you don't know what you are storing inside your bucket, how are you accessing your objects in the first place ?
If your use-case is storing random things you don't know the path of, maybe it's the wrong product to use.
- shric 5y agoA common pattern is to dump log files or to use as a dead letter queue for failed event processing. These things typically are arbitrarily named by e.g. a prefix and a unix timestamp
- afandian 5y agoOne of the great features of S3 is that it has an arbitrary prefix index on keys! And the list API paginates on batches of 1000 which is useful. You can retrieve all objects with a given prefix, which is great for storing content-addressed files, and being able to iterate on them. You can also partition on arbitrary prefixes too.
- CobrastanJorji 5y agoHere's an example. You're doing a big machine learning workflow and you've got a gazillion large files representing training data. You run a large training job in Amazon's cloud somewhere that is composed of one worker that lists the files and delegates work to various learning jobs, and then those jobs each stream in one training file, process it, and then grab the next item and repeat. That's a pretty common type of work.