3 ms·
No. The standard practice is to use a DynamoDB table as the index for your objects in S3. This article misunderstood S3 and could as well have the title: "An A
by belter 3y ago
No. The standard practice is to use a DynamoDB table as the index for your objects in S3.
This article misunderstood S3 and could as well have the title: "An Airplane is not a Car" :-)
- macintux 3y agoI don't know that you can characterize that as a "standard practice". Maybe it's widespread, but I've not encountered it.
- belter 3y ago"Building and Maintaining an Amazon S3 Metadata Index without Servers" - https://aws.amazon.com/pt/blogs/big-data/building-and-maintaining-an-amazon-s3-metadata-index-without-servers/ https://aws.amazon.com/pt/blogs/big-data/building-and-mainta... Here is the architecture of Amazon Drive and the storage of metadata. "AWS re:Invent 2014 | (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users" - https://youtu.be/R2pKtmhyNoA https://youtu.be/R2pKtmhyNoA And you can see the use here at correct time: https://youtu.be/R2pKtmhyNoA?t=546 https://youtu.be/R2pKtmhyNoA?t=546
- ianburrell 3y agoThat article is old. DynamoDB was used because of the old, weak consistency model of S3. Writes were atomic, but lists could return old results so needed consistent list of objects. But in 2020, S3 changed to strong consistency model. There is no need to use DynamoDB now.
- belter 3y agoThe problem was not the eventual consistency model, was the speed of the object list. "...Finding objects based on other attributes, however, requires doing a linear search using the LIST operation. Because each listing can return at most 1000 keys, it may require many requests before finding the object. Because of these additional requests, implementing attribute-based queries in S3 alone can be challenging..."
- orf 3y agoIt was both actually, but more for the listing issue. Netflix built a lot of tooling around this. But yeah: things like filtering on tags or created at dates requires another approach.
- mr_toad 3y agoHive stores metadata in a relational database. So does Snowflake.
- fijiaarone 3y agoSo in reality S3 takes about 2 seconds to retrieve a single file, under ideal conditions. 1 second round trip for the request to DynamoDB to get the object key of the file and 1 second round trip to S3 to get the file contents (assuming no CPU cost on the search because you’re getting the key by ID from the DynamoDB in a flat single table store. And that the file has no network IO because it is a trivial number of bytes, so the HTTP header overwhelmed the content.) I know what you’re thinking — 2 seconds, that’s faster than I can type the 300 character file key with its pseudo prefixes)! Ah, but what if you wanted to get 2 files from S3?
- orf 3y ago2 seconds is a nuts response time, but I guess it depends entirely on your file size. TTFB is usually 50ms.
- deathanatos 3y ago… S3's response time is nowhere near 2 seconds. (Or even 1 second.) Like a sibling poster says, 50ms is a much more realistic ballpark for TTFB.