7 ms·

Is listing really such a key feature that people use it as a database to find objects? Have not used S3, but that is not how I imagined using it.

by tjoff 3y ago

orf 3y ago
Sure. It's kind of an index - limited to prefix-only searching, but useful. Say you store uploads associated with a company and a user. You'd maybe naively store them as `[company-uuid]/[user-id].[timestamp]`. If you need to list a given users (123) uploads after a given date, you'd list keys after `[company-uuid]/123.[date]`. If you need to list all users uploads, you'd list `[company-uuid]/123.`. If you need to get a set of all users who have photos, you'd list `[company-uuid]/` with a Delimiter set to `.` The point is that it's flexible and with a bit of thought it allows you to "remove all a users uploads between two dates", "remove all a companies uploads" or "remove all a users uploads" with a single call. Or whatever specific stuff is important to your use-case, that might otherwise need a separate DB. It's not perfect - you can't reverse the listing (i.e you can't get the latest photo for a given user by sorting descending for example), and needs some thought about your key structure.
- tjoff 3y ago
  But surely you need to track that elsewhere anyway? That some niche edge-case runs efficiently doesn't sound like a defining feature of S3. On the contrary many common operations map terrible to S3, so you kind of need the logic to be elsewhere.
  - orf 3y ago
    My overall point can be summarised as this: - Listing things is a very common operation to do. - The POSIX api and the directory/file hierarchy it provides is a restrictive one. - S3 does not suffer from this, you can recursively list and group keys into directories at “list time”. - If you find yourself needing to list gigantic numbers of keys in one go, you can do better by only listing a subset. S3 isn’t a filesystem, you shouldn’t need to list 1k+ keys sequentially apart from during maintenance tasks. - This is actually quite fast, compared to alternatives. Whether or not you see a use case for this is sort of irrelevant: they exist. it’s what allows you to easily put data into s3 and flexibly group/scan it by specific attributes.
    - tjoff 3y ago
      Listing things is very common, so why would you outsource that to S3 when all your bookkeeping is elsewhere? It's not like you would ever rely on the POSIX API for that anyway, even for when your files actually are on a POSIX filesystem. For sure, for maintenance tasks etc. it sounds quite useful. And good hygiene with prefixes sounds like a sane idea. But listing being a critical part of what "makes S3 useful"? That seems like an huge stretch that your points don't seem to address.
      - orf 3y ago
        > It's not like you would ever rely on the POSIX API for that anyway, even for when your files actually are on a POSIX filesystem. Because there is no POSIX api for this. Depending on your requirements and query patterns, you may not need a completely separate database that you need to keep in sync.
      - IanCal 3y ago
        You may not need other bookkeeping. The prefix listing properties can be enough, removing the need to have two distinct systems kept in sync.
  - kbolino 3y ago
    > But surely you need to track that elsewhere anyway? Why? If the S3 structure and listing is sufficient, I don't need to store anything else anywhere else. Many use cases may involve other requirements that S3 can't meet, such as being able to find the same object via different keys, or being able to search through the metadata fields. However, if the requirements match up with S3's structure, then additional services are unnecessary and keeping them in sync with S3 is more hassle than it's worth.
    - tjoff 3y ago
      I agree, but something as simple (in functionality) as that ought to be an edge-case. Not a defining feature of S3.
      - dekhn 3y ago
        it's a property of the system that I, as an architect, would seriously consider as part of my system's design. I've worked with many systems where iterating over items in order starting from a prefix is extremely cheap (sstables).
      - orf 3y ago
        It’s fundamental to how S3 works and its ability to scale, so it is a defining feature of S3. If you think wider, a bucket itself is just a prefix.
        tjoff 3y ago
        From amazons perspective, sure! But that's not what we are discussing.
belter 3y ago
No. The standard practice is to use a DynamoDB table as the index for your objects in S3. This article misunderstood S3 and could as well have the title: "An Airplane is not a Car" :-)
- macintux 3y ago
  I don't know that you can characterize that as a "standard practice". Maybe it's widespread, but I've not encountered it.
  - belter 3y ago
    "Building and Maintaining an Amazon S3 Metadata Index without Servers" - https://aws.amazon.com/pt/blogs/big-data/building-and-maintaining-an-amazon-s3-metadata-index-without-servers/ https://aws.amazon.com/pt/blogs/big-data/building-and-mainta... Here is the architecture of Amazon Drive and the storage of metadata. "AWS re:Invent 2014 | (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users" - https://youtu.be/R2pKtmhyNoA https://youtu.be/R2pKtmhyNoA And you can see the use here at correct time: https://youtu.be/R2pKtmhyNoA?t=546 https://youtu.be/R2pKtmhyNoA?t=546
    - ianburrell 3y ago
      That article is old. DynamoDB was used because of the old, weak consistency model of S3. Writes were atomic, but lists could return old results so needed consistent list of objects. But in 2020, S3 changed to strong consistency model. There is no need to use DynamoDB now.
      - belter 3y ago
        The problem was not the eventual consistency model, was the speed of the object list. "...Finding objects based on other attributes, however, requires doing a linear search using the LIST operation. Because each listing can return at most 1000 keys, it may require many requests before finding the object. Because of these additional requests, implementing attribute-based queries in S3 alone can be challenging..."

Is listing really such a key feature that people use it as a database to find objects? Have not used S3, but that is not how I imagined using it.

by tjoff 3y ago

Is listing really such a key feature that people use it as a database to find objects? Have not used S3, but that is not how I imagined using it.

orf 3y ago

Sure. It's kind of an index - limited to prefix-only searching, but useful. Say you store uploads associated with a company and a user. You'd maybe naively store them as `[company-uuid]/[user-id].[timestamp]`. If you need to list a given users (123) uploads after a given date, you'd list keys after `[company-uuid]/123.[date]`. If you need to list all users uploads, you'd list `[company-uuid]/123.`. If you need to get a set of all users who have photos, you'd list `[company-uuid]/` with a Delimiter set to `.` The point is that it's flexible and with a bit of thought it allows you to "remove all a users uploads between two dates", "remove all a companies uploads" or "remove all a users uploads" with a single call. Or whatever specific stuff is important to your use-case, that might otherwise need a separate DB. It's not perfect - you can't reverse the listing (i.e you can't get the latest photo for a given user by sorting descending for example), and needs some thought about your key structure.

tjoff 3y ago
But surely you need to track that elsewhere anyway? That some niche edge-case runs efficiently doesn't sound like a defining feature of S3. On the contrary many common operations map terrible to S3, so you kind of need the logic to be elsewhere.
- orf 3y ago
  My overall point can be summarised as this: - Listing things is a very common operation to do. - The POSIX api and the directory/file hierarchy it provides is a restrictive one. - S3 does not suffer from this, you can recursively list and group keys into directories at “list time”. - If you find yourself needing to list gigantic numbers of keys in one go, you can do better by only listing a subset. S3 isn’t a filesystem, you shouldn’t need to list 1k+ keys sequentially apart from during maintenance tasks. - This is actually quite fast, compared to alternatives. Whether or not you see a use case for this is sort of irrelevant: they exist. it’s what allows you to easily put data into s3 and flexibly group/scan it by specific attributes.
  - tjoff 3y ago
    Listing things is very common, so why would you outsource that to S3 when all your bookkeeping is elsewhere? It's not like you would ever rely on the POSIX API for that anyway, even for when your files actually are on a POSIX filesystem. For sure, for maintenance tasks etc. it sounds quite useful. And good hygiene with prefixes sounds like a sane idea. But listing being a critical part of what "makes S3 useful"? That seems like an huge stretch that your points don't seem to address.
    - orf 3y ago
      > It's not like you would ever rely on the POSIX API for that anyway, even for when your files actually are on a POSIX filesystem. Because there is no POSIX api for this. Depending on your requirements and query patterns, you may not need a completely separate database that you need to keep in sync.
    - IanCal 3y ago
      You may not need other bookkeeping. The prefix listing properties can be enough, removing the need to have two distinct systems kept in sync.
- kbolino 3y ago
  > But surely you need to track that elsewhere anyway? Why? If the S3 structure and listing is sufficient, I don't need to store anything else anywhere else. Many use cases may involve other requirements that S3 can't meet, such as being able to find the same object via different keys, or being able to search through the metadata fields. However, if the requirements match up with S3's structure, then additional services are unnecessary and keeping them in sync with S3 is more hassle than it's worth.
  - tjoff 3y ago
    I agree, but something as simple (in functionality) as that ought to be an edge-case. Not a defining feature of S3.
    - dekhn 3y ago
      it's a property of the system that I, as an architect, would seriously consider as part of my system's design. I've worked with many systems where iterating over items in order starting from a prefix is extremely cheap (sstables).
    - orf 3y ago
      It’s fundamental to how S3 works and its ability to scale, so it is a defining feature of S3. If you think wider, a bucket itself is just a prefix.
      - tjoff 3y ago
        From amazons perspective, sure! But that's not what we are discussing.

belter 3y ago

No. The standard practice is to use a DynamoDB table as the index for your objects in S3. This article misunderstood S3 and could as well have the title: "An Airplane is not a Car" :-)

macintux 3y ago
I don't know that you can characterize that as a "standard practice". Maybe it's widespread, but I've not encountered it.
- belter 3y ago
  "Building and Maintaining an Amazon S3 Metadata Index without Servers" - https://aws.amazon.com/pt/blogs/big-data/building-and-maintaining-an-amazon-s3-metadata-index-without-servers/ https://aws.amazon.com/pt/blogs/big-data/building-and-mainta... Here is the architecture of Amazon Drive and the storage of metadata. "AWS re:Invent 2014 | (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users" - https://youtu.be/R2pKtmhyNoA https://youtu.be/R2pKtmhyNoA And you can see the use here at correct time: https://youtu.be/R2pKtmhyNoA?t=546 https://youtu.be/R2pKtmhyNoA?t=546
  - ianburrell 3y ago
    That article is old. DynamoDB was used because of the old, weak consistency model of S3. Writes were atomic, but lists could return old results so needed consistent list of objects. But in 2020, S3 changed to strong consistency model. There is no need to use DynamoDB now.
    - belter 3y ago
      The problem was not the eventual consistency model, was the speed of the object list. "...Finding objects based on other attributes, however, requires doing a linear search using the LIST operation. Because each listing can return at most 1000 keys, it may require many requests before finding the object. Because of these additional requests, implementing attribute-based queries in S3 alone can be challenging..."