4 ms·

You probably got told off because S3 isn't a file store. Asking for such a feature displays great ignorance about what the S3 service actually is.

by shats 7y ago

scarface74 7y ago
How would that even work on the API level efficiency given how S3 is architected? It’s clear given their optimization techniques they tell you to use that S3 is optimized for prefix style searching. That’s all they offer from the web console, the cli or the SDKs. It would amount to retrieving everything from the S3 conceptual “index database” and doing a non optimized sort when everything is architected to search based on the key and a key prefix. It’s far more efficient to let the client get a subset by prefix (ie “folder”) and sort/filter by other meta data client side.
- tomc1985 7y ago
  The web console doesn't even do global prefix searching Why bother giving me a file browser if I'm supposedly not supposed to be browsing files?
  - scarface74 7y ago
    What do you mean by “global prefix searching” you mean across multiple buckets? Why bother giving me a file browser if I'm supposedly not supposed to be browsing files? Welcome to the world of leaky abstractions.
    - tomc1985 7y ago
      As in, search results reflect only the current visible page and not all pages. If you use a bucket as a dump for, say, daily backups, finding a specific file from 3 years ago is really time-consuming and needlessly difficult. Can't sort, can't search.... just pure frustration I haven't checked this feature in years -- maybe they fixed it
      - scarface74 7y ago
        It comes from a fundamental lack of understanding of what S3 is and how it is organized. Objects are indexed as a key value store optimized for sorting on a key and a key prefix. The fact that keys can have backslashes as a character doesn’t mean that objects are stored in “folders”. Knowing how S3 indexes data - by key only and all other metadata is tagged on the object and not in the index, should inform how to name your keys and your access patterns. For instance sorting by date, would involve AWS querying each objects meta data and then sorting it. If having to find objects by date is an important use case, why not create keys in the format of YYYYMMDD? Or just use the CLI. https://github.com/aws/aws-cli/issues/2095 https://github.com/aws/aws-cli/issues/2095

You probably got told off because S3 isn't a file store. Asking for such a feature displays great ignorance about what the S3 service actually is.

by shats 7y ago

You probably got told off because S3 isn't a file store. Asking for such a feature displays great ignorance about what the S3 service actually is.

scarface74 7y ago

How would that even work on the API level efficiency given how S3 is architected? It’s clear given their optimization techniques they tell you to use that S3 is optimized for prefix style searching. That’s all they offer from the web console, the cli or the SDKs. It would amount to retrieving everything from the S3 conceptual “index database” and doing a non optimized sort when everything is architected to search based on the key and a key prefix. It’s far more efficient to let the client get a subset by prefix (ie “folder”) and sort/filter by other meta data client side.

tomc1985 7y ago
The web console doesn't even do global prefix searching Why bother giving me a file browser if I'm supposedly not supposed to be browsing files?
- scarface74 7y ago
  What do you mean by “global prefix searching” you mean across multiple buckets? Why bother giving me a file browser if I'm supposedly not supposed to be browsing files? Welcome to the world of leaky abstractions.
  - tomc1985 7y ago
    As in, search results reflect only the current visible page and not all pages. If you use a bucket as a dump for, say, daily backups, finding a specific file from 3 years ago is really time-consuming and needlessly difficult. Can't sort, can't search.... just pure frustration I haven't checked this feature in years -- maybe they fixed it
    - scarface74 7y ago
      It comes from a fundamental lack of understanding of what S3 is and how it is organized. Objects are indexed as a key value store optimized for sorting on a key and a key prefix. The fact that keys can have backslashes as a character doesn’t mean that objects are stored in “folders”. Knowing how S3 indexes data - by key only and all other metadata is tagged on the object and not in the index, should inform how to name your keys and your access patterns. For instance sorting by date, would involve AWS querying each objects meta data and then sorting it. If having to find objects by date is an important use case, why not create keys in the format of YYYYMMDD? Or just use the CLI. https://github.com/aws/aws-cli/issues/2095 https://github.com/aws/aws-cli/issues/2095