4 ms·

This is amazing. I’m thinking of building the same thing for self hosted S3 (ie. Minio). I love AWS, they're very reliable and i feel like depending on them is

This is amazing. I’m thinking of building the same thing for self hosted S3 (ie. Minio). I love AWS, they're very reliable and i feel like depending on them is not risky. But being 100% dependent on a single provider scares me a lot, and I often feel like S3 is the one thing that would cost me a lot if I tried to migrate. I have not accounted for all the data I have there, but it's lots of small files (all the user uploads), and what scares me is the per request cost of deletion (I think I read some horror story here sometime ago). So I keep pushing this off.

supriyo-biswas 3y ago
Deletions do not encounter costs, but obtaining a list of objects in order to delete them certainly does.
albert_e 3y ago
Bulk deletes on S3 should be done using a Life Cycle Policy to avoid the per object delete cost
- toomuchtodo 3y ago
  Deletes are free, but you have to pay to list to delete if not individually tracking.
  - ludjer 3y ago
    Turn on weekly manifest to get a list of files and then you can delete based off that. Much faster and better then listing billions of files.
    - HatchedLake721 3y ago
      What’s a weekly manifest?
      - uw_rob 3y ago
        I believe ludjer is referring to S3 Storage Inventory. This is a daily, or weekly, file produced containing metadata on every file within a S3 Bucket. It does not use the synchronous List APIs. > You can use Amazon S3 Inventory to help manage your storage. For example, you can use it to audit and report on the replication and encryption status of your objects for business, compliance, and regulatory needs. You can also simplify and speed up business workflows and big data jobs by using Amazon S3 Inventory, which provides a scheduled alternative to the Amazon S3 synchronous List API operations. Amazon S3 Inventory does not use the List API operations to audit your objects and does not affect the request rate of your bucket. > > Amazon S3 Inventory provides comma-separated values (CSV), Apache optimized row columnar (ORC) or Apache Parquet output files that list your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or objects with a shared prefix (that is, objects that have names that begin with a common string). If you set up a weekly inventory, a report is generated every Sunday (UTC time zone) after the initial report. For information about Amazon S3 Inventory pricing, see Amazon S3 pricing. https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/storag...
  - orf 3y ago
    Ignore the replies about inventories. $0.005 Per 1k list requests. A list request returns 1k items. 1 million items for $0.005, or 1 billion items for 5 dollars. Still use lifecycle policies (I visualized how it ages out 1 billion items here[1]), but list request prices are not a factor worth mentioning. 1. https://tomforb.es/visualizing-how-s3-deletes-1-billion-objects-with-athena-and-rust/ https://tomforb.es/visualizing-how-s3-deletes-1-billion-obje...
- ericpauley 3y ago
  This (although as others said it’s the list cost, not the delete). If you want to nuke a bucket don’t waste your time with the API and just age the files out. I’ve done this with buckets with tens of millions of files and five minute’s work.
  - albert_e 3y ago
    Both list objects and delete object are API actions that incur the standard cost for "requests" ... isn't it? >> PUT, COPY, POST, LIST requests (per 1,000 requests) = $0.005 GET, SELECT, and all other requests (per 1,000 requests) = $0.0004 >> "You pay for requests made against your S3 buckets and objects. S3 request costs are based on the request type, and are charged on the quantity of requests as listed in the table below" https://aws.amazon.com/s3/pricing/#:~:text=You%20pay%20for%20requests%20made%20against%20your%20S3%20buckets%20and%20objects.%20S3%20request%20costs%20are%20based%20on%20the%20request%20type%2C%20and%20are%20charged%20on%20the%20quantity%20of%20requests%20as%20listed%20in%20the%20table%20below https://aws.amazon.com/s3/pricing/#:~:text=You%20pay%20for%2...
    - ericpauley 3y ago
      From the same paragraph: “DELETE and CANCEL requests are free.”

supriyo-biswas 3y ago

Deletions do not encounter costs, but obtaining a list of objects in order to delete them certainly does.

albert_e 3y ago

Bulk deletes on S3 should be done using a Life Cycle Policy to avoid the per object delete cost

toomuchtodo 3y ago
Deletes are free, but you have to pay to list to delete if not individually tracking.
- ludjer 3y ago
  Turn on weekly manifest to get a list of files and then you can delete based off that. Much faster and better then listing billions of files.
  - HatchedLake721 3y ago
    What’s a weekly manifest?
    - uw_rob 3y ago
      I believe ludjer is referring to S3 Storage Inventory. This is a daily, or weekly, file produced containing metadata on every file within a S3 Bucket. It does not use the synchronous List APIs. > You can use Amazon S3 Inventory to help manage your storage. For example, you can use it to audit and report on the replication and encryption status of your objects for business, compliance, and regulatory needs. You can also simplify and speed up business workflows and big data jobs by using Amazon S3 Inventory, which provides a scheduled alternative to the Amazon S3 synchronous List API operations. Amazon S3 Inventory does not use the List API operations to audit your objects and does not affect the request rate of your bucket. > > Amazon S3 Inventory provides comma-separated values (CSV), Apache optimized row columnar (ORC) or Apache Parquet output files that list your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or objects with a shared prefix (that is, objects that have names that begin with a common string). If you set up a weekly inventory, a report is generated every Sunday (UTC time zone) after the initial report. For information about Amazon S3 Inventory pricing, see Amazon S3 pricing. https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/storag...
- orf 3y ago
  Ignore the replies about inventories. $0.005 Per 1k list requests. A list request returns 1k items. 1 million items for $0.005, or 1 billion items for 5 dollars. Still use lifecycle policies (I visualized how it ages out 1 billion items here[1]), but list request prices are not a factor worth mentioning. 1. https://tomforb.es/visualizing-how-s3-deletes-1-billion-objects-with-athena-and-rust/ https://tomforb.es/visualizing-how-s3-deletes-1-billion-obje...
ericpauley 3y ago
This (although as others said it’s the list cost, not the delete). If you want to nuke a bucket don’t waste your time with the API and just age the files out. I’ve done this with buckets with tens of millions of files and five minute’s work.
- albert_e 3y ago
  Both list objects and delete object are API actions that incur the standard cost for "requests" ... isn't it? >> PUT, COPY, POST, LIST requests (per 1,000 requests) = $0.005 GET, SELECT, and all other requests (per 1,000 requests) = $0.0004 >> "You pay for requests made against your S3 buckets and objects. S3 request costs are based on the request type, and are charged on the quantity of requests as listed in the table below" https://aws.amazon.com/s3/pricing/#:~:text=You%20pay%20for%20requests%20made%20against%20your%20S3%20buckets%20and%20objects.%20S3%20request%20costs%20are%20based%20on%20the%20request%20type%2C%20and%20are%20charged%20on%20the%20quantity%20of%20requests%20as%20listed%20in%20the%20table%20below https://aws.amazon.com/s3/pricing/#:~:text=You%20pay%20for%2...
  - ericpauley 3y ago
    From the same paragraph: “DELETE and CANCEL requests are free.”