4 ms·

DELETE is free LIST is not I guess one could spam DELETE calls while bruteforcing filenames to make it free.

by malka 5y ago

toomuchtodo 5y ago
An S3 inventory report [1] is $0.0025 per million objects listed versus $0.005 per LIST request. Make an inventory report request, retrieve the inventory report from S3, make your free delete calls synchronously with API calls or pay for a batch job [2] if you'd rather not write code ($0.25 per batch job and $1.00 per million object operations performed). It's listed in the post as "extra credit", but its trivial having done it myself for a client. A bit disingenuous to state, "if you like to do things the hard way." It literally takes less than hour to do, and you can preserve the inventory report for housekeeping if needed. [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/storag... [2] https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-create-job.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-... (OP: feel free to steal this comment's info if you want to update your post)
- boplicity 5y ago
  > It literally takes less than hour to do Yikes. An hour just to delete some files in a folder? It seems to me that S3 should be avoided unless you really, really, really need it.
  - emaginniss 5y ago
    When the scale of data goes up, so does the complexity of managing it. Deleting 100 rows from a live psql table with 1000 entries is very different from deleting 100M rows from a live table with 1B entries.
- vdm 5y ago
  S3 Batch Operations list of Supported Operations[0] does not include DeleteObject or DeleteObjects. 0. https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-operations.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-...
  - toomuchtodo 5y ago
    I appreciate you pointing out my mistake. That wasn't clear from the documentation I referred to [1] (when I performed the operation, I created a queue from the inventory report and performed delete requests popping the items off the queue), and I've let our TAM know the documentation I referred to needs to be improved to reflect that DeleteObject or DeleteObjects operations with a batch job aren't currently supported. [1] "Specify the operation that you want S3 Batch Operations to run against the objects in the manifest. Each operation type accepts parameters that are specific to that operation. This enables you to perform the same tasks as if you performed the operation one-by-one on each object." https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-create-job.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-...

DELETE is free LIST is not I guess one could spam DELETE calls while bruteforcing filenames to make it free.

by malka 5y ago

DELETE is free LIST is not I guess one could spam DELETE calls while bruteforcing filenames to make it free.

An S3 inventory report [1] is $0.0025 per million objects listed versus $0.005 per LIST request. Make an inventory report request, retrieve the inventory report from S3, make your free delete calls synchronously with API calls or pay for a batch job [2] if you'd rather not write code ($0.25 per batch job and $1.00 per million object operations performed). It's listed in the post as "extra credit", but its trivial having done it myself for a client. A bit disingenuous to state, "if you like to do things the hard way." It literally takes less than hour to do, and you can preserve the inventory report for housekeeping if needed. [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/storag... [2] https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-create-job.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-... (OP: feel free to steal this comment's info if you want to update your post)

boplicity 5y ago
> It literally takes less than hour to do Yikes. An hour just to delete some files in a folder? It seems to me that S3 should be avoided unless you really, really, really need it.
- emaginniss 5y ago
  When the scale of data goes up, so does the complexity of managing it. Deleting 100 rows from a live psql table with 1000 entries is very different from deleting 100M rows from a live table with 1B entries.
vdm 5y ago
S3 Batch Operations list of Supported Operations[0] does not include DeleteObject or DeleteObjects. 0. https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-operations.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-...
- toomuchtodo 5y ago
  I appreciate you pointing out my mistake. That wasn't clear from the documentation I referred to [1] (when I performed the operation, I created a queue from the inventory report and performed delete requests popping the items off the queue), and I've let our TAM know the documentation I referred to needs to be improved to reflect that DeleteObject or DeleteObjects operations with a batch job aren't currently supported. [1] "Specify the operation that you want S3 Batch Operations to run against the objects in the manifest. Each operation type accepts parameters that are specific to that operation. This enables you to perform the same tasks as if you performed the operation one-by-one on each object." https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-create-job.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-...