5 ms·

Batch delete, batch delete, wherefore art thou batch delete?

giaour 7y ago
https://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html https://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectd...
- chocolatemario 7y ago
  This has a limit of 1000 keys, does not handle redriving failed requests and offers no report of the job. What you could do is use s3's inventory report feature, give the manifest generated to batch operations and handle the delete logic in a lambda. A lifecycle policy with some tagging could also fit your needs here.
meritt 7y ago
It's not exactly what you're asking for, but we have a large bucket with billions of files (don't ever do this, it was a terrible idea) and we manage deletions via lifecycle rules. If your file naming convention and data retention policy permits it, far easier than calling delete with 1,000 keys at a time. Also just a word of warning, if you do have a lot of files, and you're thinking "let's transition them to glacier", don't do it. The transfer cost from S3->Glacier is absolutely insane ($0.05 per 1,000 objects). I managed to generate $11k worth of charges doing a "small" test of 218M files and a lifecycle policy. Only use glacier for large individual files. [1] https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html https://docs.aws.amazon.com/AmazonS3/latest/user-guide/creat...
- toomuchtodo 7y ago
  I have to ask: what’s performance like for operations on the bucket objects? Edit: I ask because AWS suggests a key naming convention for large object amounts to ensure that you're distributing your objects across storage nodes, to prevent bottlenecks. https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate...
  - DVassallo 7y ago
    No difference for Put, Get and Delete. Don't know about List, but if it degrades it's not significant. I worked with buckets with exabytes of data and billions of objects.
  - meritt 7y ago
    Never noticed any speed difference due to bucket size. S3 is generally slow anyway (250ms for a write isn't uncommon) but it scales very well and we use it for raw data storage that's not in our critical path, so the latency isn't a problem. Edit Response: I've always used the partitioning conventions they suggest so not sure what sort of impact you encounter without.
  - votepaunchy 7y ago
    “This S3 request rate performance increase removes any previous guidance to randomize object prefixes to achieve faster performance. That means you can now use logical or sequential naming patterns in S3 object naming without any performance implications.” https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3-announces-increased-request-rate-performance/ https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3...
    - deleted 7y ago
      [deleted]
- duality 7y ago
  But would a batch delete by any other name smell just as sweet?
- cheeze 7y ago
  > don't ever do this, it was a terrible idea I have billions of files. What do?
  - meritt 7y ago
    For us, it was due to the relatively high PUT cost if you're storing a large number of small files. We ended up changing our approach and we now store blocks (~10MB archives) instead of individual files into S3. The S3 portion of our AWS bill was previously 50% PUT / 50% long-term storage charges. After the change, we managed to reduce the PUT aspect to nearly $0 and reduced our overall AWS bill by almost 30%, while still storing the same amount of data per month. e.g. if you write 1 million 10KB files per day to S3, you're looking at $150/mo in PUT costs. If you instead write 1,000 10MB blocks, you're looking at $0.15/mo in PUT costs. Due to S3's support of HTTP range requests, we can still request individual files without an intermediate layer (though our write layer did slightly increase in complexity) and our GET (and storage) costs are identical.
neonate 7y ago
Wherefore means why, kind of like "what for". You want just "where", though that breaks the quote.
- johnhenry 7y ago
  It's plausible that zacharyozer refers to the batch delete that exists in his or her heart. In which case "wherefore" would be appropriate.

https://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html https://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectd...

chocolatemario 7y ago
This has a limit of 1000 keys, does not handle redriving failed requests and offers no report of the job. What you could do is use s3's inventory report feature, give the manifest generated to batch operations and handle the delete logic in a lambda. A lifecycle policy with some tagging could also fit your needs here.

meritt 7y ago

It's not exactly what you're asking for, but we have a large bucket with billions of files (don't ever do this, it was a terrible idea) and we manage deletions via lifecycle rules. If your file naming convention and data retention policy permits it, far easier than calling delete with 1,000 keys at a time. Also just a word of warning, if you do have a lot of files, and you're thinking "let's transition them to glacier", don't do it. The transfer cost from S3->Glacier is absolutely insane ($0.05 per 1,000 objects). I managed to generate $11k worth of charges doing a "small" test of 218M files and a lifecycle policy. Only use glacier for large individual files. [1] https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html https://docs.aws.amazon.com/AmazonS3/latest/user-guide/creat...

toomuchtodo 7y ago
I have to ask: what’s performance like for operations on the bucket objects? Edit: I ask because AWS suggests a key naming convention for large object amounts to ensure that you're distributing your objects across storage nodes, to prevent bottlenecks. https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate...
- DVassallo 7y ago
  No difference for Put, Get and Delete. Don't know about List, but if it degrades it's not significant. I worked with buckets with exabytes of data and billions of objects.
- meritt 7y ago
  Never noticed any speed difference due to bucket size. S3 is generally slow anyway (250ms for a write isn't uncommon) but it scales very well and we use it for raw data storage that's not in our critical path, so the latency isn't a problem. Edit Response: I've always used the partitioning conventions they suggest so not sure what sort of impact you encounter without.
- votepaunchy 7y ago
  “This S3 request rate performance increase removes any previous guidance to randomize object prefixes to achieve faster performance. That means you can now use logical or sequential naming patterns in S3 object naming without any performance implications.” https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3-announces-increased-request-rate-performance/ https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3...
  - deleted 7y ago
    [deleted]
duality 7y ago
But would a batch delete by any other name smell just as sweet?
cheeze 7y ago
> don't ever do this, it was a terrible idea I have billions of files. What do?
- meritt 7y ago
  For us, it was due to the relatively high PUT cost if you're storing a large number of small files. We ended up changing our approach and we now store blocks (~10MB archives) instead of individual files into S3. The S3 portion of our AWS bill was previously 50% PUT / 50% long-term storage charges. After the change, we managed to reduce the PUT aspect to nearly $0 and reduced our overall AWS bill by almost 30%, while still storing the same amount of data per month. e.g. if you write 1 million 10KB files per day to S3, you're looking at $150/mo in PUT costs. If you instead write 1,000 10MB blocks, you're looking at $0.15/mo in PUT costs. Due to S3's support of HTTP range requests, we can still request individual files without an intermediate layer (though our write layer did slightly increase in complexity) and our GET (and storage) costs are identical.

neonate 7y ago

Wherefore means why, kind of like "what for". You want just "where", though that breaks the quote.

johnhenry 7y ago
It's plausible that zacharyozer refers to the batch delete that exists in his or her heart. In which case "wherefore" would be appropriate.