3 ms·

We have a large s3 bucket 2 billions objects and we start thinking about cleaning it a bit. Is anyone had done such things or any tools on : - categorising wh

We have a large s3 bucket 2 billions objects and we start thinking about cleaning it a bit. Is anyone had done such things or any tools on : - categorising what's inside - checking what's used or not Thanks!

ben509 7y ago
The first step would be figuring out what's in there, so maybe look at Glue[1] and see if it can determine your existing schema. But usually you need to run run arbitrary code against the contents of a large S3 bucket, and that gets tricky. The main problem is tracking what you've done vs. what you need to do, because if you haven't categorized your data yet, you can expect that code processing it will break. One technique is queues in SQS: 1. Keys to process 2. Keys that succeeded 3. Keys that failed (Regular queues, FIFO queues probably won't be useful. A queue can have an unlimited backlog, but the maximum message timeout is two weeks. That's probably more than enough time to iterate over some code in Lambda.) Your initial lambda should be triggered by KeysToProcess, which you can initiate off a developer machine and just run ListBucket and create a pile of messages. When the lambda is done, it passes its information to KeysThatSucceeded. (Or possibly another S3 bucket, or Dynamo, or a database, or just drop its key if you determine you don't need it.) Point your dead letter queue to KeysThatFailed. Let the messages pile up in there until you've figured out the errors and are ready to try again. And then you can trigger off KeysThatFailed, point the dead letter queue at a new KeysThatFailed2, rinse, repeat until you're satisfied it's correct. [1]: https://aws.amazon.com/glue/ https://aws.amazon.com/glue/
mcqueenjordan 7y ago
https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inve... S3 Inventory is basically designed for your use case. It effectively writes a full-bucket index every day.
Reedx 7y ago
Relatedly, I'm curious how people are handling local backups with large S3 buckets? I've been using AWS CLI sync, but it's getting increasingly slow. To the point that it seems untenable.

ben509 7y ago

The first step would be figuring out what's in there, so maybe look at Glue[1] and see if it can determine your existing schema. But usually you need to run run arbitrary code against the contents of a large S3 bucket, and that gets tricky. The main problem is tracking what you've done vs. what you need to do, because if you haven't categorized your data yet, you can expect that code processing it will break. One technique is queues in SQS: 1. Keys to process 2. Keys that succeeded 3. Keys that failed (Regular queues, FIFO queues probably won't be useful. A queue can have an unlimited backlog, but the maximum message timeout is two weeks. That's probably more than enough time to iterate over some code in Lambda.) Your initial lambda should be triggered by KeysToProcess, which you can initiate off a developer machine and just run ListBucket and create a pile of messages. When the lambda is done, it passes its information to KeysThatSucceeded. (Or possibly another S3 bucket, or Dynamo, or a database, or just drop its key if you determine you don't need it.) Point your dead letter queue to KeysThatFailed. Let the messages pile up in there until you've figured out the errors and are ready to try again. And then you can trigger off KeysThatFailed, point the dead letter queue at a new KeysThatFailed2, rinse, repeat until you're satisfied it's correct. [1]: https://aws.amazon.com/glue/ https://aws.amazon.com/glue/

mcqueenjordan 7y ago

https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inve... S3 Inventory is basically designed for your use case. It effectively writes a full-bucket index every day.

Reedx 7y ago

Relatedly, I'm curious how people are handling local backups with large S3 buckets? I've been using AWS CLI sync, but it's getting increasingly slow. To the point that it seems untenable.