3 ms·
I can confirm billions of files/keys. I went the quick/dirty route to parallelize the DELETEs and just let the OS manage the async I/O for me at the proc level.
by jvaleski 16y ago
I can confirm billions of files/keys. I went the quick/dirty route to parallelize the DELETEs and just let the OS manage the async I/O for me at the proc level. Each instance spun up 150 Ruby procs that issued the DELETEs (tight, serial, while-loop style), and ran them in parallel.
I spent quite a bit of time working w/ the S3 team at Amazon to try and come up with a better model, but we all agreed on my approach (mind you, it could likely be done more efficiently) as option 'a', and option 'b' was to delete the account and just let Amazon reap the data.