13 ms·

Deleting an S3 Bucket Costs Money

290 points by fideloper 5y ago

devnull3 5y ago
How much of this is a problem in practice? I think in practice, S3 data is often indexed using other DBs e.g DynamoDB, Postgres, MySQL etc. Can't this index be used to enumerate all S3 URLs? I am off-course simplifying this a lot.
- Manfred 5y ago
  Developers are pretty sloppy with maintaining references to object in my experience. It generally only becomes a problem when you need to clean up and that usually at time of IPO when there are petabytes of data in S3.
- onion2k 5y ago
  How much of this is a problem in practice? This specific issue probably isn't a very big problem. The issue of Amazon repeatedly coming up on HN as a service that will bill you when you're not unexpecting it for things that are moderately hard to understand and might refund you later probably costs them tens or even hundreds of millions in lost revenue every year from developers being cautious about deploying things to their services.
  - jasonlotito 5y ago
    > probably costs them tens or even hundreds of millions in lost revenue every year from developers being cautious about deploying things to their services I see this as a good thing. They literally encourage you to be cautious with your pricing and resource usage, to the point where they put limits on what resources you can use without explicitly asking for more. Developers should be cautious and aware.
    - ghaff 5y ago
      I'm not sure it's a good thing. But open-ended charges come up a lot. AWS, like the other big public cloud providers, doubtless have self-interested reasons for letting people run up the meter. But they're probably also hesitant to add cut-offs that could end up being footguns in a production context to bring down services in complex infrastructures. People just playing around should be careful. There are ways to keep risk lower. But if absolute price caps are your priority you should probably be using a VPS of some sort.
  - devnull3 5y ago
    > things that are moderately hard to understand My experience with AWS is that the pricing for each service is reasonably well documented and the calculator does descent job. The problem starts when multiple combinations of services are used and it becomes harder to reason. With the advent of cloud, cost-modelling becomes an essential skill (which can be learned). One needs to be clear about total work that gets done and "how" that work gets processed. This in turn should translate to relevant cost metric (e.g PUT requests/s for S3 or IOPS for DynamoDB, amount of data scanned for Athena, etc) This needs to be evaluated for zero load, normal load, 5x load, 20x load, etc. Zero load gives what is dead weight cost of the system i.e cost incurred when no work is being done (e.g EC2, EBS volumes, etc)
Eikon 5y ago
Ok pretty obvious, but if you don't know what you are storing inside your bucket, how are you accessing your objects in the first place ? If your use-case is storing random things you don't know the path of, maybe it's the wrong product to use.
- shric 5y ago
  A common pattern is to dump log files or to use as a dead letter queue for failed event processing. These things typically are arbitrarily named by e.g. a prefix and a unix timestamp
- afandian 5y ago
  One of the great features of S3 is that it has an arbitrary prefix index on keys! And the list API paginates on batches of 1000 which is useful. You can retrieve all objects with a given prefix, which is great for storing content-addressed files, and being able to iterate on them. You can also partition on arbitrary prefixes too.
- CobrastanJorji 5y ago
  Here's an example. You're doing a big machine learning workflow and you've got a gazillion large files representing training data. You run a large training job in Amazon's cloud somewhere that is composed of one worker that lists the files and delegates work to various learning jobs, and then those jobs each stream in one training file, process it, and then grab the next item and repeat. That's a pretty common type of work.
WORMS_EAT_WORMS 5y ago
And deleting your AWS account will keep billing you [1] if you don’t delete all resources first. AWS is designed to extract dollars from big enterprise contracts. Also interesting from the article, this poor soul on StackOverflow was trying to figure out how to delete a bucket that would cost him $20,000 [2]. Can’t delete, can’t close. [1] https://www.reddit.com/r/aws/comments/j5nh4w/ive_deleted_my_account_but_amazon_keeps_billing/ https://www.reddit.com/r/aws/comments/j5nh4w/ive_deleted_my_... [2] https://stackoverflow.com/questions/54255990/cheapest-way-to-delete-2-billion-objects-from-s3-ia https://stackoverflow.com/questions/54255990/cheapest-way-to...
- bluelu 5y ago
  A long time ago, we deleted/disabled our aws account and I assumed that all my files on s3 would also get deleted. When we reactivated the account a few yearss later, we were retroactively billed for all the files in the s3 bucket. We got the money back though.
  - koolba 5y ago
    That sounds like a great long term “shit hits the fan” backup idea: 1) Upload an encrypted blob to an S3 unique AWS account with a burner credit card. 2) Cancel the account. 3) If you ever need to restore the data, restore the account and pay the difference. Since uploads are free you just do this to a new account every so often and you’ll only need to pay the time difference of the most recent backup!
    - personjerry 5y ago
      This defeats the purpose of a backup because they could delete your data at any time.
      - jopsen 5y ago
        Yeah, but the idea of a backup service where you only pay when you need to restore isn't bad :)
mweberxyz 5y ago
Would the S3 inventory help here? That would allow you to get the list of all files (albeit on a delay similar to the lifecycle rule approach), which you could process offline to generate the DELETEs. https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/storag...
- lathiat 5y ago
  “ For information about Amazon S3 inventory pricing, see Amazon S3 pricing” Is also mentioned in the article though they don’t calculate the price.
- jffry 5y ago
  S3 inventory would cost $0.0025 per million objects listed [1], while LIST requests are $0.005 per thousand requests and each LIST request can return up to 1000 objects, making them $0.005 per million objects listed. For the "Infrequent Access" storage tier, LISTs cost double that. So S3 inventory would be half price compared to LIST (or quarter price in IA storage class), but that's still small comfort if you're staring down the barrel of a bucket containing a large number of objects. [1] Management & analytics tab on https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/
  - deleted 5y ago
    [deleted]
  - vdm 5y ago
    "Requests & data retrievals" tab footnote, above the grid: > LIST requests for any storage class are charged at the same rate as S3 Standard PUT, COPY, and POST requests. I read this as LISTs do not cost double for infrequent access, even though other Tier 1 requests do.
    - jffry 5y ago
      What a confusing mess. If LIST pricing is separate then it's strange to include it in the column of a table of storage-class pricing like that. But I agree that's what that statement seems to be saying. AWS's own pricing calculator doesn't split out LIST requests: https://calculator.aws/#/createCalculator/S3 https://calculator.aws/#/createCalculator/S3 Either way, the takeaway is that using LIST or a bucket inventory, will still be O(N) cost and there's only a factor-of-2-ish difference between the two. Then again, a billion objects is $5 territory to delete, and if you have a trillion objects to delete and no pre-existing listing to go off of, then odds are you can stomach the $5000 hit more easily than you could stomach the staff time spent trying to reduce that cost!
xwdv 5y ago
Another reason to switch to R2.
- throwdecro 5y ago
  Does R2 have fewer expensive footguns than AWS?
  - andrewxdiamond 5y ago
    R2 isn’t even available yet, so only time will tell
  - xwdv 5y ago
    Yes.
jugg1es 5y ago
They have an example of some person almost paying $20k on transition fees. In my early days of AWS, I racked up $90k on S3 transition fees. Thankfully, AWS forgave it.
- deleted 5y ago
  [deleted]
  - deleted 5y ago
    [deleted]
  - deleted 5y ago
    [deleted]
- onion2k 5y ago
  Stories of forgiven fees are an example of survivorship bias. Developers who rack up thousands in AWS charges by mistake and aren't forgiven probably don't tell too many people about the time they screwed up and cost their company a lot of money.
  - caymanjim 5y ago
    I don't think this scenario happens much unless people don't try to resolve it. I've worked with AWS for a long time at many companies, and support always takes care of these things. Things is just another anecdote, but I've never heard of AWS refusing to refund in these scenarios, even when it's entirely the customer's fault.
edwnj 5y ago
Its silly that they wont just let you delete the whole bucket but this actually pretty cheap tho. Based on some quick maths, deleting a million files would only cost you like $5. P.S. Again its silly they do this and I'm probably greatly underestimating how these costs can add up for mid to large orgs.
- edwnj 5y ago
  UPDATE: i fucked up the maths, thought it was $0.005 per list request. Its $0.005 for 1k list requests, so its more like $0.005 per million objects.. thats really really cheap.
  - deleted 5y ago
    [deleted]
- 238475235243 5y ago
  Yes but some of us have many billions of objects. From memory, 4 billion object is $20k to delete.
  - mytailorisrich 5y ago
    I think the takeaway is to maintain your own database of references to objects if you can.
    - ignoramous 5y ago
      S3 isn't really built for lots of small objects, which is also quite tellingly reflected in its pricing (and performance in dealing with those). Right now, anything less than 400KB should really be plonked into its cousin, DynamoDB. Other than that, I fully expect AWS to announce a new S3 bucket type (and pricing) for high-volume, small-size blobs. There is also a small matter of addressing Cloudflare R2, which should result in a Lighsail-EC2-esque fork of S3.
  - billyhoffman 5y ago
    No it’s not. Deleting 4B objects costs $20, via DELETE and LIST requests. As the article says DELETEs are free and you can do bulk deletes of 1000 objects at a time. However you need to have the object names. You get those using LIST, which gives you 1000 items for each request. LISTs are currently priced at $0.005 per 1000 List requests. So $0.005 to delete 1M objects. Using the “empty bucket” feature does this internally and charges you that exact same amount. The only way you get near your price is if you try to delete by applying a new lifecycle policy to 4B objects that are not in Standard storage.
JackC 5y ago
Per-object costs can be tricky with S3 -- it's easy to mentally round costs less than 1/10th of a penny to zero, and then look up a few years later and realize you have hundreds of millions of things and can't afford to do anything with them. When this bit us on a project I made a tool to solve our particular problem, which tars files, writes csv indexes, and can fetch individual files from the tars if need be.[1] Running on millions of files was janky enough that I also ended up scripting an orchestrator to repeatedly attempt each step of the pipeline.[2] Not tested on data other than ours but could be a useful starting point. [1] https://github.com/harvard-lil/s3mothball https://github.com/harvard-lil/s3mothball [2] https://github.com/harvard-lil/mothball_pipeline https://github.com/harvard-lil/mothball_pipeline
deleted 5y ago
[deleted]
NabiDev 5y ago
> In 2021, anyone who comes across this question may benefit to know that AWS console now provides an empty button. source : https://stackoverflow.com/a/67834172 https://stackoverflow.com/a/67834172
- bryan0 5y ago
  According to the article the empty button still calls LIST per 1000 objects. So if the guy in the SO thread has 2B object this one click would still cost him ~$10k ??
  - fullsailor 5y ago
    LIST requests are priced by the 1000. And are priced the same at all storage classes. My estimate is that it would cost him $1 to click that button. 2M LIST requests = (2B objects / 1000 per LIST) $1 = (2M / 1000) × $0.005 At least that's my reading of the Pricing page for US East (Ohio).
    - bryan0 5y ago
      Ah ok. That would at least make more sense.
- tedivm 5y ago
  > Within the last year, AWS added a handy Empty button to the S3 console when viewing a bucket. You can click that button and watch the S3 console make API calls on your behalf. > Here's what it does: It calls a LIST on the bucket, pagination through the objects in the bucket 1000 at a time. It calls a DeleteObjects API method, deleting 1000 at a time. > The cost is 1 API LIST call per 1000 objects in the bucket. Delete operations are free, so there's no extra cost there. source: I read the article.
rad_gruchalski 5y ago
Yup. And uploading / downloading large objects from S3 incurs tons of requests because S3 client does parallel chunking with a small number of other control requests. That client works on the same premise as SFTP client. It’s amazing how often it retries. Example from go sdk: https://github.com/aws/aws-sdk-go/blob/main/service/s3/s3manager/download.go#L303 https://github.com/aws/aws-sdk-go/blob/main/service/s3/s3man....
CSDude 5y ago
You can use lifecycle policies to delete it for free, but its best to confirm it via support. Not saying this is the great way, maybe its intentionally hidden, but at least there is a way. https://stackoverflow.com/questions/59170391/s3-lifecycle-expiration-do-object-expiry-deletes-cost-money-for-sia-objects https://stackoverflow.com/questions/59170391/s3-lifecycle-ex...
- deleted 5y ago
  [deleted]
- tyingq 5y ago
  There's some caveats to that... "If you create an S3 Lifecycle expiration rule that causes objects that have been in S3 Standard-IA or S3 One Zone-IA storage for less than 30 days to expire, you are charged for 30 days" It goes on to say 90 days for Glacier and 180 days for Glacier Deep. https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecy...
  - electroly 5y ago
    That isn't a property of using the lifecycle policy, though. It would be the same if you manually deleted the files.
- fideloper 5y ago
  Lifecycle rules are the preferred (cheapest) way to do it for standard-storage objects in S3 for sure!
- treesknees 5y ago
  Yes, this was mentioned as the preferred method in the article. As the article states, it's free for objects in the standard storage tier but will incur a Transition cost for other tiers. It's not hidden, but it's not exactly advertised as a way to empty a bucket.
  - iconara 5y ago
jonplackett 5y ago
If s3 was any service the general public use, it would be banned for sure.
userbinator 5y ago
The first thing everyone who tries using cloud services should learn: everything costs money. Even the service that tells you how much it costs: https://aws.amazon.com/aws-cost-management/pricing/ https://aws.amazon.com/aws-cost-management/pricing/
- tg180 5y ago
  I'm still learning about AWS, but I don't think other clouds do the same. If there are still those who do not use the cloud, it is because the big three have taken advantage of their position a lot. The pricing of Hetzner, CloudFlare, Linode, OVH, ... seems to be cheaper and more transparent.
  - caymanjim 5y ago
    Those small players aren't comparable to AWS. They might be sufficient for your needs, but the only players at AWS scale are Azure, Google, and Alibaba. Their prices and billing practices are basically identical.
    - tg180 5y ago
      Exactly, they have a commercial offer and a pricing more suited to large industries, where the scale is really needed. For a large industry, the cost of deleting a bucket is not even a thought, for an SMB or a developer it's a problem. If you don't use special tools not available elsewhere, such as AWS' SageMaker, or Google's TPUs, ...., then it's probably not economically interesting to use the Amazon, Microsoft or Google clouds.
StratusBen 5y ago
Most things on AWS cost money and AWS makes pricing incredibly complex and opaque...where the monthly bill is usually the first way people find out about these things. While it is likely no consolation, S3 is by far one of the most complex AWS products pricing-wise with different object storage types each with their own rates, request costs with different rates for GET/PUT/POST/etc which this post mentions, and transit/egress fees. I work on https://www.vantage.sh/ https://www.vantage.sh/ which helps teams get visibility on their cloud costs which may be helpful to folks here as well on this topic.
- nostrebored 5y ago
  I've literally never had a problem explaining S3 costs to a customer. There are complex AWS products for pricing, but this just feels like a plug
Someone 5y ago
> AWS is "eventually consistent" within most services, and S3 is no exception Nowadays, it (¿almost?) is. https://aws.amazon.com/s3/consistency/ https://aws.amazon.com/s3/consistency/: “After a successful write of a new object, or an overwrite or delete of an existing object, any subsequent read request immediately receives the latest version of the object” I think that says that deletes are immediately visible, too, but they phrase it weirdly, as, after a delete, there is no latest version of the object. Also, I don’t think buckets are objects in this sense, so the caveat in the article stands.
- fideloper 5y ago
  I believe objects in the bucket now are strongly consistent, but the buckets themselves are still eventually consistent.
  - andrewxdiamond 5y ago
    Said using the official terms; data plane operations are strongly consistent whereas control plane operations are eventually consistently.
- pram 5y ago
  Yes, when you delete with bucket versioning on it will leave a record of the object deletion tombstone. The previous object versions are still there obviously.
malka 5y ago
DELETE is free LIST is not I guess one could spam DELETE calls while bruteforcing filenames to make it free.
- toomuchtodo 5y ago
  An S3 inventory report [1] is $0.0025 per million objects listed versus $0.005 per LIST request. Make an inventory report request, retrieve the inventory report from S3, make your free delete calls synchronously with API calls or pay for a batch job [2] if you'd rather not write code ($0.25 per batch job and $1.00 per million object operations performed). It's listed in the post as "extra credit", but its trivial having done it myself for a client. A bit disingenuous to state, "if you like to do things the hard way." It literally takes less than hour to do, and you can preserve the inventory report for housekeeping if needed. [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/storag... [2] https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-create-job.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-... (OP: feel free to steal this comment's info if you want to update your post)
  - boplicity 5y ago
    > It literally takes less than hour to do Yikes. An hour just to delete some files in a folder? It seems to me that S3 should be avoided unless you really, really, really need it.
    - emaginniss 5y ago
      When the scale of data goes up, so does the complexity of managing it. Deleting 100 rows from a live psql table with 1000 entries is very different from deleting 100M rows from a live table with 1B entries.
  - vdm 5y ago
    S3 Batch Operations list of Supported Operations[0] does not include DeleteObject or DeleteObjects. 0. https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-operations.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-...
pkorzeniewski 5y ago
Pricing of AWS services makes me uneasy in general, just take the S3 as an example - you go to the pricing page and you have several tabs with dozens of entries which makes calculating how much exactly will you pay difficult. I might be simple minded but I prefer a clearly defined plans with predetermined limits - you know exactly what it costs you each month and what you get and if you need more, just switch to a higher plan, no risk of nasty (and often expensive) surprises like mentioned in the article.
- namelessoracle 5y ago
  Its ALMOST like one of the most lucrative businesses in the world is optimizing extracting money from you rather than customer experience.
- mwarkentin 5y ago
  You could try their Lightsail object storage (and other lightsail services) for this type of pricing: https://aws.amazon.com/lightsail/pricing/ https://aws.amazon.com/lightsail/pricing/
  - tpetry 5y ago
    I didn‘t know they extended lightsail to more than VMs. In S3 every request costs money, but in lightsail object storage they are free and you pay bandwidth?
- dec0dedab0de 5y ago
  This is why I've never used it. It always felt like I was getting 10 records for a penny.
- citizenpaul 5y ago
  Problem is that customer pricing demands are infinitely granular, with i only want to pay for X that i use. The alternative is the tiers which have their own problems. Small Medium Big Call us No one know if they are gonna have success in the beginning or need to scale up. Usually. I guess at a size like AWS it makes more sense to have those infinite pricing tiers and let the customers figure out how much they are willing to pay rather than try to negotiate with huge swaths of people that take sales staff salary to deal with.
annoyingnoob 5y ago
This kind of thing is the best argument for your own bare metal hardware.
- marcosdumay 5y ago
  Or at least get some service with transparent pricing, instead of per-object per-action bullshit.
peanut_worm 5y ago
Stories like this make me extremely hesitant to try AWS. I was about to try S3 for a static site I was working on this weekends but I think I am gonna stick with netlify or digital ocean instead after reading this.
- certifiedloud 5y ago
  Having worked with S3+Cloudfront as well as Netlify, I can say with confidence that Netlify is better anyways. And it's hard to beat free...
- input_sh 5y ago
  In theory, I'm sure you could do that with the free tier. In practice, activating that free tier requires a valid card, and I'd highly advise never giving them your own. Whatever alternative you can think of is 100% better for your sanity.
  - danlugo92 5y ago
    They accept prepaid gift cards as valid.
- Saris 5y ago
  There are also other S3 providers with much more sensible pricing, like Backblaze B2, Wasabi, or Cloudflare R2.
known 5y ago
Now we can imagine the toxic working conditions in AMZ. "When you can't make them see the light, make them feel the heat” --Ronald Reagan (b. 1911)
bob229 5y ago
So what? What is the point of this article. It is stating the obvious to even the most n00b of AWS users, of which I count myself as one
donatj 5y ago
> .5¢ per 1000 items LISTed seems insanely expensive considering how cheaply you can transfer terabytes of data with S3. Correction: I misread - .5¢ per 1,000,000 items LISTed .5¢ per 1000 LIST operations LIST operations max out at 1000 items Still a little pricey, but way less so than I'd imagined. Do they make a lot of money off of charging for basic operations? It seems like you could make the whole pricing structure a lot more friendly by only charging for bandwidth use. I guess when you're as dominant as S3, you don't need to care about friendly pricing structures. Charging for basic operations like that is weird, it's akin to a service charging people per number of clicks on a website.
- codazoda 5y ago
  There's money in confusion... I'm terrified of using the existing cloud services for personal projects. For business projects you can mostly just get an idea of how much your month-to-month bill will increase with certain actions, but it's sure easy to blow a budget by accident.
  - mywittyname 5y ago
    You should probably set up an LLC to handle billing for your personal projects if you plan on going with a big cloud provider. It sucks, but the fact is AWS/GCP/Azure aren't really designed for hobbyist, they are designed for massive corporations. Their free tiers exist merely as a service to help train professionals to use their platforms. Luckily, there are still good, low-cost providers out there.
- dahart 5y ago
  > .5¢ per 1000 items LISTed seems insanely expensive Note it’s $0.005 per 1k requests, not $0.05 per 1k items -- that’s an extra zero from what you said, and also important to point out that one request can list 1k items. So if you list in 1k batches, it’s $5 per million items listed.
  - electroly 5y ago
    > So if you list in 1k batches, it’s $5 per million items listed. It's $0.005 per million items listed. A thousand requests of a thousand items each is your million items, and a thousand requests is $0.005.
Decabytes 5y ago
This post finally got my ass in gear to cancel an account that I thought I had closed but was still charging me a few dollars a month. I spinned up an AWS instance to practice, and once I was done I thought I closed everything down. Turns out I had just stopped my micro instances, and I didn't terminate them. I also hadn't released the my IP address. There was also a snapshot of the tiny db I had created still floating around. The documentation was a little confusing, so after I went through it I spent half an hour chatting with a support rep to make sure everything was completely good. After next month my last bill should go through and I should be free and clear. Unfortunately I have to wait for next months bill to go through as I can't just pay it all now. This was mostly my fault for letting it go on for so long, but I hate how if you don't do some very specific steps you can still be charged. And I think if an account is closed, it should absolutely terminate all services that are still running on that account, and then send you the final bill.
- roland35 5y ago
  I've made the same mistake! Also check for EBS images which can be sneaky.
- fideloper 5y ago
  If it happens again, you can try out the aws-nuke tool to help you destroy resources. (Because of course an open source tool is needed for this )
helium 5y ago
Not only is a problem that deleting a bucket costs money, but if you have a big bucket with many deeply nested files, it can take a really long time to clean it up using the AWS command line. I ran into this with a bucket full of EMR log files a few years ago and had to figure out some pretty crazy command line hackiness, plus running on a EC2 machine with lots of cores to figure it out. This a write-up I did if anyone else ever runs into this issue. https://gist.github.com/michael-erasmus/6a5acddcb56548874ffe780e19b7701d https://gist.github.com/michael-erasmus/6a5acddcb56548874ffe...
- 404mm 5y ago
  Yup, similar experience. Our devs kept using S3 as a caching backend for some small pictures. Only based on billing, we learned that we had over 17TB in tiny files, unable to groom it in any way that was feasible. Kept hitting all sorts of api limits.
- kalleboo 5y ago
  These days the "easy" way to delete a bucket of a billion tiny files is to configure a very short-term expiration rule on the bucket, and let AWS itself delete all your files as they expire. When we did this (a few years ago) it still took several days for it to remove all the files.
- klodolph 5y ago
  I believe the nesting shouldn’t affect it. When you’re iterating over objects to delete them, you can just iterate over the keys and ignore nesting—I believe that’s how the s3 tools do “recursive” deleting. The underlying S3 API provides a recursive interface (the “delimiter” parameter) but keys are really just string keys, and directories are illusory. But yes, it can take a while to iterate through the objects.
  - movedx 5y ago
    > I believe the nesting shouldn’t affect it It won't effect it, because there is no concept of nesting in an object storage engine like S3. Everything is a flat key that references an object, but we just abstract and conceptualize a directory structure because it makes it easier for us to manage our data. But in reality you just have a really, really long list of keys and a big, flat file system of objects.
shortlived 5y ago
Anyone have suggestions for S3 alternatives for storing many files sized 50-500mb each? They are mostly long audio files and there is an external index as well.
vdm 5y ago
> you can also get an export of all objects in a bucket using S3 Inventory and run the output through AWS Batch in order to delete those objects "S3 Batch Operations" sends S3 requests based on a csv file, which can but does not have to be from S3 Inventory. But S3 Batch Operations supports only a subset of APIs and this does not include DeleteObject(s). [0] An AWS Batch job could run a container which sends DeleteObjects requests but only when triggered by a job queue which seems redundant here. If I can't use an expiration lifecycle policy because I need a selection of objects not matching a prefix or object tags, I would run something with `s5cmd rm` [1]. Alternatively roll your own golang which parses the CSV and sends many DeleteObjects requests in parallel goroutines. 0. https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-operations.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-... 1. https://github.com/peak/s5cmd#delete-multiple-s3-objects https://github.com/peak/s5cmd#delete-multiple-s3-objects
Hamuko 5y ago
>The wait is often hours until AWS released a bucket name (since bucket names are globally unique, not just within your account). I think last time I did this, the wait time was pretty much exactly 60 minutes.
vrnchndk 5y ago
> Deleting a bucket won't let you re-create that bucket immediately. This is partially incorrect. I can recreate it immediately in the same account, but in different account, I need to wait for ~1 hour

290 points by fideloper 5y ago

How much of this is a problem in practice? I think in practice, S3 data is often indexed using other DBs e.g DynamoDB, Postgres, MySQL etc. Can't this index be used to enumerate all S3 URLs? I am off-course simplifying this a lot.

Manfred 5y ago
Developers are pretty sloppy with maintaining references to object in my experience. It generally only becomes a problem when you need to clean up and that usually at time of IPO when there are petabytes of data in S3.
onion2k 5y ago
How much of this is a problem in practice? This specific issue probably isn't a very big problem. The issue of Amazon repeatedly coming up on HN as a service that will bill you when you're not unexpecting it for things that are moderately hard to understand and might refund you later probably costs them tens or even hundreds of millions in lost revenue every year from developers being cautious about deploying things to their services.
- jasonlotito 5y ago
  > probably costs them tens or even hundreds of millions in lost revenue every year from developers being cautious about deploying things to their services I see this as a good thing. They literally encourage you to be cautious with your pricing and resource usage, to the point where they put limits on what resources you can use without explicitly asking for more. Developers should be cautious and aware.
  - ghaff 5y ago
    I'm not sure it's a good thing. But open-ended charges come up a lot. AWS, like the other big public cloud providers, doubtless have self-interested reasons for letting people run up the meter. But they're probably also hesitant to add cut-offs that could end up being footguns in a production context to bring down services in complex infrastructures. People just playing around should be careful. There are ways to keep risk lower. But if absolute price caps are your priority you should probably be using a VPS of some sort.
- devnull3 5y ago
  > things that are moderately hard to understand My experience with AWS is that the pricing for each service is reasonably well documented and the calculator does descent job. The problem starts when multiple combinations of services are used and it becomes harder to reason. With the advent of cloud, cost-modelling becomes an essential skill (which can be learned). One needs to be clear about total work that gets done and "how" that work gets processed. This in turn should translate to relevant cost metric (e.g PUT requests/s for S3 or IOPS for DynamoDB, amount of data scanned for Athena, etc) This needs to be evaluated for zero load, normal load, 5x load, 20x load, etc. Zero load gives what is dead weight cost of the system i.e cost incurred when no work is being done (e.g EC2, EBS volumes, etc)

Eikon 5y ago

Ok pretty obvious, but if you don't know what you are storing inside your bucket, how are you accessing your objects in the first place ? If your use-case is storing random things you don't know the path of, maybe it's the wrong product to use.

shric 5y ago
A common pattern is to dump log files or to use as a dead letter queue for failed event processing. These things typically are arbitrarily named by e.g. a prefix and a unix timestamp
afandian 5y ago
One of the great features of S3 is that it has an arbitrary prefix index on keys! And the list API paginates on batches of 1000 which is useful. You can retrieve all objects with a given prefix, which is great for storing content-addressed files, and being able to iterate on them. You can also partition on arbitrary prefixes too.
CobrastanJorji 5y ago
Here's an example. You're doing a big machine learning workflow and you've got a gazillion large files representing training data. You run a large training job in Amazon's cloud somewhere that is composed of one worker that lists the files and delegates work to various learning jobs, and then those jobs each stream in one training file, process it, and then grab the next item and repeat. That's a pretty common type of work.

WORMS_EAT_WORMS 5y ago

And deleting your AWS account will keep billing you [1] if you don’t delete all resources first. AWS is designed to extract dollars from big enterprise contracts. Also interesting from the article, this poor soul on StackOverflow was trying to figure out how to delete a bucket that would cost him $20,000 [2]. Can’t delete, can’t close. [1] https://www.reddit.com/r/aws/comments/j5nh4w/ive_deleted_my_account_but_amazon_keeps_billing/ https://www.reddit.com/r/aws/comments/j5nh4w/ive_deleted_my_... [2] https://stackoverflow.com/questions/54255990/cheapest-way-to-delete-2-billion-objects-from-s3-ia https://stackoverflow.com/questions/54255990/cheapest-way-to...

bluelu 5y ago
A long time ago, we deleted/disabled our aws account and I assumed that all my files on s3 would also get deleted. When we reactivated the account a few yearss later, we were retroactively billed for all the files in the s3 bucket. We got the money back though.
- koolba 5y ago
  That sounds like a great long term “shit hits the fan” backup idea: 1) Upload an encrypted blob to an S3 unique AWS account with a burner credit card. 2) Cancel the account. 3) If you ever need to restore the data, restore the account and pay the difference. Since uploads are free you just do this to a new account every so often and you’ll only need to pay the time difference of the most recent backup!
  - personjerry 5y ago
    This defeats the purpose of a backup because they could delete your data at any time.
    - jopsen 5y ago
      Yeah, but the idea of a backup service where you only pay when you need to restore isn't bad :)

mweberxyz 5y ago

Would the S3 inventory help here? That would allow you to get the list of all files (albeit on a delay similar to the lifecycle rule approach), which you could process offline to generate the DELETEs. https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/storag...

lathiat 5y ago
“ For information about Amazon S3 inventory pricing, see Amazon S3 pricing” Is also mentioned in the article though they don’t calculate the price.
jffry 5y ago
S3 inventory would cost $0.0025 per million objects listed [1], while LIST requests are $0.005 per thousand requests and each LIST request can return up to 1000 objects, making them $0.005 per million objects listed. For the "Infrequent Access" storage tier, LISTs cost double that. So S3 inventory would be half price compared to LIST (or quarter price in IA storage class), but that's still small comfort if you're staring down the barrel of a bucket containing a large number of objects. [1] Management & analytics tab on https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/
- deleted 5y ago
  [deleted]
- vdm 5y ago
  "Requests & data retrievals" tab footnote, above the grid: > LIST requests for any storage class are charged at the same rate as S3 Standard PUT, COPY, and POST requests. I read this as LISTs do not cost double for infrequent access, even though other Tier 1 requests do.
  - jffry 5y ago
    What a confusing mess. If LIST pricing is separate then it's strange to include it in the column of a table of storage-class pricing like that. But I agree that's what that statement seems to be saying. AWS's own pricing calculator doesn't split out LIST requests: https://calculator.aws/#/createCalculator/S3 https://calculator.aws/#/createCalculator/S3 Either way, the takeaway is that using LIST or a bucket inventory, will still be O(N) cost and there's only a factor-of-2-ish difference between the two. Then again, a billion objects is $5 territory to delete, and if you have a trillion objects to delete and no pre-existing listing to go off of, then odds are you can stomach the $5000 hit more easily than you could stomach the staff time spent trying to reduce that cost!

xwdv 5y ago

Another reason to switch to R2.

throwdecro 5y ago
Does R2 have fewer expensive footguns than AWS?
- andrewxdiamond 5y ago
  R2 isn’t even available yet, so only time will tell
- xwdv 5y ago
  Yes.

jugg1es 5y ago

They have an example of some person almost paying $20k on transition fees. In my early days of AWS, I racked up $90k on S3 transition fees. Thankfully, AWS forgave it.

deleted 5y ago
[deleted]
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
onion2k 5y ago
Stories of forgiven fees are an example of survivorship bias. Developers who rack up thousands in AWS charges by mistake and aren't forgiven probably don't tell too many people about the time they screwed up and cost their company a lot of money.
- caymanjim 5y ago
  I don't think this scenario happens much unless people don't try to resolve it. I've worked with AWS for a long time at many companies, and support always takes care of these things. Things is just another anecdote, but I've never heard of AWS refusing to refund in these scenarios, even when it's entirely the customer's fault.

edwnj 5y ago

Its silly that they wont just let you delete the whole bucket but this actually pretty cheap tho. Based on some quick maths, deleting a million files would only cost you like $5. P.S. Again its silly they do this and I'm probably greatly underestimating how these costs can add up for mid to large orgs.

edwnj 5y ago
UPDATE: i fucked up the maths, thought it was $0.005 per list request. Its $0.005 for 1k list requests, so its more like $0.005 per million objects.. thats really really cheap.
- deleted 5y ago
  [deleted]
238475235243 5y ago
Yes but some of us have many billions of objects. From memory, 4 billion object is $20k to delete.
- mytailorisrich 5y ago
  I think the takeaway is to maintain your own database of references to objects if you can.
  - ignoramous 5y ago
    S3 isn't really built for lots of small objects, which is also quite tellingly reflected in its pricing (and performance in dealing with those). Right now, anything less than 400KB should really be plonked into its cousin, DynamoDB. Other than that, I fully expect AWS to announce a new S3 bucket type (and pricing) for high-volume, small-size blobs. There is also a small matter of addressing Cloudflare R2, which should result in a Lighsail-EC2-esque fork of S3.
- billyhoffman 5y ago
  No it’s not. Deleting 4B objects costs $20, via DELETE and LIST requests. As the article says DELETEs are free and you can do bulk deletes of 1000 objects at a time. However you need to have the object names. You get those using LIST, which gives you 1000 items for each request. LISTs are currently priced at $0.005 per 1000 List requests. So $0.005 to delete 1M objects. Using the “empty bucket” feature does this internally and charges you that exact same amount. The only way you get near your price is if you try to delete by applying a new lifecycle policy to 4B objects that are not in Standard storage.

JackC 5y ago

Per-object costs can be tricky with S3 -- it's easy to mentally round costs less than 1/10th of a penny to zero, and then look up a few years later and realize you have hundreds of millions of things and can't afford to do anything with them. When this bit us on a project I made a tool to solve our particular problem, which tars files, writes csv indexes, and can fetch individual files from the tars if need be.[1] Running on millions of files was janky enough that I also ended up scripting an orchestrator to repeatedly attempt each step of the pipeline.[2] Not tested on data other than ours but could be a useful starting point. [1] https://github.com/harvard-lil/s3mothball https://github.com/harvard-lil/s3mothball [2] https://github.com/harvard-lil/mothball_pipeline https://github.com/harvard-lil/mothball_pipeline

deleted 5y ago

[deleted]

NabiDev 5y ago

> In 2021, anyone who comes across this question may benefit to know that AWS console now provides an empty button. source : https://stackoverflow.com/a/67834172 https://stackoverflow.com/a/67834172

bryan0 5y ago
According to the article the empty button still calls LIST per 1000 objects. So if the guy in the SO thread has 2B object this one click would still cost him ~$10k ??
- fullsailor 5y ago
  LIST requests are priced by the 1000. And are priced the same at all storage classes. My estimate is that it would cost him $1 to click that button. 2M LIST requests = (2B objects / 1000 per LIST) $1 = (2M / 1000) × $0.005 At least that's my reading of the Pricing page for US East (Ohio).
  - bryan0 5y ago
    Ah ok. That would at least make more sense.
tedivm 5y ago
> Within the last year, AWS added a handy Empty button to the S3 console when viewing a bucket. You can click that button and watch the S3 console make API calls on your behalf. > Here's what it does: It calls a LIST on the bucket, pagination through the objects in the bucket 1000 at a time. It calls a DeleteObjects API method, deleting 1000 at a time. > The cost is 1 API LIST call per 1000 objects in the bucket. Delete operations are free, so there's no extra cost there. source: I read the article.

rad_gruchalski 5y ago

Yup. And uploading / downloading large objects from S3 incurs tons of requests because S3 client does parallel chunking with a small number of other control requests. That client works on the same premise as SFTP client. It’s amazing how often it retries. Example from go sdk: https://github.com/aws/aws-sdk-go/blob/main/service/s3/s3manager/download.go#L303 https://github.com/aws/aws-sdk-go/blob/main/service/s3/s3man....

CSDude 5y ago

You can use lifecycle policies to delete it for free, but its best to confirm it via support. Not saying this is the great way, maybe its intentionally hidden, but at least there is a way. https://stackoverflow.com/questions/59170391/s3-lifecycle-expiration-do-object-expiry-deletes-cost-money-for-sia-objects https://stackoverflow.com/questions/59170391/s3-lifecycle-ex...

deleted 5y ago
[deleted]
tyingq 5y ago
There's some caveats to that... "If you create an S3 Lifecycle expiration rule that causes objects that have been in S3 Standard-IA or S3 One Zone-IA storage for less than 30 days to expire, you are charged for 30 days" It goes on to say 90 days for Glacier and 180 days for Glacier Deep. https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecy...
- electroly 5y ago
  That isn't a property of using the lifecycle policy, though. It would be the same if you manually deleted the files.
fideloper 5y ago
Lifecycle rules are the preferred (cheapest) way to do it for standard-storage objects in S3 for sure!
treesknees 5y ago
Yes, this was mentioned as the preferred method in the article. As the article states, it's free for objects in the standard storage tier but will incur a Transition cost for other tiers. It's not hidden, but it's not exactly advertised as a way to empty a bucket.
- iconara 5y ago

jonplackett 5y ago

If s3 was any service the general public use, it would be banned for sure.

userbinator 5y ago

The first thing everyone who tries using cloud services should learn: everything costs money. Even the service that tells you how much it costs: https://aws.amazon.com/aws-cost-management/pricing/ https://aws.amazon.com/aws-cost-management/pricing/

tg180 5y ago
I'm still learning about AWS, but I don't think other clouds do the same. If there are still those who do not use the cloud, it is because the big three have taken advantage of their position a lot. The pricing of Hetzner, CloudFlare, Linode, OVH, ... seems to be cheaper and more transparent.
- caymanjim 5y ago
  Those small players aren't comparable to AWS. They might be sufficient for your needs, but the only players at AWS scale are Azure, Google, and Alibaba. Their prices and billing practices are basically identical.
  - tg180 5y ago
    Exactly, they have a commercial offer and a pricing more suited to large industries, where the scale is really needed. For a large industry, the cost of deleting a bucket is not even a thought, for an SMB or a developer it's a problem. If you don't use special tools not available elsewhere, such as AWS' SageMaker, or Google's TPUs, ...., then it's probably not economically interesting to use the Amazon, Microsoft or Google clouds.

StratusBen 5y ago

Most things on AWS cost money and AWS makes pricing incredibly complex and opaque...where the monthly bill is usually the first way people find out about these things. While it is likely no consolation, S3 is by far one of the most complex AWS products pricing-wise with different object storage types each with their own rates, request costs with different rates for GET/PUT/POST/etc which this post mentions, and transit/egress fees. I work on https://www.vantage.sh/ https://www.vantage.sh/ which helps teams get visibility on their cloud costs which may be helpful to folks here as well on this topic.

nostrebored 5y ago
I've literally never had a problem explaining S3 costs to a customer. There are complex AWS products for pricing, but this just feels like a plug

Someone 5y ago

> AWS is "eventually consistent" within most services, and S3 is no exception Nowadays, it (¿almost?) is. https://aws.amazon.com/s3/consistency/ https://aws.amazon.com/s3/consistency/: “After a successful write of a new object, or an overwrite or delete of an existing object, any subsequent read request immediately receives the latest version of the object” I think that says that deletes are immediately visible, too, but they phrase it weirdly, as, after a delete, there is no latest version of the object. Also, I don’t think buckets are objects in this sense, so the caveat in the article stands.

fideloper 5y ago
I believe objects in the bucket now are strongly consistent, but the buckets themselves are still eventually consistent.
- andrewxdiamond 5y ago
  Said using the official terms; data plane operations are strongly consistent whereas control plane operations are eventually consistently.
pram 5y ago
Yes, when you delete with bucket versioning on it will leave a record of the object deletion tombstone. The previous object versions are still there obviously.

malka 5y ago

DELETE is free LIST is not I guess one could spam DELETE calls while bruteforcing filenames to make it free.

toomuchtodo 5y ago
An S3 inventory report [1] is $0.0025 per million objects listed versus $0.005 per LIST request. Make an inventory report request, retrieve the inventory report from S3, make your free delete calls synchronously with API calls or pay for a batch job [2] if you'd rather not write code ($0.25 per batch job and $1.00 per million object operations performed). It's listed in the post as "extra credit", but its trivial having done it myself for a client. A bit disingenuous to state, "if you like to do things the hard way." It literally takes less than hour to do, and you can preserve the inventory report for housekeeping if needed. [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/storag... [2] https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-create-job.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-... (OP: feel free to steal this comment's info if you want to update your post)
- boplicity 5y ago
  > It literally takes less than hour to do Yikes. An hour just to delete some files in a folder? It seems to me that S3 should be avoided unless you really, really, really need it.
  - emaginniss 5y ago
    When the scale of data goes up, so does the complexity of managing it. Deleting 100 rows from a live psql table with 1000 entries is very different from deleting 100M rows from a live table with 1B entries.
- vdm 5y ago
  S3 Batch Operations list of Supported Operations[0] does not include DeleteObject or DeleteObjects. 0. https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-operations.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-...

pkorzeniewski 5y ago

Pricing of AWS services makes me uneasy in general, just take the S3 as an example - you go to the pricing page and you have several tabs with dozens of entries which makes calculating how much exactly will you pay difficult. I might be simple minded but I prefer a clearly defined plans with predetermined limits - you know exactly what it costs you each month and what you get and if you need more, just switch to a higher plan, no risk of nasty (and often expensive) surprises like mentioned in the article.

namelessoracle 5y ago
Its ALMOST like one of the most lucrative businesses in the world is optimizing extracting money from you rather than customer experience.
mwarkentin 5y ago
You could try their Lightsail object storage (and other lightsail services) for this type of pricing: https://aws.amazon.com/lightsail/pricing/ https://aws.amazon.com/lightsail/pricing/
- tpetry 5y ago
  I didn‘t know they extended lightsail to more than VMs. In S3 every request costs money, but in lightsail object storage they are free and you pay bandwidth?
dec0dedab0de 5y ago
This is why I've never used it. It always felt like I was getting 10 records for a penny.
citizenpaul 5y ago
Problem is that customer pricing demands are infinitely granular, with i only want to pay for X that i use. The alternative is the tiers which have their own problems. Small Medium Big Call us No one know if they are gonna have success in the beginning or need to scale up. Usually. I guess at a size like AWS it makes more sense to have those infinite pricing tiers and let the customers figure out how much they are willing to pay rather than try to negotiate with huge swaths of people that take sales staff salary to deal with.

annoyingnoob 5y ago

This kind of thing is the best argument for your own bare metal hardware.

marcosdumay 5y ago
Or at least get some service with transparent pricing, instead of per-object per-action bullshit.

peanut_worm 5y ago

Stories like this make me extremely hesitant to try AWS. I was about to try S3 for a static site I was working on this weekends but I think I am gonna stick with netlify or digital ocean instead after reading this.

certifiedloud 5y ago
Having worked with S3+Cloudfront as well as Netlify, I can say with confidence that Netlify is better anyways. And it's hard to beat free...
input_sh 5y ago
In theory, I'm sure you could do that with the free tier. In practice, activating that free tier requires a valid card, and I'd highly advise never giving them your own. Whatever alternative you can think of is 100% better for your sanity.
- danlugo92 5y ago
  They accept prepaid gift cards as valid.
Saris 5y ago
There are also other S3 providers with much more sensible pricing, like Backblaze B2, Wasabi, or Cloudflare R2.

known 5y ago

Now we can imagine the toxic working conditions in AMZ. "When you can't make them see the light, make them feel the heat” --Ronald Reagan (b. 1911)

bob229 5y ago

So what? What is the point of this article. It is stating the obvious to even the most n00b of AWS users, of which I count myself as one

donatj 5y ago

> .5¢ per 1000 items LISTed seems insanely expensive considering how cheaply you can transfer terabytes of data with S3. Correction: I misread - .5¢ per 1,000,000 items LISTed .5¢ per 1000 LIST operations LIST operations max out at 1000 items Still a little pricey, but way less so than I'd imagined. Do they make a lot of money off of charging for basic operations? It seems like you could make the whole pricing structure a lot more friendly by only charging for bandwidth use. I guess when you're as dominant as S3, you don't need to care about friendly pricing structures. Charging for basic operations like that is weird, it's akin to a service charging people per number of clicks on a website.

codazoda 5y ago
There's money in confusion... I'm terrified of using the existing cloud services for personal projects. For business projects you can mostly just get an idea of how much your month-to-month bill will increase with certain actions, but it's sure easy to blow a budget by accident.
- mywittyname 5y ago
  You should probably set up an LLC to handle billing for your personal projects if you plan on going with a big cloud provider. It sucks, but the fact is AWS/GCP/Azure aren't really designed for hobbyist, they are designed for massive corporations. Their free tiers exist merely as a service to help train professionals to use their platforms. Luckily, there are still good, low-cost providers out there.
dahart 5y ago
> .5¢ per 1000 items LISTed seems insanely expensive Note it’s $0.005 per 1k requests, not $0.05 per 1k items -- that’s an extra zero from what you said, and also important to point out that one request can list 1k items. So if you list in 1k batches, it’s $5 per million items listed.
- electroly 5y ago
  > So if you list in 1k batches, it’s $5 per million items listed. It's $0.005 per million items listed. A thousand requests of a thousand items each is your million items, and a thousand requests is $0.005.

Decabytes 5y ago

This post finally got my ass in gear to cancel an account that I thought I had closed but was still charging me a few dollars a month. I spinned up an AWS instance to practice, and once I was done I thought I closed everything down. Turns out I had just stopped my micro instances, and I didn't terminate them. I also hadn't released the my IP address. There was also a snapshot of the tiny db I had created still floating around. The documentation was a little confusing, so after I went through it I spent half an hour chatting with a support rep to make sure everything was completely good. After next month my last bill should go through and I should be free and clear. Unfortunately I have to wait for next months bill to go through as I can't just pay it all now. This was mostly my fault for letting it go on for so long, but I hate how if you don't do some very specific steps you can still be charged. And I think if an account is closed, it should absolutely terminate all services that are still running on that account, and then send you the final bill.

roland35 5y ago
I've made the same mistake! Also check for EBS images which can be sneaky.
fideloper 5y ago
If it happens again, you can try out the aws-nuke tool to help you destroy resources. (Because of course an open source tool is needed for this )

helium 5y ago

Not only is a problem that deleting a bucket costs money, but if you have a big bucket with many deeply nested files, it can take a really long time to clean it up using the AWS command line. I ran into this with a bucket full of EMR log files a few years ago and had to figure out some pretty crazy command line hackiness, plus running on a EC2 machine with lots of cores to figure it out. This a write-up I did if anyone else ever runs into this issue. https://gist.github.com/michael-erasmus/6a5acddcb56548874ffe780e19b7701d https://gist.github.com/michael-erasmus/6a5acddcb56548874ffe...

404mm 5y ago
Yup, similar experience. Our devs kept using S3 as a caching backend for some small pictures. Only based on billing, we learned that we had over 17TB in tiny files, unable to groom it in any way that was feasible. Kept hitting all sorts of api limits.
kalleboo 5y ago
These days the "easy" way to delete a bucket of a billion tiny files is to configure a very short-term expiration rule on the bucket, and let AWS itself delete all your files as they expire. When we did this (a few years ago) it still took several days for it to remove all the files.
klodolph 5y ago
I believe the nesting shouldn’t affect it. When you’re iterating over objects to delete them, you can just iterate over the keys and ignore nesting—I believe that’s how the s3 tools do “recursive” deleting. The underlying S3 API provides a recursive interface (the “delimiter” parameter) but keys are really just string keys, and directories are illusory. But yes, it can take a while to iterate through the objects.
- movedx 5y ago
  > I believe the nesting shouldn’t affect it It won't effect it, because there is no concept of nesting in an object storage engine like S3. Everything is a flat key that references an object, but we just abstract and conceptualize a directory structure because it makes it easier for us to manage our data. But in reality you just have a really, really long list of keys and a big, flat file system of objects.

shortlived 5y ago

Anyone have suggestions for S3 alternatives for storing many files sized 50-500mb each? They are mostly long audio files and there is an external index as well.

vdm 5y ago

> you can also get an export of all objects in a bucket using S3 Inventory and run the output through AWS Batch in order to delete those objects "S3 Batch Operations" sends S3 requests based on a csv file, which can but does not have to be from S3 Inventory. But S3 Batch Operations supports only a subset of APIs and this does not include DeleteObject(s). [0] An AWS Batch job could run a container which sends DeleteObjects requests but only when triggered by a job queue which seems redundant here. If I can't use an expiration lifecycle policy because I need a selection of objects not matching a prefix or object tags, I would run something with `s5cmd rm` [1]. Alternatively roll your own golang which parses the CSV and sends many DeleteObjects requests in parallel goroutines. 0. https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-operations.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-... 1. https://github.com/peak/s5cmd#delete-multiple-s3-objects https://github.com/peak/s5cmd#delete-multiple-s3-objects

Hamuko 5y ago

>The wait is often hours until AWS released a bucket name (since bucket names are globally unique, not just within your account). I think last time I did this, the wait time was pretty much exactly 60 minutes.

vrnchndk 5y ago

> Deleting a bucket won't let you re-create that bucket immediately. This is partially incorrect. I can recreate it immediately in the same account, but in different account, I need to wait for ~1 hour