6 ms·

Here is a good one: deleting billions of objects can be expensive if you call delete APIs. However you can set a wildcard or bucket wide object expiry of time=

Here is a good one: deleting billions of objects can be expensive if you call delete APIs. However you can set a wildcard or bucket wide object expiry of time=now for free. You’ll immediately stop being charged for storage, and AWS will manage making sure everything is deleted.

est31 2y ago
> You’ll immediately stop being charged for storage The effect of lifecycle rules is not immediate: they get applied in a once-per-day batch job, so the removal is not immediate.
- electroly 2y ago
  That's true but OP clearly knows that already. You stop getting charged for storage as soon as the object is marked for expiration, not when the object is finally removed at AWS's leisure. You can check the expiration status of objects in the metadata panel of the S3 web console. https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecy... > There may be a delay between the expiration date and the date at which Amazon S3 removes an object. You are not charged for expiration or the storage time associated with an object that has expired.
ericpauley 2y ago
Nit: the delete call is free, it’s the list call to get the objects that costs money. In theory if you know what objects you have from another source it’s free.
bushbaba 2y ago
Because AWS gets to choose when the actual deletes happen. The metadata is marked as object deleted but AWS can process the delete off peak. It also avoids the s3 api server being hammered with qps
- yencabulator 2y ago
  An explicit delete operation can mark metadata the same way. The real difference is likely more along the lines of LSM compaction, with expiry they likely use a moral equivalent of https://github.com/facebook/rocksdb/wiki/Compaction-Filter https://github.com/facebook/rocksdb/wiki/Compaction-Filter to actually do the deletion.
  - bushbaba 2y ago
    That’s called a delete marker in s3. Less IO than a true delete but also still can lead to high qps to s3 api server. https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/Delete...

est31 2y ago

> You’ll immediately stop being charged for storage The effect of lifecycle rules is not immediate: they get applied in a once-per-day batch job, so the removal is not immediate.

electroly 2y ago
That's true but OP clearly knows that already. You stop getting charged for storage as soon as the object is marked for expiration, not when the object is finally removed at AWS's leisure. You can check the expiration status of objects in the metadata panel of the S3 web console. https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecy... > There may be a delay between the expiration date and the date at which Amazon S3 removes an object. You are not charged for expiration or the storage time associated with an object that has expired.

ericpauley 2y ago

Nit: the delete call is free, it’s the list call to get the objects that costs money. In theory if you know what objects you have from another source it’s free.

bushbaba 2y ago

Because AWS gets to choose when the actual deletes happen. The metadata is marked as object deleted but AWS can process the delete off peak. It also avoids the s3 api server being hammered with qps

yencabulator 2y ago
An explicit delete operation can mark metadata the same way. The real difference is likely more along the lines of LSM compaction, with expiry they likely use a moral equivalent of https://github.com/facebook/rocksdb/wiki/Compaction-Filter https://github.com/facebook/rocksdb/wiki/Compaction-Filter to actually do the deletion.
- bushbaba 2y ago
  That’s called a delete marker in s3. Less IO than a true delete but also still can lead to high qps to s3 api server. https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/Delete...