6 ms·

I’m surprised none of the engineers either knew about, or thought to ask AWS support, S3 Batch Operations or S3 Replication. Both of which could’ve completed t

by kondro 6y ago

I’m surprised none of the engineers either knew about, or thought to ask AWS support, S3 Batch Operations or S3 Replication. Both of which could’ve completed this process in a relatively short amount of time in a completely hands-off way. When I Google for “move large S3 bucket” the first result I get is an AWS article which goes through all the best options: https://aws.amazon.com/premiumsupport/knowledge-center/s3-large-transfer-between-buckets/ https://aws.amazon.com/premiumsupport/knowledge-center/s3-la...

booleanbetrayal 6y ago
If you read the thread, they mention that they had a requirement to empty the original bucket within the maintenance window, as well.
- kondro 6y ago
  Which you could’ve easily done with S3 Batch Operations or the EMR-based sync in a super-parallel way and little engineering effort.
- altdatathrow 6y ago
  Their biggest issue was letting someone who didn't know a thing about S3 dictate the rules. Revoke all access to bucket ~= everything is deleted. Then they could set a lifecycle rule, wait a day, and it would be gone for real.
  - dmlittle 6y ago
    From my experience if you have a bucket with billions of objects it will still take a few days after the objects are deleted by the lifecycle rules to be able to delete the bucket. The objects are non-recoverable but I believe due to the eventual consistency nature of S3 you can't delete the bucket until the objects are all fully deleted.
    - altdatathrow 6y ago
      Yeah, S3 is frankly a disaster. People use it like a filesystem and if you have billions of objects, it very quickly becomes a nightmare to manage. Size of data is a complete non-issue, it's all about the number of objects.
      - grogenaut 6y ago
        I will point out deleting that much data from harddrives isn't a fast operation either if you were storing it on drive. Same for tape. Sure you can destructively destroy the drives. But you could also just use KMS on the S3 bucket object and then toss the key and close the buckets. Gonezo.
        dylan604 6y ago
        except you'd still be on the hook for the monthly cost for the useless bucket full of data
        the8472 6y ago
        > I will point out deleting that much data from harddrives isn't a fast operation either Dropping a btrfs subvolume or snapshot is pretty fast. ditto for zfs.
        danielheath 6y ago
        That doesn't help if you want the data to be gone instead of hidden.
        Dylan16807 6y ago
        If it's encrypted then destroying the key gives you a blank-with-noise drive. There's no equivalent for S3.
        jiggawatts 6y ago
        It's strange that people just shrug their shoulders and accept this state of things in 2020. Any system these days should allow near-instantaneous admin operations for just about any reasonable request. Subtree move, delete, or rename should all be instant. Similarly, any simple ACL changes should be instant. Similarly, data movement should scale linearly with the volume of data, not the number of objects.
  - Dylan16807 6y ago
    > wait a day How does that solve the problem of having the bucket accessible and empty in two hours?
hal9000-tng 6y ago
> I’m surprised none of the engineers either knew about You'll probably also be surprised to learn that the original article not only specified that link but also that they chose the first listed option, which probably wasn't a surprising choice, given that it was listed first.
- kondro 6y ago
  You’re right. I was tired when I first read this last night on Reddit and I could swear the link wasn’t there then.
  - lumost 6y ago
    I'd bet good money that this team was tired, when they read the docs/googled as well.
ed25519FUUU 6y ago
At least when I look into bucket replication (a long time ago, I don't think it was GA at the time), the sync operation only affected _new_ keys, not existing keys, so it wouldn't have worked for this purpose.
- kondro 6y ago
  By default, you'd have to do something to touch the keys. But AWS support is more than happy to fire off a full init for a new replication if you ask them.
idunno246 6y ago
I had to do this before batch, replication, inventory. Support, SA, TAM were all like... good luck, though they did offer to make an inventory but they delivered that two weeks later. Everything at the time said s3distcp was the only option. It honestly wasn’t that bad, except it did a serial list before it started copying which would have taken days so needed some tomfoolery.

by kondro 6y ago

booleanbetrayal 6y ago

If you read the thread, they mention that they had a requirement to empty the original bucket within the maintenance window, as well.

kondro 6y ago
Which you could’ve easily done with S3 Batch Operations or the EMR-based sync in a super-parallel way and little engineering effort.
altdatathrow 6y ago
Their biggest issue was letting someone who didn't know a thing about S3 dictate the rules. Revoke all access to bucket ~= everything is deleted. Then they could set a lifecycle rule, wait a day, and it would be gone for real.
- dmlittle 6y ago
  From my experience if you have a bucket with billions of objects it will still take a few days after the objects are deleted by the lifecycle rules to be able to delete the bucket. The objects are non-recoverable but I believe due to the eventual consistency nature of S3 you can't delete the bucket until the objects are all fully deleted.
  - altdatathrow 6y ago
    Yeah, S3 is frankly a disaster. People use it like a filesystem and if you have billions of objects, it very quickly becomes a nightmare to manage. Size of data is a complete non-issue, it's all about the number of objects.
    - grogenaut 6y ago
      I will point out deleting that much data from harddrives isn't a fast operation either if you were storing it on drive. Same for tape. Sure you can destructively destroy the drives. But you could also just use KMS on the S3 bucket object and then toss the key and close the buckets. Gonezo.
      - dylan604 6y ago
        except you'd still be on the hook for the monthly cost for the useless bucket full of data
      - the8472 6y ago
        > I will point out deleting that much data from harddrives isn't a fast operation either Dropping a btrfs subvolume or snapshot is pretty fast. ditto for zfs.
        danielheath 6y ago
        That doesn't help if you want the data to be gone instead of hidden.
        Dylan16807 6y ago
        If it's encrypted then destroying the key gives you a blank-with-noise drive. There's no equivalent for S3.
      - jiggawatts 6y ago
        It's strange that people just shrug their shoulders and accept this state of things in 2020. Any system these days should allow near-instantaneous admin operations for just about any reasonable request. Subtree move, delete, or rename should all be instant. Similarly, any simple ACL changes should be instant. Similarly, data movement should scale linearly with the volume of data, not the number of objects.
- Dylan16807 6y ago
  > wait a day How does that solve the problem of having the bucket accessible and empty in two hours?

hal9000-tng 6y ago

> I’m surprised none of the engineers either knew about You'll probably also be surprised to learn that the original article not only specified that link but also that they chose the first listed option, which probably wasn't a surprising choice, given that it was listed first.

kondro 6y ago
You’re right. I was tired when I first read this last night on Reddit and I could swear the link wasn’t there then.
- lumost 6y ago
  I'd bet good money that this team was tired, when they read the docs/googled as well.

ed25519FUUU 6y ago

At least when I look into bucket replication (a long time ago, I don't think it was GA at the time), the sync operation only affected _new_ keys, not existing keys, so it wouldn't have worked for this purpose.

kondro 6y ago
By default, you'd have to do something to touch the keys. But AWS support is more than happy to fire off a full init for a new replication if you ask them.

idunno246 6y ago

I had to do this before batch, replication, inventory. Support, SA, TAM were all like... good luck, though they did offer to make an inventory but they delivered that two weeks later. Everything at the time said s3distcp was the only option. It honestly wasn’t that bad, except it did a serial list before it started copying which would have taken days so needed some tomfoolery.