4 ms·

Even if you were moving 25TB of data on a server under your physical control, it could be a significant engineering effort. Maybe not 14 person-days but at leas

Even if you were moving 25TB of data on a server under your physical control, it could be a significant engineering effort. Maybe not 14 person-days but at least a few. What's amazing is that some teams assume the same operation will be more performant in the cloud, despite being wrapped in layers of HTTP-based API abstractions.

brundolf 6y ago
If it's within the same cloud, though, why wouldn't there by an internal shortcut so it doesn't have to go fully "out" then "back in"? Heck, why does it have to physically move at all? Why doesn't this effectively come down to a rename within AWS' system, like when you "move" a file within the same hard drive?
- SteveNuts 6y ago
  That's probably possible, but not publicly exposed. Like a lot of the people on the Reddit post said, reach out to support. They have a LOT of data and knobs to turn, in my experience.
  - jiggawatts 6y ago
    Your advice is completely valid, yet it is a little absurd to be in a situation where something as basic as "mv" is a support-only technology, not available to mere mortals spending millions a month. I've seen many similar situations that boil down to "UPDATE SET x = y WHERE z" require support tickets at a minimum, or are flat impossible because even their internal staff don't know how to do it.
  - dmlittle 6y ago
    Not OP but we also had to move a relatively large S3 bucket (significantly larger than 25TB) and unfortunately AWS doesn't have a way to change the ownership of the S3 bucket. My guess is it has to do with how the underlying system stores the objects. We ended up writing a Go program to copy the objects from one bucket to another to help with the parallelization of migration. This was also before AWS had announced S3 Batch operations so I'm not sure how much better it would be to use that today. The deletion of the bucket also took us over a week. Even though we had deleted all objects in the bucket due to the eventual consistency nature of S3 we weren't allowed to delete the bucket until all objects were fully removed from S3. All we could get from AWS support was to wait a few more days and reach back if we couldn't delete it then. Edit: depending on your object naming scheme you might also run into the S3 prefix rate limits.
- kinkrtyavimoodh 6y ago
  If you are moving it across data-centers (Availability Zones) it won't be possible to do it purely 'symbolically'..
  - brundolf 6y ago
    Sure, but you still shouldn't have to manually squeeze it through public HTTP APIs
- idunno246 6y ago
  At least when I did similar, moving the data itself was not the problem - it is entirely s3 server side. The problem is round trip times on the empty-body api calls themselves, aws cli being python is slow and maxes out on https/signatures. If the copy Api supported wildcards there wouldn’t be any discussion at all on this
deleted 6y ago
[deleted]
userbinator 6y ago
What's amazing is that some teams assume the same operation will be more performant in the cloud, despite being wrapped in layers of HTTP-based API abstractions. I have worked with such developers, whose thoughts have been --- for lack of a better word --- clouded by marketing.
manigandham 6y ago
It should be more performant in the cloud. That flexibility and scalability is what you're paying for. There are various tools and strategies for moving this data as explained by several posts here that would've made the process much easier.
Aerroon 6y ago
>Even if you were moving 25TB of data on a server under your physical control, it could be a significant engineering effort. But why? I'm genuinely curious. The other week I had a HDD throw up some errors. I copy pasted some of its contents of around 1-2 TB in Windows onto another drive. Sure, it took a while, but it was all done by morning. It didn't even occur to me that problems could arise with this.
Dylan16807 6y ago
You seem to be conflating person-days and performance? I'll address performance first: Sure, it could be slow if I can't move the actual hard drives and I only have 1Gbps links. But of course I expect the cloud to be faster than "artificially bottlenecked at 1Gbps"! How is that expectation "amazing"? And for engineering effort, something would have to be deeply wrong for a 25TB transfer of one directory of normal files to take a significant amount. This is a problem that you point rsync at, watch for a couple minutes, and go to lunch.

brundolf 6y ago

If it's within the same cloud, though, why wouldn't there by an internal shortcut so it doesn't have to go fully "out" then "back in"? Heck, why does it have to physically move at all? Why doesn't this effectively come down to a rename within AWS' system, like when you "move" a file within the same hard drive?

SteveNuts 6y ago
That's probably possible, but not publicly exposed. Like a lot of the people on the Reddit post said, reach out to support. They have a LOT of data and knobs to turn, in my experience.
- jiggawatts 6y ago
  Your advice is completely valid, yet it is a little absurd to be in a situation where something as basic as "mv" is a support-only technology, not available to mere mortals spending millions a month. I've seen many similar situations that boil down to "UPDATE SET x = y WHERE z" require support tickets at a minimum, or are flat impossible because even their internal staff don't know how to do it.
- dmlittle 6y ago
  Not OP but we also had to move a relatively large S3 bucket (significantly larger than 25TB) and unfortunately AWS doesn't have a way to change the ownership of the S3 bucket. My guess is it has to do with how the underlying system stores the objects. We ended up writing a Go program to copy the objects from one bucket to another to help with the parallelization of migration. This was also before AWS had announced S3 Batch operations so I'm not sure how much better it would be to use that today. The deletion of the bucket also took us over a week. Even though we had deleted all objects in the bucket due to the eventual consistency nature of S3 we weren't allowed to delete the bucket until all objects were fully removed from S3. All we could get from AWS support was to wait a few more days and reach back if we couldn't delete it then. Edit: depending on your object naming scheme you might also run into the S3 prefix rate limits.
kinkrtyavimoodh 6y ago
If you are moving it across data-centers (Availability Zones) it won't be possible to do it purely 'symbolically'..
- brundolf 6y ago
  Sure, but you still shouldn't have to manually squeeze it through public HTTP APIs
idunno246 6y ago
At least when I did similar, moving the data itself was not the problem - it is entirely s3 server side. The problem is round trip times on the empty-body api calls themselves, aws cli being python is slow and maxes out on https/signatures. If the copy Api supported wildcards there wouldn’t be any discussion at all on this

deleted 6y ago

[deleted]

userbinator 6y ago

What's amazing is that some teams assume the same operation will be more performant in the cloud, despite being wrapped in layers of HTTP-based API abstractions. I have worked with such developers, whose thoughts have been --- for lack of a better word --- clouded by marketing.

manigandham 6y ago

It should be more performant in the cloud. That flexibility and scalability is what you're paying for. There are various tools and strategies for moving this data as explained by several posts here that would've made the process much easier.

Aerroon 6y ago

>Even if you were moving 25TB of data on a server under your physical control, it could be a significant engineering effort. But why? I'm genuinely curious. The other week I had a HDD throw up some errors. I copy pasted some of its contents of around 1-2 TB in Windows onto another drive. Sure, it took a while, but it was all done by morning. It didn't even occur to me that problems could arise with this.

Dylan16807 6y ago

You seem to be conflating person-days and performance? I'll address performance first: Sure, it could be slow if I can't move the actual hard drives and I only have 1Gbps links. But of course I expect the cloud to be faster than "artificially bottlenecked at 1Gbps"! How is that expectation "amazing"? And for engineering effort, something would have to be deeply wrong for a 25TB transfer of one directory of normal files to take a significant amount. This is a problem that you point rsync at, watch for a couple minutes, and go to lunch.