4 ms·

Nothing against the author, but it sounds like another story that Google PR pitched and influenced. It is normal these days. I am sure AWS does the same. That i

Nothing against the author, but it sounds like another story that Google PR pitched and influenced. It is normal these days. I am sure AWS does the same. That is why the technical details may be light. The submitter is likely a Social Media Manager for Google (nothing wrong with that - kudos for not obscuring it!) I, too, have wondered about the technical details when I see stories like this. If the data was originally stored in S3, they probably used the Cloud Storage Transfer Service (https://cloud.google.com/storage/transfer/ https://cloud.google.com/storage/transfer/). The secret sauce appears to be parallelization of the transfers. When I copied hundreds of GB of data from one bucket of to another AWS bucket, it took me days to do! Then I realized it was because I was running a single threaded process. The slowest part was getting the list of objects to copy. If I had a separate, faster index of the objects, then I could have done a copy in parallel much more quickly and max out the bandwidth available to me. (Maybe that is part of the rationale of Netflix's S3mper?). I'm curious to hear other thoughts on accomplishing fast transfers of S3 data. Edit: theoretically speaking, if you had 1 TB of data to transfer, and a paltry 100 Mbps (12.5 MB/s) connection, you could do this in under 24 hours if the entire network was fully utilized. Not a network engineer so go easy on me if I am wrong! As for costs, sending 1 TB out will cost $90,000 in S3 transfer costs, before any other discounts.

yeukhon 10y ago
The most straightforward is running aws s3 command but as you noted it's single threaded, slow as hell. Imagine you have 100TB of data to migrate from one bucket to another. If the command encounter an error (which is very likely to happen), you have to start again. I think the only other option left is use Spot Fleet EMR. I believe the S3 API takes a marker for pagination so you can probably keep paginate to list the next N keys (of course assume your current bucket is no longer accepting new objects... otherwise create a Lambda function and let new file to be copied to the new bucket on create event). I don't remember if aws s3 cp --recursive bucket1 bucket2 requires local copies first, or it is a remote copy-only operation. I have always wanted AWS to allow S3 bucket to be transferred from one account to another with simply transferring ownership.
- aluminussoma 10y ago
  I did try EMR + s3distcp a while ago. AWS Support recommended it and provided instructions in their docs. It was not very fast for me. The data I was trying to copy was about 400 GB of small 5-20 kb files - not ideal for fast copying. I gave up and used a different method. I suspect Evernote, if using S3, would have a similar data profile of many small objects.
  - yeukhon 10y ago
    Do you think you can share with the public this different method? If not, I completely understand.
    - aluminussoma 10y ago
      No secret - I used s3s3mirror. Multithreaded copying.
belril 10y ago
Hi! Author here. (Nice to meet you.) I can tell you that Evernote, not Google, pitched this story to me. While the Google Cloud PR team was aware of its existence, they did not contact or attempt to influence me when it came to its content. (Literally, all that they did was confirm the on-site presence of their engineers, and that an engagement of that sort wasn't a one-off favor to Evernote.) Unfortunately, a technical deep dive on the exact parameters of the transition wasn't really in scope for this piece. I tried to include detail where I could, and Evernote has a really wonderful 5-part blog series that goes into many of the details of the transition. (It's now linked in this post.) To provide some additional details on the transfer: Evernote was going from data centers in California that it owned and operated to Google, not from S3. Technically speaking, the migration isn't 100% complete, as Evernote is still finishing the transfer of its attachment files to Google. Evernote did not disclose the exact details of how that transfer was executed on the record, and thus it did not end up in my piece. Thanks for reading!
- aluminussoma 10y ago
  Thank you for your response and adding clarity around how the story came to be. You must forgive us if we are inherently suspicious.
  - belril 10y ago
    Happy to add clarity where I can -- it is my job, after all. Suspicion and skepticism are healthy, and I totally hear the criticism that the piece feels marketing copy-esque. For what it's worth, the Evernote team seems genuinely happy with the way the transition went, along with their interactions with Google.
ranman 10y ago
>As for costs, sending 1 TB out will cost $90,000 in S3 transfer costs, before any other discounts. 1TB out would be closer to $0.09 * 1024 == $92.16 (still expensive) -- think you may have misplaced a decimal there.

yeukhon 10y ago

The most straightforward is running aws s3 command but as you noted it's single threaded, slow as hell. Imagine you have 100TB of data to migrate from one bucket to another. If the command encounter an error (which is very likely to happen), you have to start again. I think the only other option left is use Spot Fleet EMR. I believe the S3 API takes a marker for pagination so you can probably keep paginate to list the next N keys (of course assume your current bucket is no longer accepting new objects... otherwise create a Lambda function and let new file to be copied to the new bucket on create event). I don't remember if aws s3 cp --recursive bucket1 bucket2 requires local copies first, or it is a remote copy-only operation. I have always wanted AWS to allow S3 bucket to be transferred from one account to another with simply transferring ownership.

aluminussoma 10y ago
I did try EMR + s3distcp a while ago. AWS Support recommended it and provided instructions in their docs. It was not very fast for me. The data I was trying to copy was about 400 GB of small 5-20 kb files - not ideal for fast copying. I gave up and used a different method. I suspect Evernote, if using S3, would have a similar data profile of many small objects.
- yeukhon 10y ago
  Do you think you can share with the public this different method? If not, I completely understand.
  - aluminussoma 10y ago
    No secret - I used s3s3mirror. Multithreaded copying.

belril 10y ago

Hi! Author here. (Nice to meet you.) I can tell you that Evernote, not Google, pitched this story to me. While the Google Cloud PR team was aware of its existence, they did not contact or attempt to influence me when it came to its content. (Literally, all that they did was confirm the on-site presence of their engineers, and that an engagement of that sort wasn't a one-off favor to Evernote.) Unfortunately, a technical deep dive on the exact parameters of the transition wasn't really in scope for this piece. I tried to include detail where I could, and Evernote has a really wonderful 5-part blog series that goes into many of the details of the transition. (It's now linked in this post.) To provide some additional details on the transfer: Evernote was going from data centers in California that it owned and operated to Google, not from S3. Technically speaking, the migration isn't 100% complete, as Evernote is still finishing the transfer of its attachment files to Google. Evernote did not disclose the exact details of how that transfer was executed on the record, and thus it did not end up in my piece. Thanks for reading!

aluminussoma 10y ago
Thank you for your response and adding clarity around how the story came to be. You must forgive us if we are inherently suspicious.
- belril 10y ago
  Happy to add clarity where I can -- it is my job, after all. Suspicion and skepticism are healthy, and I totally hear the criticism that the piece feels marketing copy-esque. For what it's worth, the Evernote team seems genuinely happy with the way the transition went, along with their interactions with Google.

ranman 10y ago

>As for costs, sending 1 TB out will cost $90,000 in S3 transfer costs, before any other discounts. 1TB out would be closer to $0.09 * 1024 == $92.16 (still expensive) -- think you may have misplaced a decimal there.