4 ms·
I did try EMR + s3distcp a while ago. AWS Support recommended it and provided instructions in their docs. It was not very fast for me. The data I was trying to
by aluminussoma 10y ago
I did try EMR + s3distcp a while ago. AWS Support recommended it and provided instructions in their docs. It was not very fast for me. The data I was trying to copy was about 400 GB of small 5-20 kb files - not ideal for fast copying. I gave up and used a different method.
I suspect Evernote, if using S3, would have a similar data profile of many small objects.
- yeukhon 10y agoDo you think you can share with the public this different method? If not, I completely understand.
- aluminussoma 10y agoNo secret - I used s3s3mirror. Multithreaded copying.