3 ms·
I suppose they are streaming the tarball directly to S3 using multipart uploads, maybe parallelizing this task at the same time. You wouldn't be able to do it
by rescbr 6y ago
I suppose they are streaming the tarball directly to S3 using multipart uploads, maybe parallelizing this task at the same time.
You wouldn't be able to do it using regular tar.
- nomy99 6y agohit the nail on the head
- thenickdude 6y agoYes, you can, you can pipe tar to the AWS CLI to upload a stream, like "tar -c ./myfiles | aws s3 cp - s3://my-bucket/myobject". For very large uploads you must also specify a chunksize using "multipart_chunksize" such that the upload will require 10,000 chunks or fewer (the CLI can't do this for you because it doesn't know the size of your stream up front).
- nomy99 6y agoWouldn’t you have to have an ec2 open ? We are trying to minimize infrastructure and do it all in lambdas
- thenickdude 6y agoNothing prevents calling the AWS CLI inside a Lambda.