6 ms·
Also a bit unrelated, but at my work place we reimplemented tar algorithm so we can run it in a lambda/state machine. We used the s3 multipart upload to help wi
by nomy99 6y ago
Also a bit unrelated, but at my work place we reimplemented tar algorithm so we can run it in a lambda/state machine. We used the s3 multipart upload to help with the file transfer bit. The performance was phenomenal. On avg we were tar'ing 20gb in 15 seconds or so, a lot faster than on my pc.
- OskarS 6y agoTar (as opposed to gzip) is almost entirely I/O bound, it's just reading from the filesystem, adding some headers and then outputting it again. Whatever speed your I/O runs at, you will basically tar at that speed as well.
- banana_giraffe 6y agoYep. One of the biggest speed improvements I got out of a process that creates large tar balls was moving it to use pbzip2 to allow the compression to use multiple cores.
- gnulinux 6y agoI don't understand this comment. You can run arbitrary code in AWS lambda, you can just run "tar". Why did you need to reimplement it?
- 411111111111111 6y agoIt wasn't always like that though. I think they added custom runtimes around 2yrs ago... Your options were very limited before that.
- thenickdude 6y agoBefore custom runtimes it was still possible to ship your own binaries and shell out to them from the official handler function.
- rescbr 6y agoI suppose they are streaming the tarball directly to S3 using multipart uploads, maybe parallelizing this task at the same time. You wouldn't be able to do it using regular tar.
- nomy99 6y agohit the nail on the head
- thenickdude 6y agoYes, you can, you can pipe tar to the AWS CLI to upload a stream, like "tar -c ./myfiles | aws s3 cp - s3://my-bucket/myobject". For very large uploads you must also specify a chunksize using "multipart_chunksize" such that the upload will require 10,000 chunks or fewer (the CLI can't do this for you because it doesn't know the size of your stream up front).
- nomy99 6y agoWouldn’t you have to have an ec2 open ? We are trying to minimize infrastructure and do it all in lambdas
- thenickdude 6y agoNothing prevents calling the AWS CLI inside a Lambda.