I wrote a simple tool for syncing an S3 bucket to local disk: https://github.com/newspaperclub/hank https://github.com/newspaperclub/hank I found the existing

I wrote a simple tool for syncing an S3 bucket to local disk: https://github.com/newspaperclub/hank https://github.com/newspaperclub/hank I found the existing s3 syncing tools used lots of memory when dealing with many small files, so I wrote this in Go, trying to carefully manage memory usage by concurrently listing and downloading the files. We use it for backing up 500GB across 800k files, from an S3 bucket to a ZFS filesystem for snapshotting. It does the job well, usually taking just a few minutes when there are minimal changes.