I do something similar. I have a database of all my files on S3 and their SHA-256 hashes. This not only means that I can skip uploading files which haven't ch

I do something similar. I have a database of all my files on S3 and their SHA-256 hashes. This not only means that I can skip uploading files which haven't changed, but I can do S3->S3 copies for files which are the same. But it's still too often that I have 10s or 100s of MB of data to upload. At 1 Mbit/s, it takes a half hour for every 100 MB!