3 ms·

For s3 and local filesystem, couldn't you also install aws-cli and do a sync b/w the 2 ? I do that for my stuff. Any reason you wrote this and not use aws-cli ?

cobbzilla 10y ago
from the readme: "I started with "s3cmd sync" but found that with buckets containing many thousands of objects, it was incredibly slow to start and consumed massive amounts of memory. So I designed s3s3mirror to start copying immediately with an intelligently chosen "chunk size" and to operate in a highly-threaded, streaming fashion, so memory requirements are much lower. Running with 100 threads, I found the gating factor to be how fast I could list items from the source bucket (!?!) Which makes me wonder if there is any way to do this faster. I'm sure there must be, but this is pretty damn fast."
- diggs 10y ago
  This approach works well enough for relatively small amounts of objects. Once you start getting in to the millions (and significantly higher) then it begins to break down. Every "sync" operation has to start from scratch, comparing source and target (possibly through an index) on a file by file basis. There are definitely faster ways of doing it that scale to much larger object counts, but then they have their own drawbacks. It's a shame the S3 Api doesn't let you order by modified date, or this would be trivial to do efficiently.
  - cobbzilla 10y ago
    I'm curious if you can share how to synchronize N files without doing at least N comparisons. the main innovations in s3s3mirror are (1) understanding this & going for massive parallelism to speed things up and (2) where possible, comparing etag/metadata instead of all bytes. so far, it has scaled pretty well, i know of no faster tool to synchronize buckets with millions of objects.
    - diggs 10y ago
      Sorry, I should have perhaps put a disclaimer in my original comment. I work for a company called StorReduce and built our replication feature* (an intelligent, continuous "sync" effectively). We currently have a patent pending for our method, so I'm not sure if I can offer any real insight unfortunately. I haven't looked at your project, but based on what you've said I agree the way you're doing it is conceptually as fast as it can be (massively parallel and leveraging metadata) whilst being a general purpose tool that "just works" and has no external dependencies or constraints. * http://storreduce.com/blog/replication/ http://storreduce.com/blog/replication/
teej 10y ago
AWS-cli only really got robust S3 support in the last 18 months or so. For a long time it couldn't handle multipart uploads and didn't support a bunch of corner cases (versioning, glacier, cross-account, etc)

cobbzilla 10y ago

from the readme: "I started with "s3cmd sync" but found that with buckets containing many thousands of objects, it was incredibly slow to start and consumed massive amounts of memory. So I designed s3s3mirror to start copying immediately with an intelligently chosen "chunk size" and to operate in a highly-threaded, streaming fashion, so memory requirements are much lower. Running with 100 threads, I found the gating factor to be how fast I could list items from the source bucket (!?!) Which makes me wonder if there is any way to do this faster. I'm sure there must be, but this is pretty damn fast."

diggs 10y ago
This approach works well enough for relatively small amounts of objects. Once you start getting in to the millions (and significantly higher) then it begins to break down. Every "sync" operation has to start from scratch, comparing source and target (possibly through an index) on a file by file basis. There are definitely faster ways of doing it that scale to much larger object counts, but then they have their own drawbacks. It's a shame the S3 Api doesn't let you order by modified date, or this would be trivial to do efficiently.
- cobbzilla 10y ago
  I'm curious if you can share how to synchronize N files without doing at least N comparisons. the main innovations in s3s3mirror are (1) understanding this & going for massive parallelism to speed things up and (2) where possible, comparing etag/metadata instead of all bytes. so far, it has scaled pretty well, i know of no faster tool to synchronize buckets with millions of objects.
  - diggs 10y ago
    Sorry, I should have perhaps put a disclaimer in my original comment. I work for a company called StorReduce and built our replication feature* (an intelligent, continuous "sync" effectively). We currently have a patent pending for our method, so I'm not sure if I can offer any real insight unfortunately. I haven't looked at your project, but based on what you've said I agree the way you're doing it is conceptually as fast as it can be (massively parallel and leveraging metadata) whilst being a general purpose tool that "just works" and has no external dependencies or constraints. * http://storreduce.com/blog/replication/ http://storreduce.com/blog/replication/

teej 10y ago

AWS-cli only really got robust S3 support in the last 18 months or so. For a long time it couldn't handle multipart uploads and didn't support a bunch of corner cases (versioning, glacier, cross-account, etc)