4 ms·
No, object store works better with a small number of large object than with a large number of small object. 1kb object would be terribly inefficient if you hav
by dikei 5y ago
No, object store works better with a small number of large object than with a large number of small object.
1kb object would be terribly inefficient if you have too many of them. It's slow even with regular file system, copy 1 million 1kb files would take forever. For small objects, it's best to pack them into larger block before storage, like how a database does with its data.
- chrislusf 5y agoActually SeaweedFS does just that, append small objects into larger blocks during write or update, and run garbage collection during compaction.
- dikei 5y agoSeaweedFS sounds very interesting indeed, though I'm a bit concern about the complexity. For example, there are so many Filer metadata backend options that it's difficult for me to pick one. I think the document should have a clearer guideline about best practices when deploying a SeaweedFS cluster for different situations: small cluster, large cluster, H/A cluster etc..
- chrislusf 5y agoThis is a fair complaint. :) For filer metadata, you should just pick the one you are most familiar with. There is a wiki page for production setup. https://github.com/chrislusf/seaweedfs/wiki/Production-Setup https://github.com/chrislusf/seaweedfs/wiki/Production-Setup