6 ms·
Is a 64MB object size really typical of the usecase most people will be using this for? I would imagine 1kb objects to be far more common...
by londons_explore 5y ago
Is a 64MB object size really typical of the usecase most people will be using this for?
I would imagine 1kb objects to be far more common...
- nijave 5y agoThat's common for partioned data. If you have a couple hundred GB table or log you can split it into a bunch of smaller blocks for parallel processing. I think 64MB tends to be a good medium between overhead of tons of tiny files and inability to parallelize/shard with fewer large ones
- mastazi 5y agoI suppose it depends on the type of data? If those are e.g. video files, I don't suppose there are many 1kB files...
- FridgeSeal 5y agoWhat are you storing in an object store that fits into 1kb? 64MB is far, far closer to our usecase than 1kb. I don’t even think our schema files/metadata which are arguably the smallest part fit into 1kb.
- moreati 5y agoPossibly a difference in terminology. Minio is in the same space as S3, Ceph, and HDFS - sometimes called blob store or block store. Each object might be part of a log, video or other large file/stream. I would call an object store in which the objects are mostly < 1 kiB a key value store.
- manquer 5y agoAt 1kb or less I would likely store as blob in traditional RDMS or noSQL store. It is not worth the extra network call at 1 KB size to use object storage.
- dikei 5y agoNo, object store works better with a small number of large object than with a large number of small object. 1kb object would be terribly inefficient if you have too many of them. It's slow even with regular file system, copy 1 million 1kb files would take forever. For small objects, it's best to pack them into larger block before storage, like how a database does with its data.
- chrislusf 5y agoActually SeaweedFS does just that, append small objects into larger blocks during write or update, and run garbage collection during compaction.
- dikei 5y agoSeaweedFS sounds very interesting indeed, though I'm a bit concern about the complexity. For example, there are so many Filer metadata backend options that it's difficult for me to pick one. I think the document should have a clearer guideline about best practices when deploying a SeaweedFS cluster for different situations: small cluster, large cluster, H/A cluster etc..
- chrislusf 5y agoThis is a fair complaint. :) For filer metadata, you should just pick the one you are most familiar with. There is a wiki page for production setup. https://github.com/chrislusf/seaweedfs/wiki/Production-Setup https://github.com/chrislusf/seaweedfs/wiki/Production-Setup
- chrislusf 5y agoAha, I replied above and you are saying the same thing! Small objects, 4K ~ 4MB, are very common in machine learning area(audio, video, text), or surveillance, game assets, etc. SeaweedFS saw a lot of usage in these areas.