4 ms·
Very interesting. And this is an amazing graph for small file uploading/download speed improvement. I have the feeling that all cloud drives are really not opti
by therealmarv 1y ago
Very interesting. And this is an amazing graph for small file uploading/download speed improvement. I have the feeling that all cloud drives are really not optimised for many small files like smaller than 1Mbytes in average.
https://raw.githubusercontent.com/peak/s5cmd/master/doc/benchmark2.png https://raw.githubusercontent.com/peak/s5cmd/master/doc/benc...
I've implemented at work once a rudimentary parallel uploading of many small files to S3 in Python and with boto3 (was not allowed to use a third party library or tool at that time) because it's soooo slow to upload many small files to S3. It really takes ages and even if you just upload 8 small files in parallel it makes a huge difference.
- random_savv 1y agoMost S3 systems are terrible with small files (~50KB). 1MB can still be ok though