3 ms·

Reading/writing SMALL files are SUPER slow on things like S3, Google Drive, Backblaze. Also using a lot of threads does only help a little bit but it's nowhere

Reading/writing SMALL files are SUPER slow on things like S3, Google Drive, Backblaze. Also using a lot of threads does only help a little bit but it's nowhere near reading/writing speeds of e.g. a single 600MB file. Is HopsFS helping in this area?

Galanwe 6y ago
> it's nowhere near reading/writing speeds of e.g. a single 600MB file. I have to disagree here. If you look at benchmarks on internet, yes, it will look like S3 is dead slow. But that is a client problem, not an S3 problem. For instance, boto3 (s3transfer) is an awful implementation that was so overengineered with a reimplementation of futures, task stealing, etc, that the download throughput is pathetic. Most often it will make you top below 200MB/s. But S3 itself scales very well if you know how to use it, and skip boto3. From my experience and benchmarks, each S3 connection will deliver up to 80MB/s, and with range requests you can easily have multiple parallel blocks downloaded. I wrote a simple library that does this called s3pd (https://github.com/NewbiZ/s3pd https://github.com/NewbiZ/s3pd). It's not perfect and is process based instead of coroutines, but that will give you an idea. For reference, using s3pd I can saturate any EC2 network interface that I found (tested up to 10Gb/s interfaces, with download speed >1GB/s). Boto is really doing bad press to S3.
deathanatos 6y ago
> Reading/writing SMALL files are SUPER slow on things like S3 My experience was the opposite. Small files work acceptably quickly, the cost can't be beat, and the reliability probably beats out anything else I've seen. We saw latencies of ~50ms to write out small files. Slower than an SSD, yes, but not really that slow. You do have to make sure you're not doing things like re-establishing the connection to get that, though. If you have to do a TLS handshake… it won't be fast. Also, in you're in Python, boto's API encourages, IMO, people to call `get_bucket`, which will do an existence check on the bucket (and thus, double the latency due to the extra round-trip); usually, you have out-of-band knowledge that the bucket will exist, and can skip the check. (If you're wrong, the subsequent object GET/PUT will error anyways.)
ignoramous 6y ago
How small are the files? Here are some strategies to make S3 go faster: https://news.ycombinator.com/item?id=19475726 https://news.ycombinator.com/item?id=19475726 -- You're right that S3 isn't for small files but for a lot of small files (think 500 bytes), I either plunge them to S3 through Kinesis Firehose or fit them gzipped into DynamoDB (depending on access patterns). One could also consider using Amazon FSx which can IO to S3 natively.
Matthias247 6y ago
It's not surprising that it's slow. Handling files has 2 costs associated to it: One is a fixed setup cost, which includes looking up the file metadata, creating connections between all the services that make it possible to access the file, and starting the read/write. The other part is actually transferring the file content. For small files the fixed cost will be the most important factor. The "transfer time" after this "first byte latency" might actually be 0, since all the data could be transferred within a single write call on each node.
chordysson 6y ago
Regarding small files, HopsFS can store small files in the metadata layer for improved performance. https://kth.diva-portal.org/smash/record.jsf?pid=diva2:1260838 https://kth.diva-portal.org/smash/record.jsf?pid=diva2:12608...
- jamesblonde 6y ago
  Yes, this paper did not compare small file performance in HopsFS with EMRFS/S3. But HopsFS on S3 also supports storing small files on NVMe disks in the metadata layer. Previously, we have shown that we can get up to 66X higher throughput for writes for small files versus HDFS (peer reviewed, of course) - https://www.logicalclocks.com/blog/millions-and-millions-of-files-for-deep-learning-with-hopsfs https://www.logicalclocks.com/blog/millions-and-millions-of-...

Galanwe 6y ago

> it's nowhere near reading/writing speeds of e.g. a single 600MB file. I have to disagree here. If you look at benchmarks on internet, yes, it will look like S3 is dead slow. But that is a client problem, not an S3 problem. For instance, boto3 (s3transfer) is an awful implementation that was so overengineered with a reimplementation of futures, task stealing, etc, that the download throughput is pathetic. Most often it will make you top below 200MB/s. But S3 itself scales very well if you know how to use it, and skip boto3. From my experience and benchmarks, each S3 connection will deliver up to 80MB/s, and with range requests you can easily have multiple parallel blocks downloaded. I wrote a simple library that does this called s3pd (https://github.com/NewbiZ/s3pd https://github.com/NewbiZ/s3pd). It's not perfect and is process based instead of coroutines, but that will give you an idea. For reference, using s3pd I can saturate any EC2 network interface that I found (tested up to 10Gb/s interfaces, with download speed >1GB/s). Boto is really doing bad press to S3.

deathanatos 6y ago

> Reading/writing SMALL files are SUPER slow on things like S3 My experience was the opposite. Small files work acceptably quickly, the cost can't be beat, and the reliability probably beats out anything else I've seen. We saw latencies of ~50ms to write out small files. Slower than an SSD, yes, but not really that slow. You do have to make sure you're not doing things like re-establishing the connection to get that, though. If you have to do a TLS handshake… it won't be fast. Also, in you're in Python, boto's API encourages, IMO, people to call `get_bucket`, which will do an existence check on the bucket (and thus, double the latency due to the extra round-trip); usually, you have out-of-band knowledge that the bucket will exist, and can skip the check. (If you're wrong, the subsequent object GET/PUT will error anyways.)

ignoramous 6y ago

How small are the files? Here are some strategies to make S3 go faster: https://news.ycombinator.com/item?id=19475726 https://news.ycombinator.com/item?id=19475726 -- You're right that S3 isn't for small files but for a lot of small files (think 500 bytes), I either plunge them to S3 through Kinesis Firehose or fit them gzipped into DynamoDB (depending on access patterns). One could also consider using Amazon FSx which can IO to S3 natively.

Matthias247 6y ago

It's not surprising that it's slow. Handling files has 2 costs associated to it: One is a fixed setup cost, which includes looking up the file metadata, creating connections between all the services that make it possible to access the file, and starting the read/write. The other part is actually transferring the file content. For small files the fixed cost will be the most important factor. The "transfer time" after this "first byte latency" might actually be 0, since all the data could be transferred within a single write call on each node.

chordysson 6y ago

Regarding small files, HopsFS can store small files in the metadata layer for improved performance. https://kth.diva-portal.org/smash/record.jsf?pid=diva2:1260838 https://kth.diva-portal.org/smash/record.jsf?pid=diva2:12608...

jamesblonde 6y ago
Yes, this paper did not compare small file performance in HopsFS with EMRFS/S3. But HopsFS on S3 also supports storing small files on NVMe disks in the metadata layer. Previously, we have shown that we can get up to 66X higher throughput for writes for small files versus HDFS (peer reviewed, of course) - https://www.logicalclocks.com/blog/millions-and-millions-of-files-for-deep-learning-with-hopsfs https://www.logicalclocks.com/blog/millions-and-millions-of-...