3 ms·

I think the major problem is latency. Try sshfs (fs over ssh) and see what I mean. Don't get me wrong, I use and like sshfs for a quick data transfer, but it's

I think the major problem is latency. Try sshfs (fs over ssh) and see what I mean. Don't get me wrong, I use and like sshfs for a quick data transfer, but it's just not good enough to run your application with. For a stable POSIX filesystem in production latency is key. Often times in a datacenter 10GE is recommended for network storage solutions, not because of bandwidth (which is also important), but for the 10x reduced latency of a 10GE NIC. Most applications simply expect response times of microseconds or a few milliseconds at most from a POSIX Filesystem, they simply cannot run (without modifying the codebase) on something much slower. But if you had to rewrite your application anyways, then you might want to use plain S3 without the FS, that is easier to operate in the long run.

daviesliu 6y ago
Good, that's why we choose Redis for the metadata. When Redis is deployed in same DC or VPC, the latency could be about 0.2ms - 0.5ms. Later on, We will try the client cache of Redis, which could also reduce the latency for some metadata operations down to a few microseconds.
- deleted 6y ago
  [deleted]
- tutfbhuf 6y ago
  A low latency metadata server is good, but you still have the latency to the S3 server, e.g. Amazon S3 (from your README.md). Now you could say, that I could host my own S3 e.g. MinIO in the same DC, but then I could also simply deploy Ceph, which is battle tested for years up to the petabyte range and with iSCSI, S3 and FS interfaces. So I think this project might give the wrong impression that you can simply combine a Redis with Amazon S3 and then have a good FS solution available, which is unfortunately not the case.
  - daviesliu 6y ago
    In AWS, it's yes. The latency of first byte from S3 is about 20-30ms, close to what you can expect from HDD. Ceph is great, if you can master the complexity under the hood, MinIO + Redis + JuiceFS could be the easier answer for beginners.
    - ozkatz 6y ago
      TTFB in S3 is 20-30ms around the 50th percentile. it can go much higher at p99 [1]. In any case, rotational latency for HDD drives is an order of magnitude lower (typically 2-5ms for a seek operation). S3 is great for higher throughput workloads where TTFB is amortized across larger downloads (this is why it's very common to use S3 as a "data lake" where larger columnar files are stored, usually at the order of hundreds of MiB). I think it's an interesting project but perhaps explaining the use cases where this solution is beneficial would go a long way here. [1] https://github.com/dvassallo/s3-benchmark https://github.com/dvassallo/s3-benchmark
      - daviesliu 6y ago
        Agreed, thanks for putting these number together. JuiceFS was initial designed for big data workload in Cloud, and we have tens of paying customer use it in this use case. For NAS use case, the latency would be slower when case missed, but the overall IOPS could be higher. In the case of overwhelmed HDD, the latency could also go up to hundreds of milliseconds. We have pay lots of effort to improve the caching (in kernel and on disk) and prefetching. The overall performance is comparable to HDD over NFS.
eb0la 6y ago
> Often times in a datacenter 10GE is recommended for network storage solutions, not because of bandwidth (which is also important), but for the 10x reduced latency of a 10GE NIC. Totally true. In those cases you also need your filer _not_ to compress on the fly, and not deduplicate objects... unless your hardware can do it for you. ... I remember horrible performance on VMs with images stored on a oversized netapp filer, because someone enabled deduplication and compression instead of using 1 of the 4 spare drives (when it was HDD).

daviesliu 6y ago

Good, that's why we choose Redis for the metadata. When Redis is deployed in same DC or VPC, the latency could be about 0.2ms - 0.5ms. Later on, We will try the client cache of Redis, which could also reduce the latency for some metadata operations down to a few microseconds.

deleted 6y ago
[deleted]
tutfbhuf 6y ago
A low latency metadata server is good, but you still have the latency to the S3 server, e.g. Amazon S3 (from your README.md). Now you could say, that I could host my own S3 e.g. MinIO in the same DC, but then I could also simply deploy Ceph, which is battle tested for years up to the petabyte range and with iSCSI, S3 and FS interfaces. So I think this project might give the wrong impression that you can simply combine a Redis with Amazon S3 and then have a good FS solution available, which is unfortunately not the case.
- daviesliu 6y ago
  In AWS, it's yes. The latency of first byte from S3 is about 20-30ms, close to what you can expect from HDD. Ceph is great, if you can master the complexity under the hood, MinIO + Redis + JuiceFS could be the easier answer for beginners.
  - ozkatz 6y ago
    TTFB in S3 is 20-30ms around the 50th percentile. it can go much higher at p99 [1]. In any case, rotational latency for HDD drives is an order of magnitude lower (typically 2-5ms for a seek operation). S3 is great for higher throughput workloads where TTFB is amortized across larger downloads (this is why it's very common to use S3 as a "data lake" where larger columnar files are stored, usually at the order of hundreds of MiB). I think it's an interesting project but perhaps explaining the use cases where this solution is beneficial would go a long way here. [1] https://github.com/dvassallo/s3-benchmark https://github.com/dvassallo/s3-benchmark
    - daviesliu 6y ago
      Agreed, thanks for putting these number together. JuiceFS was initial designed for big data workload in Cloud, and we have tens of paying customer use it in this use case. For NAS use case, the latency would be slower when case missed, but the overall IOPS could be higher. In the case of overwhelmed HDD, the latency could also go up to hundreds of milliseconds. We have pay lots of effort to improve the caching (in kernel and on disk) and prefetching. The overall performance is comparable to HDD over NFS.

eb0la 6y ago

> Often times in a datacenter 10GE is recommended for network storage solutions, not because of bandwidth (which is also important), but for the 10x reduced latency of a 10GE NIC. Totally true. In those cases you also need your filer _not_ to compress on the fly, and not deduplicate objects... unless your hardware can do it for you. ... I remember horrible performance on VMs with images stored on a oversized netapp filer, because someone enabled deduplication and compression instead of using 1 of the 4 spare drives (when it was HDD).