3 ms·

An alternative to user5994461's suggestion would be to switch from CephFS to a different distributed filesystem. If you need POSIX and can't go directly to S3 y

An alternative to user5994461's suggestion would be to switch from CephFS to a different distributed filesystem. If you need POSIX and can't go directly to S3 you could try ObjectiveFS[1]. Using local SSD instance store and memory for caching you can get very good performance even for small file workloads[2]. [1]: https://objectivefs.com https://objectivefs.com [2]: https://objectivefs.com/howto/performance-amazon-efs-vs-objectivefs https://objectivefs.com/howto/performance-amazon-efs-vs-obje...

sytse 10y ago
ObjectiveFS looks interesting but I wonder about the latency if you do many small random reads (this is what we're seeing with git).
- objectivefs 10y ago
  Latency for small random reads (16 parallel threads), using the Linux kernel source tree, when hitting in the SSD instance store disk cache have a 95th-percentile latency of 6ms with ~50MB/s throughput. When hitting in the memory cache the 95th-percentile latency is <1ms with ~380MB/s throughput. We have more performance data available at https://objectivefs.com/howto/performance-amazon-efs-vs-objectivefs https://objectivefs.com/howto/performance-amazon-efs-vs-obje...
  - sytse 10y ago
    That sounds very good, but I think we prefer Ceph because it is better known. Do you have a comparison between Ceph and ObjectiveFS?
jamesblonde 10y ago
Another option would be to move to HDFS. HopsFS, a new distribution, can scale to over 1m ops/sec: http://www.logicalclocks.com/index.php/2016/10/14/hops-smashes-hadoop-record-over-1-million-operations-per-second/ http://www.logicalclocks.com/index.php/2016/10/14/hops-smash...
- p_l 10y ago
  As someone who has run Hdfs in production on bare metal, I wonder how much you dislik gitlab... TL;DR - HDFS is rarely the answer, mainly due to pains usually involving crashed nameservers.
  - jamesblonde 10y ago
    HopsFS has multiple redundant, stateless NameNodes.

sytse 10y ago

ObjectiveFS looks interesting but I wonder about the latency if you do many small random reads (this is what we're seeing with git).

objectivefs 10y ago
Latency for small random reads (16 parallel threads), using the Linux kernel source tree, when hitting in the SSD instance store disk cache have a 95th-percentile latency of 6ms with ~50MB/s throughput. When hitting in the memory cache the 95th-percentile latency is <1ms with ~380MB/s throughput. We have more performance data available at https://objectivefs.com/howto/performance-amazon-efs-vs-objectivefs https://objectivefs.com/howto/performance-amazon-efs-vs-obje...
- sytse 10y ago
  That sounds very good, but I think we prefer Ceph because it is better known. Do you have a comparison between Ceph and ObjectiveFS?

jamesblonde 10y ago

Another option would be to move to HDFS. HopsFS, a new distribution, can scale to over 1m ops/sec: http://www.logicalclocks.com/index.php/2016/10/14/hops-smashes-hadoop-record-over-1-million-operations-per-second/ http://www.logicalclocks.com/index.php/2016/10/14/hops-smash...

p_l 10y ago
As someone who has run Hdfs in production on bare metal, I wonder how much you dislik gitlab... TL;DR - HDFS is rarely the answer, mainly due to pains usually involving crashed nameservers.
- jamesblonde 10y ago
  HopsFS has multiple redundant, stateless NameNodes.