6 ms·

I just want to note that the blog post you linked to was never actually published since we decided against pointing fingers when – by and large – the problems w

I just want to note that the blog post you linked to was never actually published since we decided against pointing fingers when – by and large – the problems weren't their fault and their engineers were quite helpful. See the comment at the bottom of that thread[1] from Sid. We made mistakes ourselves that were the root of the problems at the time, and based on the update[2] a few months later things definitely improved significantly. [1]: https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/1097#note_2758069 https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/... [2]: https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/1097#note_3738731 https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/...

boulos 10y ago
Sorry about that! I was on my phone, and followed the first thing from the linked post/issue; I didn't realize it wasn't published.
- connorshea 10y ago
  No problem, the thread is pretty long, and it's not necessarily easy to tell without understanding how our blog post process works. :)
user5994461 10y ago
Judging by the posts. Random thought: Go talk to google and get $100k of free cloud credits. You run 100 instances of n1-highcpu-2 (2 CPU, 2GB RAM), with a local 375 SSD. That gives you 37TB of storage and you're fine for the year. This is an absolutely ridiculous setup but it could work. 1) Servers don't need to have more memory for caching. Local SSD access is fast enough. 2) Servers have 1Gb uplink. That can be saturated easily by a single local SSD. 3) I'm a firm believer that if you put 8-16 disks in a single box, you're gonna hit a hard limit with the network bandwidth and infrastructure (among other things). 4) Do you need IOPS or do you need storage??? 5) Can CephFS shard fairly across a hundred systems? 6) Should CephFS run on raw disks, manage replica itself? and can it handle failure? Bonus question) Why are we thinking about all of this. You should have just used s3?
- geofft 10y ago
  > Bonus question) Why are we thinking about all of this. You should have just used s3? NFS and Ceph both support the standard POSIX file access model, notably including random-access reads and writes on a file without replacing the entire file and consistency models that let you reason about directories and their contents usefully. If your goal is to run the reference implementation of git, or something that is compatible with it, on the filesystem, you absolutely need something that supports this. You could write something new that speaks the git protocol but is designed to be backed by an eventually-consistent object store instead of a POSIX-compliant file store ... but that seems like an equally big challenge, honestly.
  - kyrra 10y ago
    Fun aside. Google wrote a backend to Mercurial so it could be stored on bigtable. They did a talk about it 7 years ago. http://m.youtube.com/watch?v=ri796Hx8las http://m.youtube.com/watch?v=ri796Hx8las
    - geofft 10y ago
      And to SVN! https://lwn.net/Articles/194667/ https://lwn.net/Articles/194667/ It's not a bad decision; getting POSIX semantics to play well with modern expectations of a highly-consistent and highly-available distributed system is very difficult. But it's a work-intensive decision, and figuring out how to deploy NFS or Ceph at scale might be less work. Especially if Google's SVN/hg developers can work directly with the Bigtable developers on features, and GitLab's developers can't work directly with the S3 team (but can own their own Ceph or NFS stack).
      - zaphar 10y ago
        Their Git hosting was also running on that same backend (I was on the team managing those products). But to a previous posters point That is a lot of work both to implement as well as to maintain. If you can just use a distributed filesystem you might save yourself some trouble.
  - 10y ago
objectivefs 10y ago
An alternative to user5994461's suggestion would be to switch from CephFS to a different distributed filesystem. If you need POSIX and can't go directly to S3 you could try ObjectiveFS[1]. Using local SSD instance store and memory for caching you can get very good performance even for small file workloads[2]. [1]: https://objectivefs.com https://objectivefs.com [2]: https://objectivefs.com/howto/performance-amazon-efs-vs-objectivefs https://objectivefs.com/howto/performance-amazon-efs-vs-obje...
- sytse 10y ago
  ObjectiveFS looks interesting but I wonder about the latency if you do many small random reads (this is what we're seeing with git).
  - objectivefs 10y ago
    Latency for small random reads (16 parallel threads), using the Linux kernel source tree, when hitting in the SSD instance store disk cache have a 95th-percentile latency of 6ms with ~50MB/s throughput. When hitting in the memory cache the 95th-percentile latency is <1ms with ~380MB/s throughput. We have more performance data available at https://objectivefs.com/howto/performance-amazon-efs-vs-objectivefs https://objectivefs.com/howto/performance-amazon-efs-vs-obje...
    - sytse 10y ago
      That sounds very good, but I think we prefer Ceph because it is better known. Do you have a comparison between Ceph and ObjectiveFS?
- jamesblonde 10y ago
  Another option would be to move to HDFS. HopsFS, a new distribution, can scale to over 1m ops/sec: http://www.logicalclocks.com/index.php/2016/10/14/hops-smashes-hadoop-record-over-1-million-operations-per-second/ http://www.logicalclocks.com/index.php/2016/10/14/hops-smash...

boulos 10y ago

Sorry about that! I was on my phone, and followed the first thing from the linked post/issue; I didn't realize it wasn't published.

connorshea 10y ago
No problem, the thread is pretty long, and it's not necessarily easy to tell without understanding how our blog post process works. :)

user5994461 10y ago

Judging by the posts. Random thought: Go talk to google and get $100k of free cloud credits. You run 100 instances of n1-highcpu-2 (2 CPU, 2GB RAM), with a local 375 SSD. That gives you 37TB of storage and you're fine for the year. This is an absolutely ridiculous setup but it could work. 1) Servers don't need to have more memory for caching. Local SSD access is fast enough. 2) Servers have 1Gb uplink. That can be saturated easily by a single local SSD. 3) I'm a firm believer that if you put 8-16 disks in a single box, you're gonna hit a hard limit with the network bandwidth and infrastructure (among other things). 4) Do you need IOPS or do you need storage??? 5) Can CephFS shard fairly across a hundred systems? 6) Should CephFS run on raw disks, manage replica itself? and can it handle failure? Bonus question) Why are we thinking about all of this. You should have just used s3?

geofft 10y ago
> Bonus question) Why are we thinking about all of this. You should have just used s3? NFS and Ceph both support the standard POSIX file access model, notably including random-access reads and writes on a file without replacing the entire file and consistency models that let you reason about directories and their contents usefully. If your goal is to run the reference implementation of git, or something that is compatible with it, on the filesystem, you absolutely need something that supports this. You could write something new that speaks the git protocol but is designed to be backed by an eventually-consistent object store instead of a POSIX-compliant file store ... but that seems like an equally big challenge, honestly.
- kyrra 10y ago
  Fun aside. Google wrote a backend to Mercurial so it could be stored on bigtable. They did a talk about it 7 years ago. http://m.youtube.com/watch?v=ri796Hx8las http://m.youtube.com/watch?v=ri796Hx8las
  - geofft 10y ago
    And to SVN! https://lwn.net/Articles/194667/ https://lwn.net/Articles/194667/ It's not a bad decision; getting POSIX semantics to play well with modern expectations of a highly-consistent and highly-available distributed system is very difficult. But it's a work-intensive decision, and figuring out how to deploy NFS or Ceph at scale might be less work. Especially if Google's SVN/hg developers can work directly with the Bigtable developers on features, and GitLab's developers can't work directly with the S3 team (but can own their own Ceph or NFS stack).
    - zaphar 10y ago
      Their Git hosting was also running on that same backend (I was on the team managing those products). But to a previous posters point That is a lot of work both to implement as well as to maintain. If you can just use a distributed filesystem you might save yourself some trouble.
- 10y ago

objectivefs 10y ago

An alternative to user5994461's suggestion would be to switch from CephFS to a different distributed filesystem. If you need POSIX and can't go directly to S3 you could try ObjectiveFS[1]. Using local SSD instance store and memory for caching you can get very good performance even for small file workloads[2]. [1]: https://objectivefs.com https://objectivefs.com [2]: https://objectivefs.com/howto/performance-amazon-efs-vs-objectivefs https://objectivefs.com/howto/performance-amazon-efs-vs-obje...

sytse 10y ago
ObjectiveFS looks interesting but I wonder about the latency if you do many small random reads (this is what we're seeing with git).
- objectivefs 10y ago
  Latency for small random reads (16 parallel threads), using the Linux kernel source tree, when hitting in the SSD instance store disk cache have a 95th-percentile latency of 6ms with ~50MB/s throughput. When hitting in the memory cache the 95th-percentile latency is <1ms with ~380MB/s throughput. We have more performance data available at https://objectivefs.com/howto/performance-amazon-efs-vs-objectivefs https://objectivefs.com/howto/performance-amazon-efs-vs-obje...
  - sytse 10y ago
    That sounds very good, but I think we prefer Ceph because it is better known. Do you have a comparison between Ceph and ObjectiveFS?
jamesblonde 10y ago
Another option would be to move to HDFS. HopsFS, a new distribution, can scale to over 1m ops/sec: http://www.logicalclocks.com/index.php/2016/10/14/hops-smashes-hadoop-record-over-1-million-operations-per-second/ http://www.logicalclocks.com/index.php/2016/10/14/hops-smash...