4 ms·

Why do they need one gigantic distributed fs? Seems like a design miss to me.

btgeekboy 10y ago
Indeed. If there's one thing I've learned in >10 years of building large, multi-tenant systems, it's that you need the ability to partition as you grow. Partitioning eases growth, reduces blast radius, and limits complexity.
- Rapzid 10y ago
  I believe GitHub is or was distributing repos instead of files. AWS code commit likely stores pack files in S3 or otherwise takes advantage of existing AWS systems.. Using a single Ceph FS is an odd choice IMO. You need to have or acquire a ton of expertise to run stuff like that, and it can be finicky. I'm not convinced you can't get this running well enough on AWS though.. So I'd be worried the move to bare metal would just cause the team more problems.
radicalbyte 10y ago
It might be, but you have to look at what's important for a product like GitLab. It's in a market where those who'll pay want to run their own special version of the system. So it's naturally partitioned. Architecting, or even spending mental cycles on day 1 on distribution isn't going to win you as much as focusing on making an awesome product. This move will probably buy them another year or two, which will give them enough time hopefully implement some form of partitioning.
jondubois 10y ago
Agreed, I think the wrong conclusion was drawn here. But I get where they're coming from; container orchestrators like Kubernetes are heavily promoting distributed file systems as being the 'cloud-native approach'. But maybe this issue is more relevant to 'CephFS' specifically than to all distributed file systems in general.
elygre 10y ago
GitLab supports using multiple filesystems, and while waiting for their bare-metal solution, they switch to running 8 file systems: https://gitlab.com/gitlab-com/infrastructure/issues/711 https://gitlab.com/gitlab-com/infrastructure/issues/711
- merb 10y ago
  multiple filesystem != multiple storage backends unfortunatly.
  - elygre 10y ago
    Yes, multiple storage backends: 8 file systems, on 8 independent servers, each with 16tb disk. See https://gitlab.com/gitlab-com/infrastructure/issues/727#note_18193225 https://gitlab.com/gitlab-com/infrastructure/issues/727#note....

Indeed. If there's one thing I've learned in >10 years of building large, multi-tenant systems, it's that you need the ability to partition as you grow. Partitioning eases growth, reduces blast radius, and limits complexity.

Rapzid 10y ago
I believe GitHub is or was distributing repos instead of files. AWS code commit likely stores pack files in S3 or otherwise takes advantage of existing AWS systems.. Using a single Ceph FS is an odd choice IMO. You need to have or acquire a ton of expertise to run stuff like that, and it can be finicky. I'm not convinced you can't get this running well enough on AWS though.. So I'd be worried the move to bare metal would just cause the team more problems.

radicalbyte 10y ago

It might be, but you have to look at what's important for a product like GitLab. It's in a market where those who'll pay want to run their own special version of the system. So it's naturally partitioned. Architecting, or even spending mental cycles on day 1 on distribution isn't going to win you as much as focusing on making an awesome product. This move will probably buy them another year or two, which will give them enough time hopefully implement some form of partitioning.

jondubois 10y ago

Agreed, I think the wrong conclusion was drawn here. But I get where they're coming from; container orchestrators like Kubernetes are heavily promoting distributed file systems as being the 'cloud-native approach'. But maybe this issue is more relevant to 'CephFS' specifically than to all distributed file systems in general.

elygre 10y ago

GitLab supports using multiple filesystems, and while waiting for their bare-metal solution, they switch to running 8 file systems: https://gitlab.com/gitlab-com/infrastructure/issues/711 https://gitlab.com/gitlab-com/infrastructure/issues/711

merb 10y ago
multiple filesystem != multiple storage backends unfortunatly.
- elygre 10y ago
  Yes, multiple storage backends: 8 file systems, on 8 independent servers, each with 16tb disk. See https://gitlab.com/gitlab-com/infrastructure/issues/727#note_18193225 https://gitlab.com/gitlab-com/infrastructure/issues/727#note....