4 ms·

Thanks! The decision to move to metal was because of performance problems https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ https://about.gitlab.com/20

by sytse 10y ago

Thanks! The decision to move to metal was because of performance problems https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ It is nice that we'll save on costs but we anticipate a lot of extra complexity that will slow us down. So if it wasn't needed we would have stayed in the cloud. But it is interesting that both our competitors (GitHub.com and BitBucket.org) also moved to metal.

theptip 10y ago
I've been following the technical discussions around this move, and I'm wondering if you guys looked at making architectural changes to shard your data into more manageable chunks? Naively it seems like you should be able to reduce your peak filesystem iops by sharding the data at the application layer. That does introduce application complexity, but it might shake out as being less work than the operational complexity of running my own metal. Of course, easier said than done -- I just didn't spot any discussion of this option, and it seemed like the design choice of having one filesystem served by Ceph was taken for granted.
- sytse 10y ago
  We have sharding on the application layer in GitLab right now https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7273 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7273 and we're using it heavily to split the load among NFS servers. Then we have to think about redundancy. The simple solution is to have an secondary NFS server and use DRBD. For the shortcomings of that read http://githubengineering.com/introducing-dgit/ http://githubengineering.com/introducing-dgit/ The next step is introducing more granular redundancy, failover, and rebalancing. For this you have to be good in distributed computing. This is not something we are now so we rather outsource it to the experts that make CephFS. The problem of CephFS is that each file need to be tracked. If we would do it ourselves we could do it on the repository level. But we rather reuse a project that many people have already made better than go through the pain of making all the mistakes ourselves. It could be that using CephFS will not solve our latency problems and we have to do application sharing anyway.
  - theptip 10y ago
    That's a fair comment RE: outsourcing, but at my company I'd bias towards bringing some distributed computing knowledge in-house rather than bringing ops expertise plus maintenance burden in-house; sounds like you're going to have to add new expertise to your team either way. Worth investigating if you can bolt on a distributed datastore like etcd or ZooKeeper to store the cluster membership and data locations; this might not be as complex as it sounds at first. etcd gives you some very powerful primitives to work with. (For example, etcd has the concept of expiring keys, so you can keep an up-to-date list of live nodes in your network. And you can use those same primitives to keep a strongly consistent prioritized list of repos and their backed up locations. The reconciliation component might just have to listen for node keys expiring and create and register new data copies in response.)
    - sytse 10y ago
      I agree we need to add new expertise to our team anyway. But I think adding bare metal expertise is easier than distributed system expertise. Etcd is indeed very interesting. I'm thinking about using it for active active replication in https://gitlab.com/gitlab-org/gitlab-ee/issues/1381 https://gitlab.com/gitlab-org/gitlab-ee/issues/1381
      - daxelrod 10y ago
        Think about it this way: your EE customers probably have the easier bare metal knowledge, but would be willing to pay for you to solve the distributed system problems for them :)
cobookman 10y ago
Would love to chat with you to see if a switch to GCP might solve your performance and pricing issues. It's also always interesting to see how GCP vs AWS vs bare metal fairs. email me at bookman@google.com and I'll be sure to get you in contact with the right people at google.
mwcampbell 10y ago
Have you considered hosting with Packet.net? You'd be on bare metal, thus solving your performance problems, but you'd still be renting by the hour as you are now, and you wouldn't have to deal with buying your own hardware and all the complexity that comes with that.
- sytse 10y ago
  I looked at their site and they talk about bring your own block, anycast, and IPv6. But I can't find any information about networking speeds. What if we end up needing 40 Gbps between the CephFS servers?
  - justincormack 10y ago
    They provide dual 10Gb as standard. But talk to them about options.

by sytse 10y ago

theptip 10y ago

I've been following the technical discussions around this move, and I'm wondering if you guys looked at making architectural changes to shard your data into more manageable chunks? Naively it seems like you should be able to reduce your peak filesystem iops by sharding the data at the application layer. That does introduce application complexity, but it might shake out as being less work than the operational complexity of running my own metal. Of course, easier said than done -- I just didn't spot any discussion of this option, and it seemed like the design choice of having one filesystem served by Ceph was taken for granted.

sytse 10y ago
We have sharding on the application layer in GitLab right now https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7273 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7273 and we're using it heavily to split the load among NFS servers. Then we have to think about redundancy. The simple solution is to have an secondary NFS server and use DRBD. For the shortcomings of that read http://githubengineering.com/introducing-dgit/ http://githubengineering.com/introducing-dgit/ The next step is introducing more granular redundancy, failover, and rebalancing. For this you have to be good in distributed computing. This is not something we are now so we rather outsource it to the experts that make CephFS. The problem of CephFS is that each file need to be tracked. If we would do it ourselves we could do it on the repository level. But we rather reuse a project that many people have already made better than go through the pain of making all the mistakes ourselves. It could be that using CephFS will not solve our latency problems and we have to do application sharing anyway.
- theptip 10y ago
  That's a fair comment RE: outsourcing, but at my company I'd bias towards bringing some distributed computing knowledge in-house rather than bringing ops expertise plus maintenance burden in-house; sounds like you're going to have to add new expertise to your team either way. Worth investigating if you can bolt on a distributed datastore like etcd or ZooKeeper to store the cluster membership and data locations; this might not be as complex as it sounds at first. etcd gives you some very powerful primitives to work with. (For example, etcd has the concept of expiring keys, so you can keep an up-to-date list of live nodes in your network. And you can use those same primitives to keep a strongly consistent prioritized list of repos and their backed up locations. The reconciliation component might just have to listen for node keys expiring and create and register new data copies in response.)
  - sytse 10y ago
    I agree we need to add new expertise to our team anyway. But I think adding bare metal expertise is easier than distributed system expertise. Etcd is indeed very interesting. I'm thinking about using it for active active replication in https://gitlab.com/gitlab-org/gitlab-ee/issues/1381 https://gitlab.com/gitlab-org/gitlab-ee/issues/1381
    - daxelrod 10y ago
      Think about it this way: your EE customers probably have the easier bare metal knowledge, but would be willing to pay for you to solve the distributed system problems for them :)

cobookman 10y ago

Would love to chat with you to see if a switch to GCP might solve your performance and pricing issues. It's also always interesting to see how GCP vs AWS vs bare metal fairs. email me at bookman@google.com and I'll be sure to get you in contact with the right people at google.

mwcampbell 10y ago

Have you considered hosting with Packet.net? You'd be on bare metal, thus solving your performance problems, but you'd still be renting by the hour as you are now, and you wouldn't have to deal with buying your own hardware and all the complexity that comes with that.

sytse 10y ago
I looked at their site and they talk about bring your own block, anycast, and IPv6. But I can't find any information about networking speeds. What if we end up needing 40 Gbps between the CephFS servers?
- justincormack 10y ago
  They provide dual 10Gb as standard. But talk to them about options.