13 ms·

I appreciate that you didn't sling mud at Azure, but re-reading the commit for the move to Azure [1] there were tell tale signs then that it might be bumpy for

by boulos 10y ago

I appreciate that you didn't sling mud at Azure, but re-reading the commit for the move to Azure [1] there were tell tale signs then that it might be bumpy for the storage layer. For what it's worth, hardware doesn't provide an IOPS/latency SLA either ;). In all seriousness, we (Google, all providers) struggle with deciding what we can strictly promise. Offering you a "guaranteed you can hit 50k IOPS" SLA isn't much comfort if we know that all it takes is a single ToR failure for that to be not true (providers could still offer it, have you ask for a refund if affected, etc. but your experience isn't changed). All that said, I would encourage you to reconsider. I know you're frustrated, but rolling your own infrastructure just means you have to build systems even better than the providers. On the plus side, when it's your fault, it's your fault (or the hardware vendor, or the colo facility). You've been through a lot already, but I'd suggest you'd be better off returning to AWS or coming to us (Google) [Note: Our PD offering historically allowed up to 10 TiB per disk and is now a full 64 TiB, I'm sorry if the docs were confusing]. Again, I'm not saying this to have you come to us or another cloud provider, but because I honestly believe this would be a huge time sink for GitLab. Instead of focusing on your great product, you'd have to play "Let's order more storage" (honestly managing Ceph has a similar annoyance). I'm sorry you had a bad experience with your provider, but it's not all the same. Feel free to reach out to me or others, if you want to chat further. Disclosure: I work on Google Cloud. [1] https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/1097/diffs https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/...

Dwolb 10y ago
I think what you're seeing in this post has product implications for Google Cloud. The OP is asking for guaranteed IOPS/latency SLA for which they're willing to pay HUGE money for by ROLLING THEIR OWN. Possibly think about high service pricing tiers for systems that require it. This is a standard operations problem that can co-exist within pooled service models.
- boulos 10y ago
  See my note about SLAs and ToR failures. We probably could promise something for our Local SSD offering (tail latency < 1ms!), but high-performance, guaranteed networked storage is just tricky. As I said, rolling your own will not give you a guarantee, it will just give you the responsibility for failure. We don't offer the guarantee, because we don't want you to believe it can't fail.
  - ocdtrekkie 10y ago
    At a certain scale, it's better to just take that responsibility rather than rely on someone to be able to handle that burden. When something fails, the worst thing is being out of control of it and unable to do anything about it. Last time I had a failed storage controller, HP delivered a replacement in four hours. Last time a service provider went down, I had no visibility into the repair process, and had to explain there was nothing I could do... For five days. It seems like you've outright admitted you can't guarantee what they need, yet you still urge them not to leave your business model. Maybe come back when your business model can meet their needs.
    - smallnamespace 10y ago
      I think you're missing the point a bit -- the claim is not that the business model prevents these guarantees, but that it's inherently difficult for any party to do. > had to explain there was nothing I could do... For five days The good alternative would be that you controlled the infrastructure and it didn't break. The bad alternative is you directly control infrastructure, it breaks, and then you get fired.
      - toomuchtodo 10y ago
        Wikipedia, Stack Overflow, and Github all do just fine hosting their own physical infrastructure. It's untrue that "the cloud" is always the right way. When you're on a multi-tenant system, you will never be the top priority. When you build your own, you are. Google and AWS have vested interests (~30% margins) in getting you into the cloud. Always do the math to see if it's cost effective comparatively speaking.
        solipsism 10y ago
        That's not a strawman. It's just a false statement.
        toomuchtodo 10y ago
- user5994461 10y ago
  > The OP is asking for guaranteed IOPS/latency SLA for which they're willing to pay HUGE money for by ROLLING THEIR OWN. There is no guarantee of SLA in any distributed system. The best you can do is measure things and know what you'll get most of the time. If you want SLA, you can make a single server with 10TB of memory as storage. That's solid choice! :D
- sytse 10y ago
  I'm not sure we want to pay huge money. Right now it looks like our cloud hosting bill for 2 months (about $250k) can pay for the hardware to host 4x as much https://docs.google.com/spreadsheets/d/1XG9VXdDxNd8ipgPlEr7Nb7Eg22twXPuzgDwsOhtdYKQ/edit#gid=2003595229 https://docs.google.com/spreadsheets/d/1XG9VXdDxNd8ipgPlEr7N...
  - Dwolb 10y ago
    How are you calculating cost to your organization to run your own hardware? With a cloud provider you're benefitting from their own engineering pooled across thousands of customers. When you run your own hardware you have all the engineering you were already doing plus investing to upkeep and to improve your architecture. As Google's boulos said, that's where the real costs are.
    - sytse 10y ago
      Indeed with metal you have less flexibility and much higher engineering costs that offset your savings. We think metal will be more affordable as we scale but that is not the reason for doing it. We do it because it is the only way to scale Ceph.
      - nixgeek 10y ago
        I know at least two 500TB+ clusters running on IaaS and don't think "only way to scale Ceph" is to buy and rack machines. In an earlier comment you said EBS only goes to 16TB and that is "an order of magnitude less" than your requirement, however, thats per volume, you can attach many volumes in much the same way as servers have many disks. Scale horizontally not vertically, add more OSD instances? With each you can attach a number of EBS or PD volumes which each IOPS characteristics that in aggregate are sufficient to service your workload? If you want to avoid EBS or PD entirely, is there a reason you can't look at 'i2' or 'd2' instance types? https://cloud.google.com/compute/docs/disks/performance https://cloud.google.com/compute/docs/disks/performance https://aws.amazon.com/ebs/details/#VolumeTypes https://aws.amazon.com/ebs/details/#VolumeTypes At a fundamental level you're just moving the problem and trading managing metal (which is hard) for I/O guarantees. "Why is this harder than you might expect?" - you stated elsewhere that you'll have Remote Hands do rack/stack. Providers like Equinix refer to this as "Smart Hands". Everyone who's managed a reasonable-sized environment finds this term highly ironic, as the technician can and will replace the wrong drive, pull the wrong cable, etc. I've done an non-trivial amount of infrastructure 'stuff' (design, procurement, install, maintenance, migration) for some well-known companies, if you want to Hangout for an hour and pick my brain, gratis, my e-mail is in my profile.
WhatIsDukkha 10y ago
"""For what it's worth, hardware doesn't provide an IOPS/latency SLA either ;).""" Actually it really does for all practical purposes. When it fails its SLA you replace it.
- diziet 10y ago
  On the storage front, even enterprise grade SSDs with hundreds of thousands of 4k read and write IOPs set up in over-provisioned arrays won't give you a great SLA unless you really lowball it and say the SLA is 99th percentile latency / IOPS is 1/2 benchmarked and the hardware has to sustain that latency/IOPS for 99.99% (after benchmarking out to figure out cache vs sustained performance) That's just storage, now you need to add so many layers on top of it. Like @boulos said, it is then your fault, but your customers still see the issues.
merb 10y ago
I sense a collaboration oppurtunity between gcloud and gitlab might be benefitical, while google helps gitlab getting git running against your object storage ;) would be a win-win for both I guess!
Spooky23 10y ago
If you have an application where the performance and growth patterns are well known, cloud isn't a choice made for lowest cost. It may be best value for different reasons. I've done ROI studies for several applications like that, and usually cloud has a higher total cost unless there are specific availability requirements or you don't have a facility that can meet a 99.9 SLA. The key assumption is that you know a lot about the app and it's in a operating mode. If you're in a hyper growth mode, have fluctuating or seasonal demand, or you have no capital funds, cloud is a no brainer.
- brianwawok 10y ago
  For sure, I think you can often save a decent amount of $$ with dedicated hardware. It does have downsides of course... If your demand is 50% stable 50% fluctuating (say some base load + a big spike at US prime time), I still think you can win with a hybrid cloud... i.e. serve base load from a COLO, and serve spike load from the cloud. That does mean you need to configure at least 2 networks, but not a terrible idea from a DR standpoint anyway (Main DC fail? Push a button and run off the cloud until it is fixed)
  - northrup 10y ago
    This thinking (hybrid cloud for dynamic scale out) is going into our decision at GitLab and makes a lot of sense for us.
connorshea 10y ago
I just want to note that the blog post you linked to was never actually published since we decided against pointing fingers when – by and large – the problems weren't their fault and their engineers were quite helpful. See the comment at the bottom of that thread[1] from Sid. We made mistakes ourselves that were the root of the problems at the time, and based on the update[2] a few months later things definitely improved significantly. [1]: https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/1097#note_2758069 https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/... [2]: https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/1097#note_3738731 https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/...
- boulos 10y ago
  Sorry about that! I was on my phone, and followed the first thing from the linked post/issue; I didn't realize it wasn't published.
  - connorshea 10y ago
    No problem, the thread is pretty long, and it's not necessarily easy to tell without understanding how our blog post process works. :)
- user5994461 10y ago
  Judging by the posts. Random thought: Go talk to google and get $100k of free cloud credits. You run 100 instances of n1-highcpu-2 (2 CPU, 2GB RAM), with a local 375 SSD. That gives you 37TB of storage and you're fine for the year. This is an absolutely ridiculous setup but it could work. 1) Servers don't need to have more memory for caching. Local SSD access is fast enough. 2) Servers have 1Gb uplink. That can be saturated easily by a single local SSD. 3) I'm a firm believer that if you put 8-16 disks in a single box, you're gonna hit a hard limit with the network bandwidth and infrastructure (among other things). 4) Do you need IOPS or do you need storage??? 5) Can CephFS shard fairly across a hundred systems? 6) Should CephFS run on raw disks, manage replica itself? and can it handle failure? Bonus question) Why are we thinking about all of this. You should have just used s3?
taneq 10y ago
> rolling your own infrastructure just means you have to build systems even better than the providers As a cloud provider, though, you're trying to provide shared resources to a group of clients. A company rolling their own system doesn't have to share, and they can optimise specifically for their own requirements. You don't get these luxuries, and it's reasonable to expect a customised system to perform better than a general one.
- boulos 10y ago
  As a counter argument: very few teams at Google run on dedicated machines. Those that do are enormous, both in the scale of their infrastructure and in their team sizes. I'm not saying always go with a cloud provider, I'm reiterating that you'd better be certain you need to.
  - jiqiren 10y ago
    But some workloads get all the priority while others get zero/idle priority. Not true in public cloud.
    - datadata 10y ago
      Using a resource abstraction layer such as mesos can alleviate this downside by consolidating many of your workloads onto a pool of large dedicated machines.
    - dastbe 10y ago
      Multitenancy is a large part of what makes public cloud providers profitable, but they all understand the need to isolate customer resources as much as possible.
  - deleted 10y ago
    [deleted]
  - taneq 10y ago
    Interesting, presumably they're very well informed and they obviously feel that Google's cloud offerings are the best way to go. If I may ask a few questions: - Are they charged the same as external customers or do they get a 'wholesale' rate? - As internal clients, do they run under the same conditions as external clients? Or is there a shared internal server pool that they use? - Do they get any say in the hardware or low-level configuration of the systems they use? (ie. if someone needs ultra low latency or more storage, can they just ask Joe down the hall for a machine on a more lightly loaded network, or with bunch more RAM, for the week?) - Do they have the same type of performance constraints as the ones encountered by gitlab? I feel like most of the reason to use cloud services is when you have little idea what your actual requirements are, and need the ability to scale fast in both directions. Once you're established and have a fairly predictable workload, it makes more sense to move your hosting in-house.
hitekker 10y ago
Technical details aside, the "buts", parentheses, and generally weird structure of this reply made it very confusing. I personally believe that, when a prospect feels wronged or poorly served by a competitor, it's the company's responsibility to reach out to that prospect directly. Ideally 1-on-1, the company should aim to listen carefully to the prospect's concerns and not to immediately start spouting opinions or solutions. From there, an honest dialogue can take place which can form the basis of a sale. And, more importantly, a relationship. Of course, if the goal is simply to castigate the competition or to ward off uncomfortable questions by implying the dissatisfied prospect is thinking poorly or emotionally, then a public forum post works better, I guess.
- boulos 10y ago
  Sorry if you found my writing style confusing. I'm not looking to shame anyone. Instead, my hope was to counter the overly broad "It's clear we must not be in the Cloud" with my experience that this may be particular to their provider. I'm not here trying to sell them on coming to Google. If they're interested, my contact info is in my profile. It's quite likely that the best option for them would be to return to AWS, as they have experience running there and EBS has improved a lot over the years.
stuntprogrammer 10y ago
I always recommend these: https://scalableinformatics.com/assets/documents/Unison_Petabyte.pdf https://scalableinformatics.com/assets/documents/Unison_Peta... Surprisingly cost effective given the massive performance and support. Cloud is great for some things but not everything.
- sijoe 10y ago
  Thanks for the shout out! Without being a commercial, this high performance storage/analytics realm is what we focus on. We were demoing units like this: https://scalability.org/images/30GBps.png https://scalability.org/images/30GBps.png for years now. Insanely fast, tastes great, less filling. Our NVM versions are pretty awesome as well. [edit] I should point out that we build systems that the OP wants ... they likely don't know about us as we are a small company ... https://scalableinformatics.com https://scalableinformatics.com and we do use gitlab https://gitlab.scalableinformatics.com https://gitlab.scalableinformatics.com ... Always happy to help ...
  - sytse 10y ago
    Thanks for using GitLab and posting in https://gitlab.com/gitlab-com/infrastructure/issues/727#note_18226191 https://gitlab.com/gitlab-com/infrastructure/issues/727#note... This looks really interesting.
qaq 10y ago
hardware provides so much headroom that it's not even funny to compare :) a good PCIE board will do 850K IOPS for reads for 10K (4TB). You can have a sane understandable setup. In our experience our clients on AWS had more downtime than on dedicated clusters. All major outages of public clouds have being due to control plane issues because it's too complex. Unless you are a major customer have fun trying to get things resolved with the cloud provider. At gitlub scale they would need 4 ops guys and some remote hands for hardware installs considering cloud pricing on bandwidth and storage and the cost of the ops team (+/- 24K month in Ukraine tops) they will not only get much better performance but will actually save decent amount of money.
jfim 10y ago
ToR failure? Tried googling it but it wasn't much help.
- kevan 10y ago
  Top of Rack = the network switch that ties the servers on a rack to each other and provides a gateway to the rest of the datacenter.
ex3ndr 10y ago
Ah, really? We are moved from Google Cloud, we just needed one machine with average IOPS to store our files (50-100gb only), but we are constantly hitting limits of IOPS, file upload for our users at some point became a nightmare. What's a problem with cloud when we pay for it 1k$+? Renting one server costs ~50$/mo and will cover everything. Only thing is needed is some adjustments in configuration. But, thanks to k8s we can roll out servers in a week. I am wonder now why we was needed to sit on clouds when everything works 10x times slower at 10x price?
- static_noise 10y ago
  What if your hardware fails? What if you need to scale up (or scale down again)? Making servers run as reliable as in cloud datacenters is really hard work and imposes additional cost. I.e. you need not one, but two or three datacenters with 2-3 times the amount of servers you actually use. Then you need admin staff who knows their work and keeps things running smoothly and reliably. You need not one but at least 3 of that staff because they might get sick or go on vacation. Your cost savings probably comes from cutting corners. Depending on the structure of your applications, downtime requirements and recovery plans, that may even be OK. You don't always need a fleet of tanks to deliver a box of milk bottles.
  - ex3ndr 10y ago
    Hardware fails? It will start on other node automagically. May be we will need to switch psql instance. What kind of additional cost? Do you ever hit something really awful with k8s on self-hosted servers? What can go wrong except losing node?
    - static_noise 10y ago
      So, you have redundancy and failover at the application level. You don't need reliable servers then.
      - empthought 10y ago
gressquel 10y ago
I did not realize they were on Azure until I read this comment. GitLab post did not mention it, why did you then?
- NetStrikeForce 10y ago
  I'm glad I wasn't the only one to notice! Literally the first thing the Google engineer does is mention Azure, while I had to go back to Gitlab's article all confused as I didn't notice at any point they were running on Azure. I can't say it's a bad thing to do; I just can't help but notice how Google invests in social media participation. That's why they own the conversation in places like HN and can pull off this stuff. What I fail to understand is, how does GCE or AWS tackle the issue described in the article? As far as I understand, their problem seems difficult to work around due to the nature of the Cloud (shared). How would GCE be better than AWS or Azure at this? I would be really interested to know and I'm sure that'll be useful for other HNers with the same worries.
  - illumin8 10y ago
    Disclaimer: I work for AWS. To solve problems just like these, we offer EBS (elastic block storage) with provisioned IOPs guarantees. Essentially, you can get guaranteed IOPs if you need it for I/O intensive applications; up to 30,000 IOPs per EBS volume. But, PIOPs EBS volumes wouldn't be my first recommendation. It sounds like what they really need is an elastic, scale-out filesystem with NFS semantics. We have Elastic File System, or EFS, which is exactly that. It's a petabyte scale filesystem that is highly available across multiple availability zones, and scales in IOPs and performance as it scales in size. Their application should also look at leveraging S3 object storage, rather than NFS, because that is a highly distributed, highly available object storage system, that is likely to give better scalability, availability, and performance, than rolling your own Ceph infrastructure.
    - NetStrikeForce 10y ago
      This is a great answer, thanks illumin8! (and cool nick too :D)

by boulos 10y ago

Dwolb 10y ago

I think what you're seeing in this post has product implications for Google Cloud. The OP is asking for guaranteed IOPS/latency SLA for which they're willing to pay HUGE money for by ROLLING THEIR OWN. Possibly think about high service pricing tiers for systems that require it. This is a standard operations problem that can co-exist within pooled service models.

boulos 10y ago
See my note about SLAs and ToR failures. We probably could promise something for our Local SSD offering (tail latency < 1ms!), but high-performance, guaranteed networked storage is just tricky. As I said, rolling your own will not give you a guarantee, it will just give you the responsibility for failure. We don't offer the guarantee, because we don't want you to believe it can't fail.
- ocdtrekkie 10y ago
  At a certain scale, it's better to just take that responsibility rather than rely on someone to be able to handle that burden. When something fails, the worst thing is being out of control of it and unable to do anything about it. Last time I had a failed storage controller, HP delivered a replacement in four hours. Last time a service provider went down, I had no visibility into the repair process, and had to explain there was nothing I could do... For five days. It seems like you've outright admitted you can't guarantee what they need, yet you still urge them not to leave your business model. Maybe come back when your business model can meet their needs.
  - smallnamespace 10y ago
    I think you're missing the point a bit -- the claim is not that the business model prevents these guarantees, but that it's inherently difficult for any party to do. > had to explain there was nothing I could do... For five days The good alternative would be that you controlled the infrastructure and it didn't break. The bad alternative is you directly control infrastructure, it breaks, and then you get fired.
    - toomuchtodo 10y ago
      Wikipedia, Stack Overflow, and Github all do just fine hosting their own physical infrastructure. It's untrue that "the cloud" is always the right way. When you're on a multi-tenant system, you will never be the top priority. When you build your own, you are. Google and AWS have vested interests (~30% margins) in getting you into the cloud. Always do the math to see if it's cost effective comparatively speaking.
      - solipsism 10y ago
        That's not a strawman. It's just a false statement.
        toomuchtodo 10y ago
user5994461 10y ago
> The OP is asking for guaranteed IOPS/latency SLA for which they're willing to pay HUGE money for by ROLLING THEIR OWN. There is no guarantee of SLA in any distributed system. The best you can do is measure things and know what you'll get most of the time. If you want SLA, you can make a single server with 10TB of memory as storage. That's solid choice! :D
sytse 10y ago
I'm not sure we want to pay huge money. Right now it looks like our cloud hosting bill for 2 months (about $250k) can pay for the hardware to host 4x as much https://docs.google.com/spreadsheets/d/1XG9VXdDxNd8ipgPlEr7Nb7Eg22twXPuzgDwsOhtdYKQ/edit#gid=2003595229 https://docs.google.com/spreadsheets/d/1XG9VXdDxNd8ipgPlEr7N...
- Dwolb 10y ago
  How are you calculating cost to your organization to run your own hardware? With a cloud provider you're benefitting from their own engineering pooled across thousands of customers. When you run your own hardware you have all the engineering you were already doing plus investing to upkeep and to improve your architecture. As Google's boulos said, that's where the real costs are.
  - sytse 10y ago
    Indeed with metal you have less flexibility and much higher engineering costs that offset your savings. We think metal will be more affordable as we scale but that is not the reason for doing it. We do it because it is the only way to scale Ceph.
    - nixgeek 10y ago
      I know at least two 500TB+ clusters running on IaaS and don't think "only way to scale Ceph" is to buy and rack machines. In an earlier comment you said EBS only goes to 16TB and that is "an order of magnitude less" than your requirement, however, thats per volume, you can attach many volumes in much the same way as servers have many disks. Scale horizontally not vertically, add more OSD instances? With each you can attach a number of EBS or PD volumes which each IOPS characteristics that in aggregate are sufficient to service your workload? If you want to avoid EBS or PD entirely, is there a reason you can't look at 'i2' or 'd2' instance types? https://cloud.google.com/compute/docs/disks/performance https://cloud.google.com/compute/docs/disks/performance https://aws.amazon.com/ebs/details/#VolumeTypes https://aws.amazon.com/ebs/details/#VolumeTypes At a fundamental level you're just moving the problem and trading managing metal (which is hard) for I/O guarantees. "Why is this harder than you might expect?" - you stated elsewhere that you'll have Remote Hands do rack/stack. Providers like Equinix refer to this as "Smart Hands". Everyone who's managed a reasonable-sized environment finds this term highly ironic, as the technician can and will replace the wrong drive, pull the wrong cable, etc. I've done an non-trivial amount of infrastructure 'stuff' (design, procurement, install, maintenance, migration) for some well-known companies, if you want to Hangout for an hour and pick my brain, gratis, my e-mail is in my profile.

WhatIsDukkha 10y ago

"""For what it's worth, hardware doesn't provide an IOPS/latency SLA either ;).""" Actually it really does for all practical purposes. When it fails its SLA you replace it.

diziet 10y ago
On the storage front, even enterprise grade SSDs with hundreds of thousands of 4k read and write IOPs set up in over-provisioned arrays won't give you a great SLA unless you really lowball it and say the SLA is 99th percentile latency / IOPS is 1/2 benchmarked and the hardware has to sustain that latency/IOPS for 99.99% (after benchmarking out to figure out cache vs sustained performance) That's just storage, now you need to add so many layers on top of it. Like @boulos said, it is then your fault, but your customers still see the issues.

merb 10y ago

I sense a collaboration oppurtunity between gcloud and gitlab might be benefitical, while google helps gitlab getting git running against your object storage ;) would be a win-win for both I guess!

Spooky23 10y ago

If you have an application where the performance and growth patterns are well known, cloud isn't a choice made for lowest cost. It may be best value for different reasons. I've done ROI studies for several applications like that, and usually cloud has a higher total cost unless there are specific availability requirements or you don't have a facility that can meet a 99.9 SLA. The key assumption is that you know a lot about the app and it's in a operating mode. If you're in a hyper growth mode, have fluctuating or seasonal demand, or you have no capital funds, cloud is a no brainer.

brianwawok 10y ago
For sure, I think you can often save a decent amount of $$ with dedicated hardware. It does have downsides of course... If your demand is 50% stable 50% fluctuating (say some base load + a big spike at US prime time), I still think you can win with a hybrid cloud... i.e. serve base load from a COLO, and serve spike load from the cloud. That does mean you need to configure at least 2 networks, but not a terrible idea from a DR standpoint anyway (Main DC fail? Push a button and run off the cloud until it is fixed)
- northrup 10y ago
  This thinking (hybrid cloud for dynamic scale out) is going into our decision at GitLab and makes a lot of sense for us.

connorshea 10y ago

I just want to note that the blog post you linked to was never actually published since we decided against pointing fingers when – by and large – the problems weren't their fault and their engineers were quite helpful. See the comment at the bottom of that thread[1] from Sid. We made mistakes ourselves that were the root of the problems at the time, and based on the update[2] a few months later things definitely improved significantly. [1]: https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/1097#note_2758069 https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/... [2]: https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/1097#note_3738731 https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/...

boulos 10y ago
Sorry about that! I was on my phone, and followed the first thing from the linked post/issue; I didn't realize it wasn't published.
- connorshea 10y ago
  No problem, the thread is pretty long, and it's not necessarily easy to tell without understanding how our blog post process works. :)
user5994461 10y ago
Judging by the posts. Random thought: Go talk to google and get $100k of free cloud credits. You run 100 instances of n1-highcpu-2 (2 CPU, 2GB RAM), with a local 375 SSD. That gives you 37TB of storage and you're fine for the year. This is an absolutely ridiculous setup but it could work. 1) Servers don't need to have more memory for caching. Local SSD access is fast enough. 2) Servers have 1Gb uplink. That can be saturated easily by a single local SSD. 3) I'm a firm believer that if you put 8-16 disks in a single box, you're gonna hit a hard limit with the network bandwidth and infrastructure (among other things). 4) Do you need IOPS or do you need storage??? 5) Can CephFS shard fairly across a hundred systems? 6) Should CephFS run on raw disks, manage replica itself? and can it handle failure? Bonus question) Why are we thinking about all of this. You should have just used s3?

taneq 10y ago

> rolling your own infrastructure just means you have to build systems even better than the providers As a cloud provider, though, you're trying to provide shared resources to a group of clients. A company rolling their own system doesn't have to share, and they can optimise specifically for their own requirements. You don't get these luxuries, and it's reasonable to expect a customised system to perform better than a general one.

boulos 10y ago
As a counter argument: very few teams at Google run on dedicated machines. Those that do are enormous, both in the scale of their infrastructure and in their team sizes. I'm not saying always go with a cloud provider, I'm reiterating that you'd better be certain you need to.
- jiqiren 10y ago
  But some workloads get all the priority while others get zero/idle priority. Not true in public cloud.
  - datadata 10y ago
    Using a resource abstraction layer such as mesos can alleviate this downside by consolidating many of your workloads onto a pool of large dedicated machines.
  - dastbe 10y ago
    Multitenancy is a large part of what makes public cloud providers profitable, but they all understand the need to isolate customer resources as much as possible.
- deleted 10y ago
  [deleted]
- taneq 10y ago
  Interesting, presumably they're very well informed and they obviously feel that Google's cloud offerings are the best way to go. If I may ask a few questions: - Are they charged the same as external customers or do they get a 'wholesale' rate? - As internal clients, do they run under the same conditions as external clients? Or is there a shared internal server pool that they use? - Do they get any say in the hardware or low-level configuration of the systems they use? (ie. if someone needs ultra low latency or more storage, can they just ask Joe down the hall for a machine on a more lightly loaded network, or with bunch more RAM, for the week?) - Do they have the same type of performance constraints as the ones encountered by gitlab? I feel like most of the reason to use cloud services is when you have little idea what your actual requirements are, and need the ability to scale fast in both directions. Once you're established and have a fairly predictable workload, it makes more sense to move your hosting in-house.

hitekker 10y ago

Technical details aside, the "buts", parentheses, and generally weird structure of this reply made it very confusing. I personally believe that, when a prospect feels wronged or poorly served by a competitor, it's the company's responsibility to reach out to that prospect directly. Ideally 1-on-1, the company should aim to listen carefully to the prospect's concerns and not to immediately start spouting opinions or solutions. From there, an honest dialogue can take place which can form the basis of a sale. And, more importantly, a relationship. Of course, if the goal is simply to castigate the competition or to ward off uncomfortable questions by implying the dissatisfied prospect is thinking poorly or emotionally, then a public forum post works better, I guess.

boulos 10y ago
Sorry if you found my writing style confusing. I'm not looking to shame anyone. Instead, my hope was to counter the overly broad "It's clear we must not be in the Cloud" with my experience that this may be particular to their provider. I'm not here trying to sell them on coming to Google. If they're interested, my contact info is in my profile. It's quite likely that the best option for them would be to return to AWS, as they have experience running there and EBS has improved a lot over the years.

stuntprogrammer 10y ago

I always recommend these: https://scalableinformatics.com/assets/documents/Unison_Petabyte.pdf https://scalableinformatics.com/assets/documents/Unison_Peta... Surprisingly cost effective given the massive performance and support. Cloud is great for some things but not everything.

sijoe 10y ago
Thanks for the shout out! Without being a commercial, this high performance storage/analytics realm is what we focus on. We were demoing units like this: https://scalability.org/images/30GBps.png https://scalability.org/images/30GBps.png for years now. Insanely fast, tastes great, less filling. Our NVM versions are pretty awesome as well. [edit] I should point out that we build systems that the OP wants ... they likely don't know about us as we are a small company ... https://scalableinformatics.com https://scalableinformatics.com and we do use gitlab https://gitlab.scalableinformatics.com https://gitlab.scalableinformatics.com ... Always happy to help ...
- sytse 10y ago
  Thanks for using GitLab and posting in https://gitlab.com/gitlab-com/infrastructure/issues/727#note_18226191 https://gitlab.com/gitlab-com/infrastructure/issues/727#note... This looks really interesting.

qaq 10y ago

hardware provides so much headroom that it's not even funny to compare :) a good PCIE board will do 850K IOPS for reads for 10K (4TB). You can have a sane understandable setup. In our experience our clients on AWS had more downtime than on dedicated clusters. All major outages of public clouds have being due to control plane issues because it's too complex. Unless you are a major customer have fun trying to get things resolved with the cloud provider. At gitlub scale they would need 4 ops guys and some remote hands for hardware installs considering cloud pricing on bandwidth and storage and the cost of the ops team (+/- 24K month in Ukraine tops) they will not only get much better performance but will actually save decent amount of money.

jfim 10y ago

ToR failure? Tried googling it but it wasn't much help.

kevan 10y ago
Top of Rack = the network switch that ties the servers on a rack to each other and provides a gateway to the rest of the datacenter.

ex3ndr 10y ago

Ah, really? We are moved from Google Cloud, we just needed one machine with average IOPS to store our files (50-100gb only), but we are constantly hitting limits of IOPS, file upload for our users at some point became a nightmare. What's a problem with cloud when we pay for it 1k$+? Renting one server costs ~50$/mo and will cover everything. Only thing is needed is some adjustments in configuration. But, thanks to k8s we can roll out servers in a week. I am wonder now why we was needed to sit on clouds when everything works 10x times slower at 10x price?

static_noise 10y ago
What if your hardware fails? What if you need to scale up (or scale down again)? Making servers run as reliable as in cloud datacenters is really hard work and imposes additional cost. I.e. you need not one, but two or three datacenters with 2-3 times the amount of servers you actually use. Then you need admin staff who knows their work and keeps things running smoothly and reliably. You need not one but at least 3 of that staff because they might get sick or go on vacation. Your cost savings probably comes from cutting corners. Depending on the structure of your applications, downtime requirements and recovery plans, that may even be OK. You don't always need a fleet of tanks to deliver a box of milk bottles.
- ex3ndr 10y ago
  Hardware fails? It will start on other node automagically. May be we will need to switch psql instance. What kind of additional cost? Do you ever hit something really awful with k8s on self-hosted servers? What can go wrong except losing node?
  - static_noise 10y ago
    So, you have redundancy and failover at the application level. You don't need reliable servers then.
    - empthought 10y ago

gressquel 10y ago

I did not realize they were on Azure until I read this comment. GitLab post did not mention it, why did you then?

NetStrikeForce 10y ago
I'm glad I wasn't the only one to notice! Literally the first thing the Google engineer does is mention Azure, while I had to go back to Gitlab's article all confused as I didn't notice at any point they were running on Azure. I can't say it's a bad thing to do; I just can't help but notice how Google invests in social media participation. That's why they own the conversation in places like HN and can pull off this stuff. What I fail to understand is, how does GCE or AWS tackle the issue described in the article? As far as I understand, their problem seems difficult to work around due to the nature of the Cloud (shared). How would GCE be better than AWS or Azure at this? I would be really interested to know and I'm sure that'll be useful for other HNers with the same worries.
- illumin8 10y ago
  Disclaimer: I work for AWS. To solve problems just like these, we offer EBS (elastic block storage) with provisioned IOPs guarantees. Essentially, you can get guaranteed IOPs if you need it for I/O intensive applications; up to 30,000 IOPs per EBS volume. But, PIOPs EBS volumes wouldn't be my first recommendation. It sounds like what they really need is an elastic, scale-out filesystem with NFS semantics. We have Elastic File System, or EFS, which is exactly that. It's a petabyte scale filesystem that is highly available across multiple availability zones, and scales in IOPs and performance as it scales in size. Their application should also look at leveraging S3 object storage, rather than NFS, because that is a highly distributed, highly available object storage system, that is likely to give better scalability, availability, and performance, than rolling your own Ceph infrastructure.
  - NetStrikeForce 10y ago
    This is a great answer, thanks illumin8! (and cool nick too :D)