8 ms·

At GitLab we went from AWS to Azure, then to Google Cloud (this was a few years ago). AWS was what we started with, and I think like most companies very little

At GitLab we went from AWS to Azure, then to Google Cloud (this was a few years ago). AWS was what we started with, and I think like most companies very little attention was paid to the costs, setup, etc. The result was that we were basically setting money on fire. I think at some point Azure announced $X in free credits for YC members, and GitLab determined this would save us something like a year's worth in bills (quite a bit of money at the time). Moving over was rather painful, and I think in the months that we used Azure literally nobody was happy with it. In addition, I recall we burned through the free credits _very_ fast. I don't recall the exact reasoning for switching to GCP, but I do recall it being quite a challenging process that took quite some time. Our experiences with GCP were much better, but I wouldn't call it perfect. In particular, GCP had/has a tendency to just randomly terminate VMs for no clear reason whatsoever. Sometimes they would terminate cleanly, other times they would end up in a sort of purgatory/in between state, resulting in other systems still trying to connect to them but eventually timing out, instead of just erroring right away. IIRC over time we got better at handling it, but it felt very Google-like to me to just go "Something broke, but we'll never tell you why". Looking back, if I were to start a company I'd probably stick with something like Hetzner or another affordable bare metal provider. Cloud services are great _if_ you use their services to the fullest extend possible, but I suspect for 90% of the cases it just ends up being a huge cost factor, without the benefits making it worth it.

zegerjan 5y ago
Azure to GCP was right after GV became an investor. Again, credits was the reason to change over. Not the poor Azure performance.
k8sToGo 5y ago
I wish Hetzner provided RDS
- hardwaresofton 5y ago
  So it’s a bit in the future but managed postgres is something I want to offer through nimbus[0]. In the Meantime if you’re interested in other managed services, please sign up! [0]: https://nimbusws.com https://nimbusws.com
  - k8sToGo 5y ago
    I think your website is broken on mobile. The alignment is off and the contact form is cut off.
    - hardwaresofton 5y ago
      oh thanks -- I think I've got some image-jumping-border issues, looking into it right now!
  - anonymousDan 5y ago
    Wow, never heard of Nimbus but the kickbacks for open source projects are an awesome idea - fair play.
    - hardwaresofton 5y ago
      Thanks! It's not out yet :) I've been working on it for a bit and I really think this is what's missing from the hyperscaler/cloud model. If we just put a LITTLE bit back into (like 5% of revenue for any of the big companies honestly -- but I'm not big yet so more for me) the F/OSS ecosystem, can you imagine what kind of world we'd be in?? I want to live in that world, so I'm trying to make it happen.
- orware 5y ago
  It's still a bit early for the company but if you like RDS, PlanetScale might be worth a look too: https://planetscale.com/ https://planetscale.com/
InflexHQ 5y ago
I switched from Azure to DigitalOcean to Hetzner. Reasons were as you stated, simpler cost model, simpler technology.
- bootcat 5y ago
  does hetzner have servers in US ? if not - if your app ok to be hosted in the EU ?
  - bee_rider 5y ago
    I was also wondering this so I just looked it up -- looks like they recently (well, Nov. 2021) expanded to Virginia.
    - InflexHQ 5y ago
      My business is based in the UK which is compatible with EU’s GDPR, so it’s fine to be hosted there.
Johnny555 5y ago
I think at some point Azure announced $X in free credits for YC members, and GitLab determined this would save us something like a year's worth in bills (quite a bit of money at the time). Moving over was rather painful, and I think in the months that we used Azure literally nobody was happy with it. In addition, I recall we burned through the free credits _very_ fast. That sounds like the worst reason to change cloud providers: "because that provider bribed me to"
netcoyote 5y ago
One of the things I'm immensely curious about is how you handle security/networking/firewalls when working with Hetzner or other bare metal providers? It seems they don't provide network gear for firewalls and to protect against DDOS attacks? Do you just use iptables? Or do you build out more complex solutions like software routers running on Linux/BSD? I work in online gaming, and we're constantly seeing attacks on our infra.
- mbreese 5y ago
  If you have zero-cost internal network costs, I'd consider adding another server in front of the primary servers to act as a reverse-proxy and/or firewall. Basically, you'd use that server as a firewall and then pass only the good traffic onwards to your game servers, which are probably bigger and more expensive. If there isn't a possibility for internal-networking (free), then I'd probably use the included iptables for a firewall on each machine. You should honestly have this running on the game servers anyway, if only to restrict communication to between the reverse-proxy and game server.
- hardwaresofton 5y ago
  Hetzner has some built in DDOS protection but you should add your own: https://www.hetzner.com/unternehmen/ddos-schutz https://www.hetzner.com/unternehmen/ddos-schutz
- dijit 5y ago
  Hey! I also make games and we solved the issue by doing stateless ACLs on the border switches and having a really fat pipe. You can add a magic header to traffic and drop anything that doesn’t contain the header. Since this is done in hardware it operates at line speed. In our case 100GBit/s.
  - ramshanker 5y ago
    This seems such a good trick. We could even do it at cloudflare rule level I guess.
vidarh 5y ago
I love Hetzner, and host most of my own stuff on it. And it takes so little to be prepared to move. E.g. a basic service discovery mechanism, a reverse proxy and putting things in containers and you can migrate anywhere. Now that Hetzner has some cloud features too I see even less reason to go elsewhere (though in my current job we use AWS, but we use AWS with the explicit understanding that we're low volume - currently mostly running internal tools - and can afford the premium; if we needed to scale I'd push for putting our base load somewhere cheaper, like Hetzner) One additional suggestion to people considering bare metal: Consider baking in a VPN setup from the start, and pick a service discovery mechanism (such as e.g Consul) that is reasonably easy to operate across data centres. Now you have what you need to do migration if you need to, but you also have the backbone to turn your setup into a hybrid setup that can extend into whichever cloud provider you want too. A reason for wanting that is that one of the best ways I've found of cutting the cost of using bare metal even further is to have the ability to handle bursts by spinning up cloud instances in a pinch. It allows you to safely increase the utilisation levels of your bare metal setup substantially with according cost savings even if you in practice rarely end up needing the burst capability. It doesn't even need to be fully automated, as long as your infra setup is flexible enough to accommodate it reasonably rapidly. E.g. just having an AMI ready to go with whatever you need to have it connect to a VPN endpoint and hook into your service discovery/orchestration on startup can be enough.

zegerjan 5y ago

Azure to GCP was right after GV became an investor. Again, credits was the reason to change over. Not the poor Azure performance.

k8sToGo 5y ago

I wish Hetzner provided RDS

hardwaresofton 5y ago
So it’s a bit in the future but managed postgres is something I want to offer through nimbus[0]. In the Meantime if you’re interested in other managed services, please sign up! [0]: https://nimbusws.com https://nimbusws.com
- k8sToGo 5y ago
  I think your website is broken on mobile. The alignment is off and the contact form is cut off.
  - hardwaresofton 5y ago
    oh thanks -- I think I've got some image-jumping-border issues, looking into it right now!
- anonymousDan 5y ago
  Wow, never heard of Nimbus but the kickbacks for open source projects are an awesome idea - fair play.
  - hardwaresofton 5y ago
    Thanks! It's not out yet :) I've been working on it for a bit and I really think this is what's missing from the hyperscaler/cloud model. If we just put a LITTLE bit back into (like 5% of revenue for any of the big companies honestly -- but I'm not big yet so more for me) the F/OSS ecosystem, can you imagine what kind of world we'd be in?? I want to live in that world, so I'm trying to make it happen.
orware 5y ago
It's still a bit early for the company but if you like RDS, PlanetScale might be worth a look too: https://planetscale.com/ https://planetscale.com/

InflexHQ 5y ago

I switched from Azure to DigitalOcean to Hetzner. Reasons were as you stated, simpler cost model, simpler technology.

bootcat 5y ago
does hetzner have servers in US ? if not - if your app ok to be hosted in the EU ?
- bee_rider 5y ago
  I was also wondering this so I just looked it up -- looks like they recently (well, Nov. 2021) expanded to Virginia.
  - InflexHQ 5y ago
    My business is based in the UK which is compatible with EU’s GDPR, so it’s fine to be hosted there.

Johnny555 5y ago

I think at some point Azure announced $X in free credits for YC members, and GitLab determined this would save us something like a year's worth in bills (quite a bit of money at the time). Moving over was rather painful, and I think in the months that we used Azure literally nobody was happy with it. In addition, I recall we burned through the free credits _very_ fast. That sounds like the worst reason to change cloud providers: "because that provider bribed me to"

netcoyote 5y ago

One of the things I'm immensely curious about is how you handle security/networking/firewalls when working with Hetzner or other bare metal providers? It seems they don't provide network gear for firewalls and to protect against DDOS attacks? Do you just use iptables? Or do you build out more complex solutions like software routers running on Linux/BSD? I work in online gaming, and we're constantly seeing attacks on our infra.

mbreese 5y ago
If you have zero-cost internal network costs, I'd consider adding another server in front of the primary servers to act as a reverse-proxy and/or firewall. Basically, you'd use that server as a firewall and then pass only the good traffic onwards to your game servers, which are probably bigger and more expensive. If there isn't a possibility for internal-networking (free), then I'd probably use the included iptables for a firewall on each machine. You should honestly have this running on the game servers anyway, if only to restrict communication to between the reverse-proxy and game server.
hardwaresofton 5y ago
Hetzner has some built in DDOS protection but you should add your own: https://www.hetzner.com/unternehmen/ddos-schutz https://www.hetzner.com/unternehmen/ddos-schutz
dijit 5y ago
Hey! I also make games and we solved the issue by doing stateless ACLs on the border switches and having a really fat pipe. You can add a magic header to traffic and drop anything that doesn’t contain the header. Since this is done in hardware it operates at line speed. In our case 100GBit/s.
- ramshanker 5y ago
  This seems such a good trick. We could even do it at cloudflare rule level I guess.

vidarh 5y ago

I love Hetzner, and host most of my own stuff on it. And it takes so little to be prepared to move. E.g. a basic service discovery mechanism, a reverse proxy and putting things in containers and you can migrate anywhere. Now that Hetzner has some cloud features too I see even less reason to go elsewhere (though in my current job we use AWS, but we use AWS with the explicit understanding that we're low volume - currently mostly running internal tools - and can afford the premium; if we needed to scale I'd push for putting our base load somewhere cheaper, like Hetzner) One additional suggestion to people considering bare metal: Consider baking in a VPN setup from the start, and pick a service discovery mechanism (such as e.g Consul) that is reasonably easy to operate across data centres. Now you have what you need to do migration if you need to, but you also have the backbone to turn your setup into a hybrid setup that can extend into whichever cloud provider you want too. A reason for wanting that is that one of the best ways I've found of cutting the cost of using bare metal even further is to have the ability to handle bursts by spinning up cloud instances in a pinch. It allows you to safely increase the utilisation levels of your bare metal setup substantially with according cost savings even if you in practice rarely end up needing the burst capability. It doesn't even need to be fully automated, as long as your infra setup is flexible enough to accommodate it reasonably rapidly. E.g. just having an AMI ready to go with whatever you need to have it connect to a VPN endpoint and hook into your service discovery/orchestration on startup can be enough.