3 ms·
> 50 machines at hetzner - install machines with ansible (using hetzner scripts for OS install) - machines communicate over vswitch/vlans, external interfaces
by nemo136 4y ago
> 50 machines at hetzner
- install machines with ansible (using hetzner scripts for OS install)
- machines communicate over vswitch/vlans, external interfaces disabled whenever possible. Pay attention to the custom mtu trick.
- harden machines, unattended-upgrades mandatory on each machine
- ssh open with IP whitelists from iptables on gateways
- machines organized as k8s clusters, took ~1 year to have everything working cleanly
- everything deployed as k8s resources (kustomize, fluxcd, gitops)
- use keepalived for external IPs with floating IPs for ingress on 3 machines per cluster
Machines are managed as cattle, it takes <1h+ hetzner provisioning time to add as many machines as we need.
- Volundr 4y ago> Pay attention to the custom mtu trick. I wish Hetzner made this more clear up front. Maybe a big red banner on the vSwitch page. I can't count the number of hours I've spent troubleshooting network issues at Hetzner that came down to MTU.
- johne20 4y agoAs someone who is trialing Hetzner is there a link to info on this?
- Volundr 4y agoThe vSwitch docs tell you to limit MTU to 1400 https://docs.hetzner.com/robot/dedicated-server/network/vswitch/ https://docs.hetzner.com/robot/dedicated-server/network/vswi.... If you fail to do this it'll default to 1500 I think and manifest in unpredictable ways. Like being able to fetch updates one moment, then being unable to connect to the update server a few minutes later.