5 ms·

What is your strategy for dealing with server failure?

mad182 5y ago
My app allows for "share nothing" architecture, basically using multiple DNS A records as load balancing. Currently it has 6 servers. Even if 5 of them would go down at the same time, the site would still work as intended (thought probably couldn't handle peak load with less than 3 or 4). If one of or two are down, nothing happens. Also completely reinstalling a server takes around an hour.
- SergeAx 5y ago
  Yes, but comment I was replying to mentioned a 48 euro budget, it is a price of a single server.
heipei 5y ago
As long as you treat servers as cattle, you can use Hetzner's own load balancer service and then you don't have a SPOF that you manage yourself. Their LBs are advertised as redundant / fault tolerant.
- SergeAx 5y ago
  I don't know if it is possible to treat "Serverbörse" servers as cattle. They are all different. I know that k8s and Docker Swarm could in theory balance load between different machines, but never tried it in practice. But I had in practice some weird glitches with different CPU/motherboards/memory. Also, comment I was replying to mentioned a 48 euro budget, it is a price of a single server.
  - hansel_der 5y ago
    i think the whole point of vm/container is to enable "servers as cattle"
    - SergeAx 5y ago
      Yes, but "Serverbörse" machines are very non-uniform, which may be bad for load balancing. See yourself: https://www.hetzner.com/sb https://www.hetzner.com/sb
      - hansel_der 5y ago
        yes, it may be bad for indiscriminate/dumb load balancing. then again this only works in the firstplace if the work units represent a somewhat equivalent and small-ish workload. once you place value on determinism (in regards to time spent on a task) you want a tightly specced distribution mechanism and/or a feedback loop to communicate busystate back to the LB.
      - halilduygulu 5y ago
        Still you can get a nice server with i7 cpu 32gb memory with 20tb traffic for ~28 euro, vat incl. This is super competitive even with DO or others.
      - withinboredom 5y ago
        I have 3 servers, 2/3 don't have hardware timestamping on the interface, 1 does. Makes a huge difference when it comes to NTP.
        csdvrx 5y ago
        Which hetzner server have hardware timestamping on the interface? I'd be really interested to know, since to the best of my knowledge, they don't have PTP solutions in their datacenter.
        withinboredom 5y ago
        I have two of the AX41-NVMe, and only one was lucky enough to get a "Intel Corporation I210 Gigabit Network Connection (rev 03)" which has the hardware timestamping. In the finland datacenter, it appears there is a PTP running, though the offset is ~2.33 seconds off from NTP. Chrony says its a false-ticker, but I haven't really figured out how to get it configured correctly nor have I asked Hetzner for help. I've mostly just played around with it. I did manage to also accidentally discover a local ISP's stratum 1's server is extremely close to me, as in a few microseconds away (I accidentally put hetzner's servers as a pool instead of a server and my NTP 'discovered' the stratum 1) ... I'm not using it, but I've thought about reaching out to them to ask if I can use it if I'm very nice.
razemio 5y ago
I do not have an HA setup BUT all of my proxmox vms are snapshot backuped by proxmox backup server every night to my home NAS and to my office NAS. You can use one of many storage providers. SSHFS also works. This is the cheapest and lowest administration solution I used till today. For production usage I would recommend 3-4 similar speced 28€ machines and run a replicated proxmox cluster or ceph proxmox cluster.

My app allows for "share nothing" architecture, basically using multiple DNS A records as load balancing. Currently it has 6 servers. Even if 5 of them would go down at the same time, the site would still work as intended (thought probably couldn't handle peak load with less than 3 or 4). If one of or two are down, nothing happens. Also completely reinstalling a server takes around an hour.

SergeAx 5y ago
Yes, but comment I was replying to mentioned a 48 euro budget, it is a price of a single server.

heipei 5y ago

As long as you treat servers as cattle, you can use Hetzner's own load balancer service and then you don't have a SPOF that you manage yourself. Their LBs are advertised as redundant / fault tolerant.

SergeAx 5y ago
I don't know if it is possible to treat "Serverbörse" servers as cattle. They are all different. I know that k8s and Docker Swarm could in theory balance load between different machines, but never tried it in practice. But I had in practice some weird glitches with different CPU/motherboards/memory. Also, comment I was replying to mentioned a 48 euro budget, it is a price of a single server.
- hansel_der 5y ago
  i think the whole point of vm/container is to enable "servers as cattle"
  - SergeAx 5y ago
    Yes, but "Serverbörse" machines are very non-uniform, which may be bad for load balancing. See yourself: https://www.hetzner.com/sb https://www.hetzner.com/sb
    - hansel_der 5y ago
      yes, it may be bad for indiscriminate/dumb load balancing. then again this only works in the firstplace if the work units represent a somewhat equivalent and small-ish workload. once you place value on determinism (in regards to time spent on a task) you want a tightly specced distribution mechanism and/or a feedback loop to communicate busystate back to the LB.
    - halilduygulu 5y ago
      Still you can get a nice server with i7 cpu 32gb memory with 20tb traffic for ~28 euro, vat incl. This is super competitive even with DO or others.
    - withinboredom 5y ago
      I have 3 servers, 2/3 don't have hardware timestamping on the interface, 1 does. Makes a huge difference when it comes to NTP.
      - csdvrx 5y ago
        Which hetzner server have hardware timestamping on the interface? I'd be really interested to know, since to the best of my knowledge, they don't have PTP solutions in their datacenter.
        withinboredom 5y ago
        I have two of the AX41-NVMe, and only one was lucky enough to get a "Intel Corporation I210 Gigabit Network Connection (rev 03)" which has the hardware timestamping. In the finland datacenter, it appears there is a PTP running, though the offset is ~2.33 seconds off from NTP. Chrony says its a false-ticker, but I haven't really figured out how to get it configured correctly nor have I asked Hetzner for help. I've mostly just played around with it. I did manage to also accidentally discover a local ISP's stratum 1's server is extremely close to me, as in a few microseconds away (I accidentally put hetzner's servers as a pool instead of a server and my NTP 'discovered' the stratum 1) ... I'm not using it, but I've thought about reaching out to them to ask if I can use it if I'm very nice.

razemio 5y ago

I do not have an HA setup BUT all of my proxmox vms are snapshot backuped by proxmox backup server every night to my home NAS and to my office NAS. You can use one of many storage providers. SSHFS also works. This is the cheapest and lowest administration solution I used till today. For production usage I would recommend 3-4 similar speced 28€ machines and run a replicated proxmox cluster or ceph proxmox cluster.