3 ms·

Lesson 1: clusters should have an odd number of nodes.

nightpool 11mo ago
I really, really think there are better lessons there. Maybe more like "Lesson 0. Don't put distributed clusters in control of your light switches"
- darkwater 11mo ago
  Why not?? It's fun!
- KetoManx64 11mo ago
  Yes, but then I'm going to have to manually go around my house and turn off all the lights when I leave the house and when I come back I have to turn them on manually instead of them just turning on when I open the door to a room. Also my AC/heating automatically turns off when I leave the house and turns on when I come back, my lights automatically dim/change to a warmer temperature in the evening as it gets closer to bed time, my desktop goes to sleep when I leave the house, my TV automatically turns on when I power on the living room media PC, etc. etc.
Spivak 11mo ago
Two node / even node clusters can work fine. For even n>2 you define a tie breaker node in advance and only the partition connected to that node can make a quorum at 50%. For n=2 going from no quorum to quorum requires both nodes but losing a node doesn't lose quorum, and when you lose a node you stop, shoot the other node, and continue. For split brain the fastest draw wins the shootout.
- darkwater 11mo ago
  > For split brain the fastest draw wins the shootout. I bet there is still space for a race condition there.
znpy 11mo ago
In fairness to proxmox, that's the recommended way. Most homelabbers ignore recommendations because if anything breaks nothing of corporate value is lost and no one's gonna lose their job.
vsviridov 11mo ago
Originally I was planning on building the NAS with just the Minisforum MS-01, but truenas and USB enclosures do not play well together. So I went for the AOOSTAR NAS mini-pc as a "proper" solution. Ended up with two machines, so why not join them into the cluster! Probably can chuck proxmox on a RasPi somewhere, just for quorum purposes :)
holysoles 11mo ago
proxmox even makes it easy by letting you run something like a raspberry pi as an additional quorum member if you dont have enough hardware for a 3rd node

I really, really think there are better lessons there. Maybe more like "Lesson 0. Don't put distributed clusters in control of your light switches"

darkwater 11mo ago
Why not?? It's fun!
KetoManx64 11mo ago
Yes, but then I'm going to have to manually go around my house and turn off all the lights when I leave the house and when I come back I have to turn them on manually instead of them just turning on when I open the door to a room. Also my AC/heating automatically turns off when I leave the house and turns on when I come back, my lights automatically dim/change to a warmer temperature in the evening as it gets closer to bed time, my desktop goes to sleep when I leave the house, my TV automatically turns on when I power on the living room media PC, etc. etc.

Spivak 11mo ago

Two node / even node clusters can work fine. For even n>2 you define a tie breaker node in advance and only the partition connected to that node can make a quorum at 50%. For n=2 going from no quorum to quorum requires both nodes but losing a node doesn't lose quorum, and when you lose a node you stop, shoot the other node, and continue. For split brain the fastest draw wins the shootout.

darkwater 11mo ago
> For split brain the fastest draw wins the shootout. I bet there is still space for a race condition there.

znpy 11mo ago

In fairness to proxmox, that's the recommended way. Most homelabbers ignore recommendations because if anything breaks nothing of corporate value is lost and no one's gonna lose their job.

vsviridov 11mo ago

Originally I was planning on building the NAS with just the Minisforum MS-01, but truenas and USB enclosures do not play well together. So I went for the AOOSTAR NAS mini-pc as a "proper" solution. Ended up with two machines, so why not join them into the cluster! Probably can chuck proxmox on a RasPi somewhere, just for quorum purposes :)

holysoles 11mo ago

proxmox even makes it easy by letting you run something like a raspberry pi as an additional quorum member if you dont have enough hardware for a 3rd node