We have a couple of 4U8N Supermicro "fat twins" hooked to a Miktrotik CRS17 (10 GbE). Each node has 6 drive bays – to simplify things, we have installed similar

We have a couple of 4U8N Supermicro "fat twins" hooked to a Miktrotik CRS17 (10 GbE). Each node has 6 drive bays – to simplify things, we have installed similar capacity drives and use ZFS raidz2. 7 nodes out of 8 are production, and the last node is staging. Each node runs 2x older v4 Xeons (to a total of 64 virtual cores per node), with 256G RAM. This cost us about US$8k upfront per server, the chassis were decommissioned 2nd hand (looked nice), 2nd hand CPUs, and everything else new. We pay around US$2k for unmetered 30M colocation in Shanghai. In EU/US it must be cheaper and faster. We provisioned the servers by hand, just installing Proxmox on each of them by hand. Proxmox is nice because it has ZFS root support out of the box without any magic. All other Proxmox services are disabled. After that we have installed K3s on it via Ansible, with each node being both master and worker. There's no need for separation unless you have some really write heavy workloads that will throttle etcd and cause split brain. On top of K3s we have three types of workloads: - stateless workloads, or workloads with volumes from ConfigMaps or Secrets, - stateful highly available workloads where HA is managed by application (Postgres, Redis...), - stateful single unit workloads. For HA workloads we use ZFS directly via openebs-zfs CSI driver, this ties workload to nodes, but since HA is managed by application, and we do minimum 5 or 7 nodes, we can usually easily turn a node off for maintenance and reseed the workload on another node. Most Helm charts will have this functionality built-in (e.g. Postgres has Patroni). For stateful non-HA workloads (e.g. our knowledgebase) we use Longhorn on top of zvol formatted in ext4. Longhorn gives us nice backups to any S3-compatible storage, plus durability. We usually go with 5 replicas, or with 3 for less important workloads. It worked nicely like this for a few years and survived several minor faults. Failing node is drained and replaced without any drama. We mostly use Helm charts for applications, Bitnami for Redis and other things, TimescaleDB's chart for Postgres, own charts for our applications. Before rolling out to production we test on the staging machines. Of course, it's not the same, but close enough. For dev we use k3d, which is multi-node K3s on top of Docker.