4 ms·

It’s been through quite a few iterations, but the current setup is a 35U Sysracks (not filled, I just like the space) with: * 3x Dell R620 running K3os on cl

It’s been through quite a few iterations, but the current setup is a 35U Sysracks (not filled, I just like the space) with: * 3x Dell R620 running K3os on clustered Proxmox * 1x X11 Supermicro 2U with ZFS handling the disk array * 1x X9 Supermicro 2U, also with ZFS, that is a backup target for the other one (WoL to ingest, then shutdown) * UniFi Dream Machine Pro * UniFi 24 port PoE 1G switch * 2x UniFi UAP-AC-PRO APs (not in the cabinet, obvs) * APC UPS with battery extension The Dells have Samsung PM863 3.84 TB drives for Ceph, which Proxmox handles. They have some ancient 2.5” enterprise pull SSDs for boot. The Supermicros have some cheap ADATA NVMe SSDs for boot (the X9 has a modded BIOS to make this possible). In general I like it, but the current in-progress work is rewriting all of the Ansible playbook that generates QEMU images for Proxmox, and setting up PXE so I can grab IPs on boot for new images. The latter will enable me to shift off of K3os in favor of Talos. Technically I can do so now, but it requires manual input to set the IP addresses. Beyond that, I want to get a 10G switch so a. RBD-backed devices are faster b. ZFS send/recv is faster c. Just because.

justinclift 3y ago
With your clustered Ceph Promox, does Ceph there provide the storage to your VM's running in Promox? And if so, how resilient are the running VMs to the (temporary) loss of a Promox node? Asking just because I'm trying out some clustered Ceph Promox stuff currently, and with 3 nodes (so far) the VMs aren't coping well the loss of a node. They seem to either freeze, fault or hard reset (after a while) when it occurs. I'm going to try with 5 nodes next and see how it goes. Am new to both Ceph and Proxmox, so it could also be something to do with my current newbie-ness. :)
- sgarland 3y ago
  At the moment, Ceph is a tangential storage solution only used for testing (the bulk of my pods use Longhorn), but yes, that’s what it does – Proxmox sets up Ceph on the NVMe drives on its nodes, and then exposes the pool over RBD to VMs when you add a disk. In that regard, it’s very much like EBS from the perspective of the VM. Nothing stops me from shifting to Ceph for existing workloads, but I have it in my head that I need to shift the K8s hosting to Talos first. I have used it successfully to run MySQL and Postgres instances FWIW. Never had any issues with either of those, even with abrupt VM shutdowns. A VM, if set up as HA, can be live-migrated to a different node to perform maintenance with zero downtime. If the node just goes away, of course, you can’t do much about that. I confess I haven’t used Ceph heavily while dropping a node, so unfortunately I can’t give you a good answer on that behavior, other than ensuring that you have adequate bandwidth for both Ceph and Proxmox. I know they both say 10G networks are preferable. FWIW I have a completely separate switch (UniFi Flex Mini) for Proxmox’s corosync; in theory the main switch would be fine, but for an extra $30 I had more peace of mind.
  - justinclift 3y ago
    Thanks, that's all good info. Yeah, the live migration bit is a useful looking feature. I'm just investigating ways to reduce single points of failure. :) Live migration is working fine with storage off-the-cluster. eg local iSCSI server sharing storage to the nodes. But then the iSCSI server is the failure point. Almost done setting up the 5 node cluster, so will test that shortly. Then probably need to read a bunch more docs. ;D
    - justinclift 3y ago
      The 5 node cluster with Ceph on all the nodes is working "ok" (resilience wise) to loosing a node. The VMs keep on going. Unfortunately, it's not a complete win though. Doing a live migration of a VM between nodes actually hung (locking the VM completely), and needed the VM to be restarted then the migration re-run (which then worked). The logs for the failed migration were completely useless. :( So, a 5 node setup seems like it might be a better way forward, but things like migration hangs need eliminating first. More experimentation and learning coming up. :)
      - sgarland 3y ago
        Do all of your nodes have identical CPUs (or if not, a generic CPU selected to be virtualized)? That’s the main thing off the top of my head that can cause a freeze.
        justinclift 3y ago
        Yeah, the whole setup (for now) is just running virtualised in my desktop (Ryzen 5950x, 64GB ram). So each node has 6x cores and 8GB ram, with 2 network interfaces (1 public, 1 for ceph cluster traffic). After I have more experience I'll look to replicate it in hardware, but starting with a virtualised setup lets me test different configurations easily. :)
        justinclift 3y ago
jawmes8 3y ago
Ansible caught my interest! I setup a dev server and have been wondering about using Ansible to install my development dependencies on a new OS
- sgarland 3y ago
  You’re welcome to take a look [0] at what I use now; it’s designed to be used with Packer to build QEMU images for Proxmox. Also it’s not really set up how I now prefer to do things, hence the total rewrite that’s underway. When that’s done, I’ll archive this repo with a link to the new one. [0]: https://github.com/stephanGarland/packer-proxmox-templates https://github.com/stephanGarland/packer-proxmox-templates

justinclift 3y ago

With your clustered Ceph Promox, does Ceph there provide the storage to your VM's running in Promox? And if so, how resilient are the running VMs to the (temporary) loss of a Promox node? Asking just because I'm trying out some clustered Ceph Promox stuff currently, and with 3 nodes (so far) the VMs aren't coping well the loss of a node. They seem to either freeze, fault or hard reset (after a while) when it occurs. I'm going to try with 5 nodes next and see how it goes. Am new to both Ceph and Proxmox, so it could also be something to do with my current newbie-ness. :)

sgarland 3y ago
At the moment, Ceph is a tangential storage solution only used for testing (the bulk of my pods use Longhorn), but yes, that’s what it does – Proxmox sets up Ceph on the NVMe drives on its nodes, and then exposes the pool over RBD to VMs when you add a disk. In that regard, it’s very much like EBS from the perspective of the VM. Nothing stops me from shifting to Ceph for existing workloads, but I have it in my head that I need to shift the K8s hosting to Talos first. I have used it successfully to run MySQL and Postgres instances FWIW. Never had any issues with either of those, even with abrupt VM shutdowns. A VM, if set up as HA, can be live-migrated to a different node to perform maintenance with zero downtime. If the node just goes away, of course, you can’t do much about that. I confess I haven’t used Ceph heavily while dropping a node, so unfortunately I can’t give you a good answer on that behavior, other than ensuring that you have adequate bandwidth for both Ceph and Proxmox. I know they both say 10G networks are preferable. FWIW I have a completely separate switch (UniFi Flex Mini) for Proxmox’s corosync; in theory the main switch would be fine, but for an extra $30 I had more peace of mind.
- justinclift 3y ago
  Thanks, that's all good info. Yeah, the live migration bit is a useful looking feature. I'm just investigating ways to reduce single points of failure. :) Live migration is working fine with storage off-the-cluster. eg local iSCSI server sharing storage to the nodes. But then the iSCSI server is the failure point. Almost done setting up the 5 node cluster, so will test that shortly. Then probably need to read a bunch more docs. ;D
  - justinclift 3y ago
    The 5 node cluster with Ceph on all the nodes is working "ok" (resilience wise) to loosing a node. The VMs keep on going. Unfortunately, it's not a complete win though. Doing a live migration of a VM between nodes actually hung (locking the VM completely), and needed the VM to be restarted then the migration re-run (which then worked). The logs for the failed migration were completely useless. :( So, a 5 node setup seems like it might be a better way forward, but things like migration hangs need eliminating first. More experimentation and learning coming up. :)
    - sgarland 3y ago
      Do all of your nodes have identical CPUs (or if not, a generic CPU selected to be virtualized)? That’s the main thing off the top of my head that can cause a freeze.
      - justinclift 3y ago
        Yeah, the whole setup (for now) is just running virtualised in my desktop (Ryzen 5950x, 64GB ram). So each node has 6x cores and 8GB ram, with 2 network interfaces (1 public, 1 for ceph cluster traffic). After I have more experience I'll look to replicate it in hardware, but starting with a virtualised setup lets me test different configurations easily. :)
        justinclift 3y ago

jawmes8 3y ago

Ansible caught my interest! I setup a dev server and have been wondering about using Ansible to install my development dependencies on a new OS

sgarland 3y ago
You’re welcome to take a look [0] at what I use now; it’s designed to be used with Packer to build QEMU images for Proxmox. Also it’s not really set up how I now prefer to do things, hence the total rewrite that’s underway. When that’s done, I’ll archive this repo with a link to the new one. [0]: https://github.com/stephanGarland/packer-proxmox-templates https://github.com/stephanGarland/packer-proxmox-templates