3 ms·

I considered deploying Talos a few weeks ago, and I ran into this: https://github.com/siderolabs/talos/issues/8367 https://github.com/siderolabs/talos/issues/8

by amluto 2y ago

I considered deploying Talos a few weeks ago, and I ran into this: https://github.com/siderolabs/talos/issues/8367 https://github.com/siderolabs/talos/issues/8367 Unless I’ve missed something, this isn’t a big deal in an AWS-style cloud where extra storage volumes (EBS, etc) have essentially no incremental cost, and maybe it’s okay on bare metal if the bare metal is explicitly designed with a completely separate boot disk (this includes Raspberry Pi using SD for boot and some other device for actual storage), but it seemed like a mostly showstopping issue for an average server that was specced with the intent to boot off a partition. I suppose one could fudge it with NVMe namespaces if the hardware cooperates. (I’ve never personally tried setting up a nontrivial namespace setup.) Has anyone set up Talos in a useful way on a server with a single disk or a single RAID array?

rstat1 2y ago
This is my main issue with it right now, I'm wasting a whole physical disk in one of my home lab machines because Talos demands control over a full disk for its uses. This is one of the more annoying problems with any "immutable" distro (or at least the ones I've tried). They demand a very specific partition layout that basically forces you to devote a whole disk to them, which as you said in a cloud environment (or any other environment where VMs are used in place of physical machines) doesn't matter, but ends up mattering a lot when using physical machines.
- sroussey 2y ago
  I’m curious on your use case. I’ve always had boot disk or boot disk array separate from any data array. Are you referring to things like a home directory?
  - amluto 2y ago
    I’m referring to a server without a dedicated boot disk. This is not especially rare. On an edge-style machine, you generally have a very limited number of M.2 slots, and dedicating one to a boot device uses up potentially 100% of your potential storage capacity, not to mention that an extra drive would be a non negligible fraction of total cost. On modern servers, there may not be a usable SATA controller, and NVMe usually costs 4 PCIe lanes. Those aren’t free. You also end up paying the absurd OEM premium for an extra disk if you go with a big name OEM. Sadly, there is no industry standard cheap, reliable boot device standard, at least as far as I’ve ever seen. Maybe someone should push USB3 for this use case — the price is certainly right, and performance is likely just fine.
    - taskforcegemini 2y ago
      is there a reason you can't just boot from usb? it sounds like the perfect match. I've built immutable bootsticks in the past that run in ram only (but with other distros). hetzner for instance lets you rent usb-sticks for their dedicated servers
      - amluto 2y ago
        FWIW, it’s been a little while since I’ve messed with the low-level details of BIOS/UEFI to kernel+united to USB root disk handoff, but I’m always slightly concerned that the system will mess up and hand off to the wrong device. For a system that uses USB for anything else, this opens up an attack/screwup vector in which the wrong disk gets used, leading to all manner of problems. Also, an external dangly thing is asking for trouble (getting dislodged). An internal device solves this. In any case, the Talos people seem to recognize this as a problem and are working on it.
        fragmede 2y ago
        fwiw an internal usb slot isn't unheard of on server-class motherboards.
    - vetinari 2y ago
      Server machines usually came with an SD-card slot on the board (and with ability to boot from it). This was often used for ESXi-like environments, which also required dedicated device for themselves. There are also SATA-DOMs or USB-DOMs. You could use one of these modules with your machine.
vbezhenar 2y ago
Just use PXE? Why use disk at all?
- robcohen 2y ago
  PXE seems like kind of a nightmare to manage. Seems better to just use Cloudinit or Ignition with proper Certificate management
  - fmajid 2y ago
    With NetBoot.xyz, it’s just a DHCP server setting, setting up a TFTP server and very easy to set up. Much easier than dealing with a bunch of USB boot drives and keeping them up to date.
    - vetinari 2y ago
      Machines with UEFI support http boot since around ~2017. Then you can forget special vulnerable server (tftp) and just use plain webserver, together with dns (and have it in different subnet, if necessary).
lknuth 2y ago
I've leaned into it: all application state is on ephemeral storage and its constantly replicated "off-site" to a NAS running minio. To accomplish this, I have restricted myself to SQLite as storage and use Litestream for replication. On start, Litestream reconstructs the last known state before the application starts. [Source](https://github.com/LukasKnuth/homeserver/blob/912cbc0111e44d044e5c7e2bbac5a8473df089cd/deploy/modules/stateful_web_app/main.tf#L73 https://github.com/LukasKnuth/homeserver/blob/912cbc0111e44d...) It works very well for my workloads (user interaction driven web apps) but there are theoretical situations in which data loss can occurr.
- amluto 2y ago
  You need to store data somewhere. If you’re willing to have that location be off-site and to pay for remote storage and possibly egress, and you’re okay with your local installation not coming up if the off-site storage is inaccessible and with the latency involved in bringing your on-site data back, fine. Of course, if you really lean in to a complete lack of local persistent state and you configure your network or some other critical service like this, good luck recovering from an upgrade and complete loss of ephemeral state.
  - lknuth 2y ago
    I'm not sure I understand your second point. Yes, if the external replication target is unavailable, I can't bring my local service back online. Same goes for if the replication target becomes unavailable and I don't realize it, there is potential for a lot of data to be lost if the application restarts. For my personal usecase this is fine. I also have monitoring setup to look for just this case. It's a tradeoff between resilience and simplicity that works for some use cases - mine included.
    - amluto 2y ago
      My second point is that, in a setup of any complexity, a black start is nontrivial. You have a network, with routes, DNS config, maybe VLANs, maybe a whole SDN. If that’s down because the machines running it are trying to pull their own configuration over the network, it won’t come back up. You can get pretty far into the weeds with situations like this. Facebook supposedly got locked out of their own datacenter due to a network outage preventing the access control system from accessing whatever service it needed to allow anyone to open the door.
INTPenis 2y ago
This is worrying because I am just in the process of migrating old CentOS clusters to Talos and we need one additional disk other than the system disk. It's used for the host based ceph cluster. But if I read this correctly there shouldn't be an issue adding one blank disk to the Talos VM, the issue is only more granular disk and partitions management.
splix 2y ago
I'd suggest to run it on top a virtualization environment, like Proxmox. That solves a lot of problems, and not just related to Talos. Basically, you split k8s and the resource management, networking, disks, etc, as well as getting backup, migration, etc.

by amluto 2y ago

rstat1 2y ago

This is my main issue with it right now, I'm wasting a whole physical disk in one of my home lab machines because Talos demands control over a full disk for its uses. This is one of the more annoying problems with any "immutable" distro (or at least the ones I've tried). They demand a very specific partition layout that basically forces you to devote a whole disk to them, which as you said in a cloud environment (or any other environment where VMs are used in place of physical machines) doesn't matter, but ends up mattering a lot when using physical machines.

sroussey 2y ago
I’m curious on your use case. I’ve always had boot disk or boot disk array separate from any data array. Are you referring to things like a home directory?
- amluto 2y ago
  I’m referring to a server without a dedicated boot disk. This is not especially rare. On an edge-style machine, you generally have a very limited number of M.2 slots, and dedicating one to a boot device uses up potentially 100% of your potential storage capacity, not to mention that an extra drive would be a non negligible fraction of total cost. On modern servers, there may not be a usable SATA controller, and NVMe usually costs 4 PCIe lanes. Those aren’t free. You also end up paying the absurd OEM premium for an extra disk if you go with a big name OEM. Sadly, there is no industry standard cheap, reliable boot device standard, at least as far as I’ve ever seen. Maybe someone should push USB3 for this use case — the price is certainly right, and performance is likely just fine.
  - taskforcegemini 2y ago
    is there a reason you can't just boot from usb? it sounds like the perfect match. I've built immutable bootsticks in the past that run in ram only (but with other distros). hetzner for instance lets you rent usb-sticks for their dedicated servers
    - amluto 2y ago
      FWIW, it’s been a little while since I’ve messed with the low-level details of BIOS/UEFI to kernel+united to USB root disk handoff, but I’m always slightly concerned that the system will mess up and hand off to the wrong device. For a system that uses USB for anything else, this opens up an attack/screwup vector in which the wrong disk gets used, leading to all manner of problems. Also, an external dangly thing is asking for trouble (getting dislodged). An internal device solves this. In any case, the Talos people seem to recognize this as a problem and are working on it.
      - fragmede 2y ago
        fwiw an internal usb slot isn't unheard of on server-class motherboards.
  - vetinari 2y ago
    Server machines usually came with an SD-card slot on the board (and with ability to boot from it). This was often used for ESXi-like environments, which also required dedicated device for themselves. There are also SATA-DOMs or USB-DOMs. You could use one of these modules with your machine.

vbezhenar 2y ago

Just use PXE? Why use disk at all?

robcohen 2y ago
PXE seems like kind of a nightmare to manage. Seems better to just use Cloudinit or Ignition with proper Certificate management
- fmajid 2y ago
  With NetBoot.xyz, it’s just a DHCP server setting, setting up a TFTP server and very easy to set up. Much easier than dealing with a bunch of USB boot drives and keeping them up to date.
  - vetinari 2y ago
    Machines with UEFI support http boot since around ~2017. Then you can forget special vulnerable server (tftp) and just use plain webserver, together with dns (and have it in different subnet, if necessary).

lknuth 2y ago

I've leaned into it: all application state is on ephemeral storage and its constantly replicated "off-site" to a NAS running minio. To accomplish this, I have restricted myself to SQLite as storage and use Litestream for replication. On start, Litestream reconstructs the last known state before the application starts. [Source](https://github.com/LukasKnuth/homeserver/blob/912cbc0111e44d044e5c7e2bbac5a8473df089cd/deploy/modules/stateful_web_app/main.tf#L73 https://github.com/LukasKnuth/homeserver/blob/912cbc0111e44d...) It works very well for my workloads (user interaction driven web apps) but there are theoretical situations in which data loss can occurr.

amluto 2y ago
You need to store data somewhere. If you’re willing to have that location be off-site and to pay for remote storage and possibly egress, and you’re okay with your local installation not coming up if the off-site storage is inaccessible and with the latency involved in bringing your on-site data back, fine. Of course, if you really lean in to a complete lack of local persistent state and you configure your network or some other critical service like this, good luck recovering from an upgrade and complete loss of ephemeral state.
- lknuth 2y ago
  I'm not sure I understand your second point. Yes, if the external replication target is unavailable, I can't bring my local service back online. Same goes for if the replication target becomes unavailable and I don't realize it, there is potential for a lot of data to be lost if the application restarts. For my personal usecase this is fine. I also have monitoring setup to look for just this case. It's a tradeoff between resilience and simplicity that works for some use cases - mine included.
  - amluto 2y ago
    My second point is that, in a setup of any complexity, a black start is nontrivial. You have a network, with routes, DNS config, maybe VLANs, maybe a whole SDN. If that’s down because the machines running it are trying to pull their own configuration over the network, it won’t come back up. You can get pretty far into the weeds with situations like this. Facebook supposedly got locked out of their own datacenter due to a network outage preventing the access control system from accessing whatever service it needed to allow anyone to open the door.

INTPenis 2y ago

This is worrying because I am just in the process of migrating old CentOS clusters to Talos and we need one additional disk other than the system disk. It's used for the host based ceph cluster. But if I read this correctly there shouldn't be an issue adding one blank disk to the Talos VM, the issue is only more granular disk and partitions management.

splix 2y ago

I'd suggest to run it on top a virtualization environment, like Proxmox. That solves a lot of problems, and not just related to Talos. Basically, you split k8s and the resource management, networking, disks, etc, as well as getting backup, migration, etc.