8 ms·

I work in a different industry and am responsible for maintaining a fleet of bare metal OSes (we currently use Ubuntu). Bare metal management really feels like

I work in a different industry and am responsible for maintaining a fleet of bare metal OSes (we currently use Ubuntu). Bare metal management really feels like an unsolved problem. Whilst everybody working with cloud environments is whisked away by the latest shiny tools like Docker and Ansible, those of us working with bare metal are still trying to find a way to keep machines up and running with an OS that doesn't get corrupted from unexpected poweroffs or permanently cut itself off from the network because of a bad config. The only existing candidate I've seen is Balena, but it only supports specific hardware and the cost is probably so high that we wouldn't be making a profit if we went with it. At my current employer we are building a custom flavor of Ubuntu and provision it with Puppet, but we still get failures, and it's far from the immutable haven that DevOps guys would be used to.

renewiltord 6y ago
Bare metal for a reason? We found benefits from virtualizing on the same bare metal infrastructure and then running on VMs. Used VMWare for everything. However, the Docker-based strategy is far ahead.
- flemhans 6y ago
  You'd still need to bootstrap the metal with a VM/container environment, anyway.
nullify88 6y ago
I'm running a few bare metal Fedora CoreOS nodes and its been a dream with IPMI, ignition files, containers, and ansible to orchestrate it all. Havent used Flatcar Linux but I have seen the experience is similar.
- dgellow 6y ago
  A bit confused here. Isn't CoreOS deprecated since ~May 2020?
  - nullify88 6y ago
    I understand your confusion. CoreOS has indeed since been deprecated, however when Redhat acquired CoreOS in Jan 2018 they merged CoreOS and Project Atomic to create Redhat CoreOS. They also released Fedora CoreOS which shares some technologies with vanilla Fedora and Silverblue. Shortly after the acquisition, Flatcar Container Linux was released which is an updated derivative of the original CoreOS.
    - dgellow 6y ago
      Wow, thank you for this, I had no idea. What a mess made by this acquisition, reusing the name of a deprecated product isn't really a great idea. Though I'm glad to know there are ways to go back to a CoreOS-like experience, that has been my favorite server setup for a while. Thanks again!
      - joshuak 6y ago
        Agreed, the original CoreOS is by far my favorite server OS so far. I was very sad to see it get sold to RedHat, though few other people shared my concern at the time[0]. I wish I could say my concern turned out to be unjustified, but alas here we are. I do recommend Flatcar linux though, and strongly encourage everyone to check it out. Not as jazzed about the name, but what’s in a name? [0] https://news.ycombinator.com/item?id=16270382 https://news.ycombinator.com/item?id=16270382
AlotOfReading 6y ago
Perhaps I'm misunderstanding, but power failure resilience is a system design property. You shouldn't be trying to solve it at the orchestration/provisioning level. Add some caps, have durable memory, change FS, avoid writes to OS partitions, do atomic updates, etc. However, you might find QBee relevant for the other stuff.
imwillofficial 6y ago
Check out Cockpit by the red hat guys. I think this might be useful for you.
- ohyeshedid 6y ago
  It's been some time since I've happened across Cockpit on Ubuntu, but previous experience showed it was several major updates behind and lacked a lot of the functionality you see in the screenshots.
- c7DJTLrn 6y ago
  Appreciate the suggestion but this looks like a provisioning tool. Puppet is more or less good enough, it's the OS and its mutability which is our problem.
  - imwillofficial 6y ago
    Maybe I read it wrong, I thought you were looking for fleet a management for bare metal. Anyone have any good suggestions of cockpit doesn’t fit the bill? https://www.redhat.com/sysadmin/intro-cockpit https://www.redhat.com/sysadmin/intro-cockpit
gmfawcett 6y ago
Have you played with NixOS? The initial learning curve is steep, but the payoffs are pretty nice.
- c7DJTLrn 6y ago
  I've had my eye on it but haven't played with it yet. From a bird's eye view and a discussion with a friend, it doesn't sound ready for production, but I'm not ruling it out. I think it would be a battle to convince my colleagues and managers to try NixOS precisely because of the learning curve and lack of experts in the hiring pool out there.
mos_6502 6y ago
Not that I’m a Canonical stan, or anything. But have you looked at MAAS [1]? It works decently well in my small-scale lab testing. 1. https://maas.io https://maas.io
- c7DJTLrn 6y ago
  Correct me if I'm wrong, but isn't this just a tool to install a chosen OS (for example Ubuntu)?
  - galgalesh 6y ago
    It's what I think of when I hear "bare metal management". It manages a fleet of physical servers and provides an API to provision servers, configure networks, and install operating systems. Basically an easier alternative to OpenStack for bare metal servers. It supports ipmi and a bunch of other BMC's, uses pxe to deploy the operating systems, optionally runs its own DNS and DHCP servers and integrated with stuff like SMART to verify server health before deploying operating systems. I use it in my research group to manage a fleet of about two dozen servers.
  - andrewjf 6y ago
    I mean yes... but in bare metal there's a lot of things that need to be handled: - Network topology management, IP address management (IPAM) and host associations (DNS, DHCP services or Static IP assignment) - IPMI to power on/off machine and change boot orders, serial consoles - Disk configuration (MD raid sets, LVM configuration, partitioning) - Firmware of all the components - Burn-in testing - Passing configuration to the host OS for application/OS configuration - Detecting hardware failures and tracking response. - Install the OS MaaS does most of these things, but it's all the life of bare metal. So "just a tool to install a chosen OS" undersells the complexity in this space.
    - busterarm 6y ago
      https://github.com/Roblox/terraform-provider-maas https://github.com/Roblox/terraform-provider-maas There's even a terraform provider in development.
srswtf123 6y ago
Have you checked out Digital Rebar? I’d be interested in hearing anyone’s experiences with it. https://rackn.com/rebar/ https://rackn.com/rebar/
- jmedefind 6y ago
  We use it for our bare metal management and like it for the most part. It does require a lot of planning though. But the company has been great to work with and super helpful in slack.
eointierney 6y ago
More and more guix or nixos seem like good practical choices for these kinds of use cases. I prefer guix as it's a little less finicky (though https://gitlab.com/nonguix/nonguix https://gitlab.com/nonguix/nonguix is probably required for all non-purists such as this lowly worm). Ubuntu OS is now stinky doodoo, which is a shame as it used to be the cat's pajamas for ease of use. Snapd is a debacle.
- c7DJTLrn 6y ago
  Snap definitely doesn't belong on servers - our flavor of Ubuntu doesn't have it. If you bootstrap from Ubuntu Base you can cut out a lot of that crap.
- spindle 6y ago
  I totally agree except that I prefer NixOS because it supports ZFS so well.
chousuke 6y ago
Out of curiosity, what sorts of issues are you having? I don't have much experience specifically with Ubuntu on bare metal, but I find kickstart easier to understand and use compared to preseed, for consistent bare-metal installations. I think there's an OpenStack project for provisioning bare metal servers via an API from images; I wonder how that's doing nowadays. Puppet is pretty good for configuration management when your systems do actually require the occasional change instead of just being continuously redeployed. Maybe there are people who rebuild their database servers every hour via CI, but I'd rather not. "Immutable infra" is definitely not the default state of things in DevOps land either. Often people talking about how their infra is immutable just conveniently ignore the parts that aren't. The data has to live somewhere.
hnlmorg 6y ago
> those of us working with bare metal are still trying to find a way to keep machines up and running with an OS that doesn't get corrupted from unexpected poweroffs or permanently cut itself off from the network because of a bad config. I'm guessing there's more to this story than you've summarised because those points are pretty easily solved with: - UPS (if the power outs are that much of a problem then you might need to invest in a generator as well). - iLo / IPMI (remote management). Though even just running a serial cable out the back of the server is good enough for a remote console in the event of a network failure. As for managing the config of them, the usual tools like Ansible and Puppet work just as well (in some cases actually better since they were initially designed for on-prem hardware). Likewise for Docker. So don't think you can't run those tools on bare metal Linux. But if you don't want the containerisation-like aspects of Docker but still wanted the deployment tools then you can go a long way with git and shell scripts. While DevOps really came into popularity with cloud hosting, there's nothing fundamentally new about a lot of the tooling that wasn't possible in the old days of bare metal UNIX and Linux. Us older sysadmins were still doing a lot of the same stuff back then too, we just didn't given it trendy names back then. 10+ years ago I was installing Linux remotely on bare metal hardware via a console server. Then once I had one machine installed I'd take one HDD out of the mirrored RAID array and plug it in the 2nd server so the RAID controller would clone. I could repeat the disk swapping as many times as I wanted as long as I remembered to change the IPs (again, via remote serial console). And this was the lazy way of deploying a fleet of servers quickly. More professional places would push the install onto the server via netboot cloning solutions.
- c7DJTLrn 6y ago
  Yes, there's more to it sadly. The hardware isn't in our control and UPS are out of the question. The machines aren't on premises or in a datacentre. Puppet works okayish but when you have a large fleet of what are basically IoT devices, you start getting an unpleasant failure rate. Push one bad network config and you've 'bricked' thousands of machines. I am being a bit vague intentionally, hope you can understand. But to help imagine, the scale of our problem is pretty big, you've probably even had some interaction with one of our machines.
  - 6y ago
coddle-hark 6y ago
Basically, you want read-only systems with separate partitions for persistent data. Have two images to boot from and overwrite just one of them when you upgrade. This is how Balena does it and it’s a pretty common setup for IoT stuff.
- CameronNemo 6y ago
  http://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format http://www.chromium.org/chromium-os/chromiumos-design-docs/d...
- hedora 6y ago
  It’s also not difficult to implement from scratch. Pretty much any linux distro with a bootable USB installer already supports it out of the box.
gjsman-1000 6y ago
Have you tried Fedora Silverblue? It uses OSTree technology to offer many of the features of Ubuntu Core including automatic updating, upgrades that work regardless of power loss, Flatpak confinement for all apps (without any arbitrary restrictions like Snaps), and it's free. "Silverblue is a variant of Fedora Workstation. It looks, feels and behaves like a regular desktop operating system, and the experience is similar to what you find with using a standard Fedora Workstation. However, unlike other operating systems, Silverblue is immutable. This means that every installation is identical to every other installation of the same version. The operating system that is on disk is exactly the same from one machine to the next, and it never changes as it is used. Silverblue’s immutable design is intended to make it more stable, less prone to bugs, and easier to test and develop. Finally, Silverblue’s immutable design also makes it an excellent platform for containerized applications as well as container-based software development. In each case, applications (apps) and containers are kept separate from the host system, improving stability and reliability. Silverblue’s core technologies have some other helpful features. OS updates are fast and there’s no waiting around for them to install: just reboot as normal to start using the next version. With Silverblue, it is also possible to roll back to the previous version of the operating system, if something goes wrong." https://silverblue.fedoraproject.org https://silverblue.fedoraproject.org
- dineshdb 6y ago
  Fedora CoreOS might be better in this regard since that is designed for server use case. Silverblue took that technology and applied to the desktop. Silverblue user here. Haven't tried CoreOS myself.
  - curt15 6y ago
    What about Fedora IoT?
    - dineshdb 6y ago
      That seems to be geared towards edge servers and IoT use cases. Otherwise, all of them use the same technologies. Also one thing to note is that CoreOS as well as IoT is bootstrappable with a configuration file via ignition(a tool for automation).
swiley 6y ago
The original author of busybox had a really good idea for this. I don't remember exactly but I think he partitioned the bootdisk in half amd had the bootloader fail a partition (and not boot it) if it didn't boot all the way. Each partition would apply OTA updates depending on which was active.
clcaev 6y ago
Have you considered FreeBSD? Many NAS devices use it, and those are sometimes updated remotely, etc. There's less churn. Keep most of your work in jails, which have been production quality since 10 years ago.
zmix 6y ago
Is there any specific reason not to use something like SmartOS?
sneak 6y ago
CoreOS was a good attempt in this direction, but seems to have died. I am also a fan of the ChromeOS partition layout, which is solving the same problem.
- dgellow 6y ago
  I still miss CoreOS, it was a really good solution. Edit: as said in another comment in this thread, there are now alternatives to CoreOS! - Fedora CoreOS: https://docs.fedoraproject.org/en-US/fedora-coreos/ https://docs.fedoraproject.org/en-US/fedora-coreos/ - Flatcar Container Linux: https://kinvolk.io/flatcar-container-linux/ https://kinvolk.io/flatcar-container-linux/
elcritch 6y ago
> Bare metal management really feels like an unsolved problem. Whilst everybody working with cloud environments is whisked away by the latest shiny tools like Docker and Ansible, those of us working with bare metal are still trying to find a way to keep machines up and running with an OS that doesn't get corrupted from unexpected poweroffs or permanently cut itself off from the network because of a bad config. It is and isn't solved. It usually takes a lot of work or custom scripts. One of the best is the Nerves Project, and is what I use for IoT deploymens [0] or even simple cloud deployments. Nerves is setup to run Elixir/Erlang, but it's really just a wrapper around buildroot and Elixir can start programs in any language desired with some work. One of the core authors wrote a tool called `fwup` for doing immutable updates on Linux [1]. The ability to do an A/B update and have the device do an automatic rollback if an update fails is crucial. A year or so ago they changed the default boot process to still allow networking and remote connections to work even if the main application crashes. Surprisingly it's all done using Erlang tooling, AFAICT. There are still rough edges, like limited ipv6 support. You can still get devices failing from dead SD cards -- even if your system boots from a read-only partition as power outages during a write on any partition can effectively destroy the SD card, so skip the SD cards. 0: https://www.nerves-project.org/ https://www.nerves-project.org/ 1: https://github.com/fwup-home/fwup https://github.com/fwup-home/fwup
mikepurvis 6y ago
I agree that it's completely unsolved. I work in mobile robots and it's just maddening the number of solutions which either won't work if you're headless or won't work if you'll need kernel upgrades, or won't work if you'll need seamless rollbacks, or won't work if you'll want OTA updates or won't work if you'll need to be able to reconfigure the network in-band (not by exiting your app for some outside management GUI). We ended up with a mostly hand rolled system based on kexec, grub chainloading, and deploying rootfs images as big tarballs. It works, but I really wish there was something we could have just taken off the shelf.
rjzzleep 6y ago
There have been various different setups here. Have you looked at Matchbox? > matchbox is a service that matches bare-metal machines to profiles that PXE boot and provision clusters. Machines are matched by labels like MAC or UUID during PXE and profiles specify a kernel/initrd, iPXE config, and Ignition config. https://github.com/poseidon/matchbox https://github.com/poseidon/matchbox
cik 6y ago
I did this for a few years with cobbler (https://cobbler.github.io/ https://cobbler.github.io/). cobbler, pxe, bootp, tftp, ansible and friends pretty much solve this problem. In fact, if you know the mac addresses (or ranges) you can fairly easily designate groups of machines, roles, and the like. Years ago I did this to "rapidly" provision a couple of thousand machines we bought for the stock exchange. You can do most of your testing locally in vagrants, even simulating the networks you need to provision. You can go a step further and trigger api updates at the end of your ansible runs so that cobbler updates collins (https://tumblr.github.io/collins/ https://tumblr.github.io/collins/) so that you can track inventorying, cm changes and the like. At one point in time we reimaged entire subnets when they lost power - and ran tests against the hardware to ensure the machines functioned post-provisioning. As far as I know the boursa still uses this system.
necovek 6y ago
You might also look at Canonical MAAS (maas.io) which will get you some of the way there too. Though, that's more of the traditional server management approach, if properly set up, you can have it worry about the network configuration and pushing it to clients. To me, it beats the entire IoT pull setup that Canonical is pushing with Ubuntu Core and snaps.

renewiltord 6y ago

Bare metal for a reason? We found benefits from virtualizing on the same bare metal infrastructure and then running on VMs. Used VMWare for everything. However, the Docker-based strategy is far ahead.

flemhans 6y ago
You'd still need to bootstrap the metal with a VM/container environment, anyway.

nullify88 6y ago

I'm running a few bare metal Fedora CoreOS nodes and its been a dream with IPMI, ignition files, containers, and ansible to orchestrate it all. Havent used Flatcar Linux but I have seen the experience is similar.

dgellow 6y ago
A bit confused here. Isn't CoreOS deprecated since ~May 2020?
- nullify88 6y ago
  I understand your confusion. CoreOS has indeed since been deprecated, however when Redhat acquired CoreOS in Jan 2018 they merged CoreOS and Project Atomic to create Redhat CoreOS. They also released Fedora CoreOS which shares some technologies with vanilla Fedora and Silverblue. Shortly after the acquisition, Flatcar Container Linux was released which is an updated derivative of the original CoreOS.
  - dgellow 6y ago
    Wow, thank you for this, I had no idea. What a mess made by this acquisition, reusing the name of a deprecated product isn't really a great idea. Though I'm glad to know there are ways to go back to a CoreOS-like experience, that has been my favorite server setup for a while. Thanks again!
    - joshuak 6y ago
      Agreed, the original CoreOS is by far my favorite server OS so far. I was very sad to see it get sold to RedHat, though few other people shared my concern at the time[0]. I wish I could say my concern turned out to be unjustified, but alas here we are. I do recommend Flatcar linux though, and strongly encourage everyone to check it out. Not as jazzed about the name, but what’s in a name? [0] https://news.ycombinator.com/item?id=16270382 https://news.ycombinator.com/item?id=16270382

AlotOfReading 6y ago

Perhaps I'm misunderstanding, but power failure resilience is a system design property. You shouldn't be trying to solve it at the orchestration/provisioning level. Add some caps, have durable memory, change FS, avoid writes to OS partitions, do atomic updates, etc. However, you might find QBee relevant for the other stuff.

imwillofficial 6y ago

Check out Cockpit by the red hat guys. I think this might be useful for you.

ohyeshedid 6y ago
It's been some time since I've happened across Cockpit on Ubuntu, but previous experience showed it was several major updates behind and lacked a lot of the functionality you see in the screenshots.
c7DJTLrn 6y ago
Appreciate the suggestion but this looks like a provisioning tool. Puppet is more or less good enough, it's the OS and its mutability which is our problem.
- imwillofficial 6y ago
  Maybe I read it wrong, I thought you were looking for fleet a management for bare metal. Anyone have any good suggestions of cockpit doesn’t fit the bill? https://www.redhat.com/sysadmin/intro-cockpit https://www.redhat.com/sysadmin/intro-cockpit

gmfawcett 6y ago

Have you played with NixOS? The initial learning curve is steep, but the payoffs are pretty nice.

c7DJTLrn 6y ago
I've had my eye on it but haven't played with it yet. From a bird's eye view and a discussion with a friend, it doesn't sound ready for production, but I'm not ruling it out. I think it would be a battle to convince my colleagues and managers to try NixOS precisely because of the learning curve and lack of experts in the hiring pool out there.

mos_6502 6y ago

Not that I’m a Canonical stan, or anything. But have you looked at MAAS [1]? It works decently well in my small-scale lab testing. 1. https://maas.io https://maas.io

c7DJTLrn 6y ago
Correct me if I'm wrong, but isn't this just a tool to install a chosen OS (for example Ubuntu)?
- galgalesh 6y ago
  It's what I think of when I hear "bare metal management". It manages a fleet of physical servers and provides an API to provision servers, configure networks, and install operating systems. Basically an easier alternative to OpenStack for bare metal servers. It supports ipmi and a bunch of other BMC's, uses pxe to deploy the operating systems, optionally runs its own DNS and DHCP servers and integrated with stuff like SMART to verify server health before deploying operating systems. I use it in my research group to manage a fleet of about two dozen servers.
- andrewjf 6y ago
  I mean yes... but in bare metal there's a lot of things that need to be handled: - Network topology management, IP address management (IPAM) and host associations (DNS, DHCP services or Static IP assignment) - IPMI to power on/off machine and change boot orders, serial consoles - Disk configuration (MD raid sets, LVM configuration, partitioning) - Firmware of all the components - Burn-in testing - Passing configuration to the host OS for application/OS configuration - Detecting hardware failures and tracking response. - Install the OS MaaS does most of these things, but it's all the life of bare metal. So "just a tool to install a chosen OS" undersells the complexity in this space.
  - busterarm 6y ago
    https://github.com/Roblox/terraform-provider-maas https://github.com/Roblox/terraform-provider-maas There's even a terraform provider in development.

srswtf123 6y ago

Have you checked out Digital Rebar? I’d be interested in hearing anyone’s experiences with it. https://rackn.com/rebar/ https://rackn.com/rebar/

jmedefind 6y ago
We use it for our bare metal management and like it for the most part. It does require a lot of planning though. But the company has been great to work with and super helpful in slack.

eointierney 6y ago

More and more guix or nixos seem like good practical choices for these kinds of use cases. I prefer guix as it's a little less finicky (though https://gitlab.com/nonguix/nonguix https://gitlab.com/nonguix/nonguix is probably required for all non-purists such as this lowly worm). Ubuntu OS is now stinky doodoo, which is a shame as it used to be the cat's pajamas for ease of use. Snapd is a debacle.

c7DJTLrn 6y ago
Snap definitely doesn't belong on servers - our flavor of Ubuntu doesn't have it. If you bootstrap from Ubuntu Base you can cut out a lot of that crap.
spindle 6y ago
I totally agree except that I prefer NixOS because it supports ZFS so well.

chousuke 6y ago

Out of curiosity, what sorts of issues are you having? I don't have much experience specifically with Ubuntu on bare metal, but I find kickstart easier to understand and use compared to preseed, for consistent bare-metal installations. I think there's an OpenStack project for provisioning bare metal servers via an API from images; I wonder how that's doing nowadays. Puppet is pretty good for configuration management when your systems do actually require the occasional change instead of just being continuously redeployed. Maybe there are people who rebuild their database servers every hour via CI, but I'd rather not. "Immutable infra" is definitely not the default state of things in DevOps land either. Often people talking about how their infra is immutable just conveniently ignore the parts that aren't. The data has to live somewhere.

hnlmorg 6y ago

> those of us working with bare metal are still trying to find a way to keep machines up and running with an OS that doesn't get corrupted from unexpected poweroffs or permanently cut itself off from the network because of a bad config. I'm guessing there's more to this story than you've summarised because those points are pretty easily solved with: - UPS (if the power outs are that much of a problem then you might need to invest in a generator as well). - iLo / IPMI (remote management). Though even just running a serial cable out the back of the server is good enough for a remote console in the event of a network failure. As for managing the config of them, the usual tools like Ansible and Puppet work just as well (in some cases actually better since they were initially designed for on-prem hardware). Likewise for Docker. So don't think you can't run those tools on bare metal Linux. But if you don't want the containerisation-like aspects of Docker but still wanted the deployment tools then you can go a long way with git and shell scripts. While DevOps really came into popularity with cloud hosting, there's nothing fundamentally new about a lot of the tooling that wasn't possible in the old days of bare metal UNIX and Linux. Us older sysadmins were still doing a lot of the same stuff back then too, we just didn't given it trendy names back then. 10+ years ago I was installing Linux remotely on bare metal hardware via a console server. Then once I had one machine installed I'd take one HDD out of the mirrored RAID array and plug it in the 2nd server so the RAID controller would clone. I could repeat the disk swapping as many times as I wanted as long as I remembered to change the IPs (again, via remote serial console). And this was the lazy way of deploying a fleet of servers quickly. More professional places would push the install onto the server via netboot cloning solutions.

c7DJTLrn 6y ago
Yes, there's more to it sadly. The hardware isn't in our control and UPS are out of the question. The machines aren't on premises or in a datacentre. Puppet works okayish but when you have a large fleet of what are basically IoT devices, you start getting an unpleasant failure rate. Push one bad network config and you've 'bricked' thousands of machines. I am being a bit vague intentionally, hope you can understand. But to help imagine, the scale of our problem is pretty big, you've probably even had some interaction with one of our machines.
- 6y ago

coddle-hark 6y ago

Basically, you want read-only systems with separate partitions for persistent data. Have two images to boot from and overwrite just one of them when you upgrade. This is how Balena does it and it’s a pretty common setup for IoT stuff.

CameronNemo 6y ago
http://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format http://www.chromium.org/chromium-os/chromiumos-design-docs/d...
hedora 6y ago
It’s also not difficult to implement from scratch. Pretty much any linux distro with a bootable USB installer already supports it out of the box.

gjsman-1000 6y ago

Have you tried Fedora Silverblue? It uses OSTree technology to offer many of the features of Ubuntu Core including automatic updating, upgrades that work regardless of power loss, Flatpak confinement for all apps (without any arbitrary restrictions like Snaps), and it's free. "Silverblue is a variant of Fedora Workstation. It looks, feels and behaves like a regular desktop operating system, and the experience is similar to what you find with using a standard Fedora Workstation. However, unlike other operating systems, Silverblue is immutable. This means that every installation is identical to every other installation of the same version. The operating system that is on disk is exactly the same from one machine to the next, and it never changes as it is used. Silverblue’s immutable design is intended to make it more stable, less prone to bugs, and easier to test and develop. Finally, Silverblue’s immutable design also makes it an excellent platform for containerized applications as well as container-based software development. In each case, applications (apps) and containers are kept separate from the host system, improving stability and reliability. Silverblue’s core technologies have some other helpful features. OS updates are fast and there’s no waiting around for them to install: just reboot as normal to start using the next version. With Silverblue, it is also possible to roll back to the previous version of the operating system, if something goes wrong." https://silverblue.fedoraproject.org https://silverblue.fedoraproject.org

dineshdb 6y ago
Fedora CoreOS might be better in this regard since that is designed for server use case. Silverblue took that technology and applied to the desktop. Silverblue user here. Haven't tried CoreOS myself.
- curt15 6y ago
  What about Fedora IoT?
  - dineshdb 6y ago
    That seems to be geared towards edge servers and IoT use cases. Otherwise, all of them use the same technologies. Also one thing to note is that CoreOS as well as IoT is bootstrappable with a configuration file via ignition(a tool for automation).

swiley 6y ago

The original author of busybox had a really good idea for this. I don't remember exactly but I think he partitioned the bootdisk in half amd had the bootloader fail a partition (and not boot it) if it didn't boot all the way. Each partition would apply OTA updates depending on which was active.

clcaev 6y ago

Have you considered FreeBSD? Many NAS devices use it, and those are sometimes updated remotely, etc. There's less churn. Keep most of your work in jails, which have been production quality since 10 years ago.

zmix 6y ago

Is there any specific reason not to use something like SmartOS?

sneak 6y ago

CoreOS was a good attempt in this direction, but seems to have died. I am also a fan of the ChromeOS partition layout, which is solving the same problem.

dgellow 6y ago
I still miss CoreOS, it was a really good solution. Edit: as said in another comment in this thread, there are now alternatives to CoreOS! - Fedora CoreOS: https://docs.fedoraproject.org/en-US/fedora-coreos/ https://docs.fedoraproject.org/en-US/fedora-coreos/ - Flatcar Container Linux: https://kinvolk.io/flatcar-container-linux/ https://kinvolk.io/flatcar-container-linux/

elcritch 6y ago

> Bare metal management really feels like an unsolved problem. Whilst everybody working with cloud environments is whisked away by the latest shiny tools like Docker and Ansible, those of us working with bare metal are still trying to find a way to keep machines up and running with an OS that doesn't get corrupted from unexpected poweroffs or permanently cut itself off from the network because of a bad config. It is and isn't solved. It usually takes a lot of work or custom scripts. One of the best is the Nerves Project, and is what I use for IoT deploymens [0] or even simple cloud deployments. Nerves is setup to run Elixir/Erlang, but it's really just a wrapper around buildroot and Elixir can start programs in any language desired with some work. One of the core authors wrote a tool called `fwup` for doing immutable updates on Linux [1]. The ability to do an A/B update and have the device do an automatic rollback if an update fails is crucial. A year or so ago they changed the default boot process to still allow networking and remote connections to work even if the main application crashes. Surprisingly it's all done using Erlang tooling, AFAICT. There are still rough edges, like limited ipv6 support. You can still get devices failing from dead SD cards -- even if your system boots from a read-only partition as power outages during a write on any partition can effectively destroy the SD card, so skip the SD cards. 0: https://www.nerves-project.org/ https://www.nerves-project.org/ 1: https://github.com/fwup-home/fwup https://github.com/fwup-home/fwup

mikepurvis 6y ago

I agree that it's completely unsolved. I work in mobile robots and it's just maddening the number of solutions which either won't work if you're headless or won't work if you'll need kernel upgrades, or won't work if you'll need seamless rollbacks, or won't work if you'll want OTA updates or won't work if you'll need to be able to reconfigure the network in-band (not by exiting your app for some outside management GUI). We ended up with a mostly hand rolled system based on kexec, grub chainloading, and deploying rootfs images as big tarballs. It works, but I really wish there was something we could have just taken off the shelf.

rjzzleep 6y ago

There have been various different setups here. Have you looked at Matchbox? > matchbox is a service that matches bare-metal machines to profiles that PXE boot and provision clusters. Machines are matched by labels like MAC or UUID during PXE and profiles specify a kernel/initrd, iPXE config, and Ignition config. https://github.com/poseidon/matchbox https://github.com/poseidon/matchbox

cik 6y ago

I did this for a few years with cobbler (https://cobbler.github.io/ https://cobbler.github.io/). cobbler, pxe, bootp, tftp, ansible and friends pretty much solve this problem. In fact, if you know the mac addresses (or ranges) you can fairly easily designate groups of machines, roles, and the like. Years ago I did this to "rapidly" provision a couple of thousand machines we bought for the stock exchange. You can do most of your testing locally in vagrants, even simulating the networks you need to provision. You can go a step further and trigger api updates at the end of your ansible runs so that cobbler updates collins (https://tumblr.github.io/collins/ https://tumblr.github.io/collins/) so that you can track inventorying, cm changes and the like. At one point in time we reimaged entire subnets when they lost power - and ran tests against the hardware to ensure the machines functioned post-provisioning. As far as I know the boursa still uses this system.

necovek 6y ago

You might also look at Canonical MAAS (maas.io) which will get you some of the way there too. Though, that's more of the traditional server management approach, if properly set up, you can have it worry about the network configuration and pushing it to clients. To me, it beats the entire IoT pull setup that Canonical is pushing with Ubuntu Core and snaps.