6 ms·

Incus-OS: Immutable Linux OS to run Incus as a hypervisor

188 points by _kb 11mo ago

kouskoush 11mo ago
It's software for a private cloud and can convert your legacy VMWare into IncusOS.
- deleted 11mo ago
  [deleted]
- HumanOstrich 11mo ago
  My home lab is not a private cloud and I don't use VMWare.
seabrookmx 11mo ago
Not _technically_ a hypervisor since these are Linux (system) containers and use the same cgroup magic under the hood as docker/containerd. But this is definitely neat. I've found Incus quite handy for development environments, and a good compliment to docker.
- k_bx 11mo ago
  Incus supports both qemu and lxc
- kosinus 11mo ago
  You can also start QEMU/KVM powered VMs with Incus, I assume that's also possible with IncusOS?
  - virtuous_sloth 11mo ago
    Yes. And most importantly, the Incus API and CLI client (which uses the API) presents a consistent management language for system containers (the default ones with a init/systemd-controlled userspace), OCI containers (unpacked, not layered), and VMs. Well, as consistent as makes sense for each. There are a number of options/properties that are specific to each, but it feels very consistent. The Incus server inside IncusOS is the same software. The difference is as little userspace as possible alongside it (not even busybox).
- udev4096 11mo ago
  It's a lot more than that. Clustering, storage drivers, networking, etc makes up a whole virtual machine manager. It never says it's a hypervisor, it's a VMM as outlined on it's github: "Powerful system container and virtual machine manager"
k_bx 11mo ago
Really excited to try this out. I have a fleet of containers on ubuntu + incus. Not only does this do ZFS optimization, I look forward having easy container optimized backup, live cluster migration (to a different machine without downtime) and so much more. I use Proxmox on fat servers, but for homelab-like setup Incus OS seems more like a sweet spot
- azov 11mo ago
  I was hoping for easy backup via zfs send as well, but turns out it’s not so easy atm. IncusOS does not give you shell access, you have to figure out IncusOS ways to do things via their CLI/API. I haven’t found an easy way to do incremental backup of the whole system yet. You can backup individual instances/volumes via incus export (which seems to use zfs send under the hood), but not the whole thing. I have mixed feelings about their decision not to give you shell access. Guess those who want flexibility can always just install Incus on top of any Linux they like, but it would be nice to have an escape hatch for when IncusOS gives you almost everything you want…
  - amluto 11mo ago
    I occasionally contemplate that, if I were designing an OS meant to be sort-of-immutable (like Incus OS or Fedora Silverblue etc or MacOS), I would probably build it like this: The main filesystem is verified and immutable. Everything that isn't configuration or the user-controlled payload is genuinely read-only, and the system will even cryptographically verify it on boot or first use. You cannot modify /bin/bash, etc. If you want to test a modification, you can configure an overlay, and you can boot with that overlay live. You can configure the overlay to also be immutable or you can make the overlay mutable. But the choice of booting into the overlay is controlled by code that cannot by overlaid, so you can always turn the overlay off no matter how much you screw it up. The user may get root access, but if your system is remotely attested or uses a TPM or such for security, then that policy will find out if you do so before you can do anything as root. So you can shell in and attach a debugger to a system service, but you cannot do that and also pretend to your orchestration tools that you have not done so. The default configuration is mostly empty. When you change a default, you are not modifying the middle of a giant plist where no one will ever understand what happened. You only create new configuration, and deleting it is just fine. The result would, I think, give system owners plenty of ability to hack on their own systems, but they could also unhack their systems easily. There are very few systems out there with both of these properties right now...
leoedin 11mo ago
I guess IncusOS (and Incus) achieve similar goals to ProxMox? Has anyone used both and have any opinions on how they perform?
- udev4096 11mo ago
  I have switched to incus and it's really great. It's lightweight, has a working terraform provider, easy-to-use cli, pre-built images (LXC and VM) of major distros (while in proxmox, you have to create templates all the time for VMs), runs on any distro (on proxmox, you're stuck with debian), clustering is nice, supports bunch of storage drivers (dir, btrfs, ceph, zfs), simple web UI and active community. The project leader is also very active and helpful while in proxmox, it's a little unresponsive. You can even install `incus-base` package which only contains LXC specific components for only running LXC containers. I have noticed incus has better security configs by default. For instance, all pre-built images come with secureboot enabled and there are ACLs which are easy to configure for fine-grained network rules. The only downside I feel like is lack of something like PBS
  - athoneycutt 11mo ago
    Though IncusOS itself is based on Debian so for the first point against Proxmox I guess using Incus on your OS of choice would be better?
    - udev4096 11mo ago
      IncusOS is different. You can use incus itself on all the major distros: https://linuxcontainers.org/incus/docs/main/installing/ https://linuxcontainers.org/incus/docs/main/installing/
  - stormking 11mo ago
    I love the ability to directly run docker containers. I think their approach to authentication / authorization is insane (not in a good way).
    - 11mo ago
genshii 11mo ago
I used Proxmox for years to run a fairly comprehensive homelab, and a few months ago replaced the entire thing with Incus (on a debian host, haven't tried IncusOS yet). Incus is amazing and it makes so many things so much easier compared to Proxmox. One thing in particular is permissions in unprivileged containers. In Proxmox, you have to do a bunch of somewhat confusing ID mapping. In Incus, it's as simple as setting "shift=true". Also the profile system in Incus is really powerful and allowed me to deduplicate a ton of config.
- udev4096 11mo ago
  Profiles are really great. It's like cloud-init on steroids
- aborsy 11mo ago
  Incus is more comparable to LXD than proxmox. IncusOS is different though. LXD containers also are unprivileged by default.
  - guipsp 11mo ago
    You might be mixing up LXC and LXD
    - gchamonlive 11mo ago
      Even I that worked for a long while with this tech would mix them up time and again, I think it's understandable.
    - madeforhnyo 11mo ago
      From Incus main page: > The Incus project was created by Aleksa Sarai as a community driven alternative to Canonical's LXD. Today, it's led and maintained by many of the same people that once created LXD. Thé confusion si real
    - aborsy 11mo ago
      No, LXD’s LXCs. I use it and it’s good. The UID mappings are correctly setup in Ubuntu so the containers run non-privileged by default. I hear Incus, a fork of LXD, is better. It’s used in truenas.
dizhn 11mo ago
In case there might be people who are not familiar with Incus, it was forked from LXD to keep it open source. It's very good software.
- gchamonlive 11mo ago
  AFAIK LXD is still opensource, as are most if not all products from Canonical. I think the fork is because LXD when it was moved to Canonical made the community uneasy because of the way that they would integrate with Ubuntu lifecycle and tooling. https://github.com/canonical/lxd https://github.com/canonical/lxd it's AGPL-V3
  - dizhn 11mo ago
    Thanks for the correction. I kind of remember this as a more hostile thing done by Canonical at the time but this fork announcement from that time does not support that view either. Perhaps I am misremembering the little I do remember. https://discuss.linuxcontainers.org/t/introducing-incus/17781 https://discuss.linuxcontainers.org/t/introducing-incus/1778...
  - stgraber 11mo ago
    It's indeed still open source, but was moved from Apache 2.0 to AGPLv3 and from not having any requirements to contributions to requiring all contributors sign a CLA. So it's definitely still open source, but the changes they made allows them to still look and import any change from Incus that they wish, whilst preventing us from looking at any LXD code without risk of tainting ourselves...
  - qskousen 11mo ago
    My number one reason for moving away from using LXD in production after this change is that LXD is only available through snap, which caused multiple downtimes in the cluster because of the forced updates.
    - gchamonlive 11mo ago
      Exactly. And depending on whether you are installing it with snap or other package managers, like pacman in arch, it'll actually use differently folders for configs, so if you are writing automation for say automatically manage remotes without relying on the cli, you'll have to account for that. Better to just use Incus whenever possible.
octagons 11mo ago
I’ve been running an Incus cluster of 3 fairly beefy servers for about a year now. It’s my go-to recommendation for anyone wanting to setup a new virtualized environment. One of my favorite features is how you can tag different cluster members for different architectures. In the same cluster, I can have traditional dual-socket x86 servers with a dozen DIMM slots as well as Raspberry Pis. The architecture tagging lets me strategize execution of ARM-based container workloads to be only on the Pis, or opt to run them via QEMU on the x86 platforms if that makes more sense in a particular scenario. Since I deal with a lot of embedded firmware, this offers a nice, flexible platform. Stephen Graeber is also a long time contributor to the LXC project and his reasoning behind this fork and other changes are quite sound. I hope the project sees continued success. Stephen’s business model of offering consulting services for Incus systems also seems quite sound.
- richardwhiuk 11mo ago
  What hypervisior environments don't have this?
- loloquwowndueo 11mo ago
  * Stéphane Graber.
chaz6 11mo ago
It seems to suffer from a chicken and egg problem. To get an image you are supposed to run `incus remote get-client-certificate` to put into the "image customizer", and you cannot generate an image without it. So how do you get started?
- stgraber 11mo ago
  You can download the CLI client for Linux, Windows and MacOS from our Github releases: https://github.com/lxc/incus/releases/latest/ https://github.com/lxc/incus/releases/latest/ I've filed https://github.com/lxc/incus-os/issues/551 https://github.com/lxc/incus-os/issues/551 which we should be able to sort out later today.
  - knowitnone3 11mo ago
    perhaps add installation instructions in the README? Most people already know they need the binary to run that command. For those who don't, I don't recommend you baby them because next thing you know, they've downloaded the wrong binary and it doesn't run.
Animats 11mo ago
Is there such a thing as DIMM modules with ROM chips? It would be useful for some applications to be able to burn the immutable OS into a read only memory as a form of tamper-resistance in key infrastructure.
- amluto 11mo ago
  There are CPUs with fuses that can store keys, e.g. Intel Boot Guard. The tooling to use it as an end user is not friendly to say the least. You can set your own Secure Boot keys. The history of outrageous security vulnerabilities that break it is long and storied. The underlying architecture is abysmal.
sklarsa 11mo ago
I've been using Incus containers (not VMs) for running tests against a "real" OS and it's been an absolute game changer for me. It's granted me the ability to simultaneously spin up-and-down a plethora of fresh OSes on my local dev machine, which I then use as testing targets for components of my codebase that require Docker or systemd. With traditional containers, it's tricky to mimic those capabilities as they would exist on a normal VM. Because both my project and Incus are written in Go, orchestrating Incus resources in my test code has been pretty seamless. And with "ephemeral" containers, if things start to get out of hand, I just need to stop the container to clean it up. Much easier than a 2-step process like it usually is. Looking forward to seeing what's to come in IncusOS!
acters 11mo ago
I use incus to pass a containerized kali os the Wayland and x11 sockets, and whatever else maybe in the /run/user/1000 folder and x11 socket folder, like pipewire. It isn't perfect, but it's really nice spawning a shell/bar/etc inside the container and it goes over the current Wayland desktop. Then I am able to use it to spawn other graphical apps. It works really well. Incus is amazing, or lxc and wayland in general.
emilfihlman 11mo ago
Incus is very nice and super featured, but suffers from a few issues, namely unintuitive/hard onboarding and bad defaults, which makes giving access to people annoying, as it requires teaching them first and that they can't just make a vm with a few clicks immediately, limited authentication and user control options, like if no external auth users must exist on underlying system, and with limited but very strict auth options requires a full domain and no proxying, currently (might get fixed partially later). And finally, it suffers from hardcore tracking upstream, ie canonical/lxd(-ui), meaning they won't really do any changes that lxd wouldn't do, and thus are slaved to them : (
- loloquwowndueo 11mo ago
  Sorry what? Lxd is NOT incus upstream. Incus was forked from lxd specifically to allow divergence and the licenses mean changes rather flow in the other direction. Not that canonical considers incus “upstream” - they’re just divergent forks at this point.
  - emilfihlman 11mo ago
    But unfortunately it pretty much is, at least according to the maintainers themselves. They try to not diverge unless "necessary", and for most parts it's not necessary to them.
    - SirGiggles 11mo ago
      Can you substantiate this? The only time this held, vaguely to my recollection, true was prior to Incus 0.4 where both were cherry picking from each other but neither were upstream of each other
      - emilfihlman 11mo ago
        https://github.com/zabbly/incus/issues/89#issuecomment-3076413659 https://github.com/zabbly/incus/issues/89#issuecomment-30764... I mean sure, it's the UI component only, and not on the lxc repo but the Zabbly one, and maybe they treat things widely differently depending. But it's the same developer for both, and I'm willing to bet it's a similar mindset for projects. And I swear I had a similar experience with regards to some other incus issues where there was a similar response about changes.
mrbluecoat 11mo ago
Nice! Reminds me of Galos: https://github.com/ascension-association/galos https://github.com/ascension-association/galos
veidr 11mo ago
Interesting. I recommend Incus over other hypervisor-oriented OS distributions (Proxmox, etc.), for many reasons, but one minor reason is that you can install it on Debian or Arch or whatever Linux you like. It's a comprehensive management layer for VMs and Linux containers, with good CLI, web UI, cloning/moving/copying/migrating etc., but... why does that need to be an OS? However, I'm confident that if that is what you want, this is probably fantastic — Incus, including the old LXD (which was mainly built by the same core developers, until Canonical behaved in ways they didn't like, and they hard-forked LXD to create Incus) has been one of my favorite open-source projects for several years. Fantastic software, steady stream of reliable releases, helpful community... Incus is great.
- _kb 11mo ago
  > why does that need to be an OS? It doesn't. You can still run Incus on other platforms of choice. Sample size 1 here, but a big advantage of the 'full-stack' approach is things like network config, storage management, boot safety etc all work out of the box and you then get a single API (and nice client) for the whole machine. I get the benefits of cloud infra, like not having to care (too much) about sysadmin, from some hardware sitting in the corner. I can literally incus launch images:nixos/unstable foo -t aws:m1.large and start hacking. Previously I would still need to be maintaining that base layer too. That still makes sense for some environments, but particularly for home I just want my lights and music to work, and be able to play.
  - veidr 11mo ago
    Yeah, I already maintain that base layer, and like being able to just run it on Debian, like I said. But, one of the awesome things about Incus is how easy it is to move instances around the LAN (or WAN, I suppose). I don't need super rigorous failovers for most things running at home, but just because it's so easy, I typically always do have a recent copy of every container I run (blogs, home automation servers, various web apps, etc) on a different machine, so when one machine goes down it's super easy to just start the equivalent instance on the other machine. I run instances I need to interact with (e.g., do development in containers via SSH and remote-editors, with occasional Remote Desktop) on my very-fast Linux workstation — that also does other stuff like local development, web browsing, etc., but most instances that don't need power run on my old 56-core Xeon enterprise server (used, they are roughly as cheap as a Mac Mini). Incus makes it super easy to move instances around, and from a skim of the announcement it looks like you could just put Incus OS on some machine you have lying around and drop it into an existing config like that with minimal effort. I look forward to trying it out, even if my "main" Incus will probably remain on my actual manually-curated Linux desktop.
cryptonector 11mo ago
There was a group at Sun that tried to make Solaris immutable based on ZFS like this back in the 00s, and... there were a lot of issues along the way, and they couldn't finish in time, so it didn't happen. So I can appreciate that Incus-OS must have been a difficult project.
AbuAssar 11mo ago
> What is Incus? > Incus is a next-generation system container, application container, and virtual > machine manager.

188 points by _kb 11mo ago

kouskoush 11mo ago

It's software for a private cloud and can convert your legacy VMWare into IncusOS.

deleted 11mo ago
[deleted]
HumanOstrich 11mo ago
My home lab is not a private cloud and I don't use VMWare.

seabrookmx 11mo ago

Not _technically_ a hypervisor since these are Linux (system) containers and use the same cgroup magic under the hood as docker/containerd. But this is definitely neat. I've found Incus quite handy for development environments, and a good compliment to docker.

k_bx 11mo ago
Incus supports both qemu and lxc
kosinus 11mo ago
You can also start QEMU/KVM powered VMs with Incus, I assume that's also possible with IncusOS?
- virtuous_sloth 11mo ago
  Yes. And most importantly, the Incus API and CLI client (which uses the API) presents a consistent management language for system containers (the default ones with a init/systemd-controlled userspace), OCI containers (unpacked, not layered), and VMs. Well, as consistent as makes sense for each. There are a number of options/properties that are specific to each, but it feels very consistent. The Incus server inside IncusOS is the same software. The difference is as little userspace as possible alongside it (not even busybox).
udev4096 11mo ago
It's a lot more than that. Clustering, storage drivers, networking, etc makes up a whole virtual machine manager. It never says it's a hypervisor, it's a VMM as outlined on it's github: "Powerful system container and virtual machine manager"

k_bx 11mo ago

Really excited to try this out. I have a fleet of containers on ubuntu + incus. Not only does this do ZFS optimization, I look forward having easy container optimized backup, live cluster migration (to a different machine without downtime) and so much more. I use Proxmox on fat servers, but for homelab-like setup Incus OS seems more like a sweet spot

azov 11mo ago
I was hoping for easy backup via zfs send as well, but turns out it’s not so easy atm. IncusOS does not give you shell access, you have to figure out IncusOS ways to do things via their CLI/API. I haven’t found an easy way to do incremental backup of the whole system yet. You can backup individual instances/volumes via incus export (which seems to use zfs send under the hood), but not the whole thing. I have mixed feelings about their decision not to give you shell access. Guess those who want flexibility can always just install Incus on top of any Linux they like, but it would be nice to have an escape hatch for when IncusOS gives you almost everything you want…
- amluto 11mo ago
  I occasionally contemplate that, if I were designing an OS meant to be sort-of-immutable (like Incus OS or Fedora Silverblue etc or MacOS), I would probably build it like this: The main filesystem is verified and immutable. Everything that isn't configuration or the user-controlled payload is genuinely read-only, and the system will even cryptographically verify it on boot or first use. You cannot modify /bin/bash, etc. If you want to test a modification, you can configure an overlay, and you can boot with that overlay live. You can configure the overlay to also be immutable or you can make the overlay mutable. But the choice of booting into the overlay is controlled by code that cannot by overlaid, so you can always turn the overlay off no matter how much you screw it up. The user may get root access, but if your system is remotely attested or uses a TPM or such for security, then that policy will find out if you do so before you can do anything as root. So you can shell in and attach a debugger to a system service, but you cannot do that and also pretend to your orchestration tools that you have not done so. The default configuration is mostly empty. When you change a default, you are not modifying the middle of a giant plist where no one will ever understand what happened. You only create new configuration, and deleting it is just fine. The result would, I think, give system owners plenty of ability to hack on their own systems, but they could also unhack their systems easily. There are very few systems out there with both of these properties right now...

leoedin 11mo ago

I guess IncusOS (and Incus) achieve similar goals to ProxMox? Has anyone used both and have any opinions on how they perform?

udev4096 11mo ago
I have switched to incus and it's really great. It's lightweight, has a working terraform provider, easy-to-use cli, pre-built images (LXC and VM) of major distros (while in proxmox, you have to create templates all the time for VMs), runs on any distro (on proxmox, you're stuck with debian), clustering is nice, supports bunch of storage drivers (dir, btrfs, ceph, zfs), simple web UI and active community. The project leader is also very active and helpful while in proxmox, it's a little unresponsive. You can even install `incus-base` package which only contains LXC specific components for only running LXC containers. I have noticed incus has better security configs by default. For instance, all pre-built images come with secureboot enabled and there are ACLs which are easy to configure for fine-grained network rules. The only downside I feel like is lack of something like PBS
- athoneycutt 11mo ago
  Though IncusOS itself is based on Debian so for the first point against Proxmox I guess using Incus on your OS of choice would be better?
  - udev4096 11mo ago
    IncusOS is different. You can use incus itself on all the major distros: https://linuxcontainers.org/incus/docs/main/installing/ https://linuxcontainers.org/incus/docs/main/installing/
- stormking 11mo ago
  I love the ability to directly run docker containers. I think their approach to authentication / authorization is insane (not in a good way).
  - 11mo ago

genshii 11mo ago

I used Proxmox for years to run a fairly comprehensive homelab, and a few months ago replaced the entire thing with Incus (on a debian host, haven't tried IncusOS yet). Incus is amazing and it makes so many things so much easier compared to Proxmox. One thing in particular is permissions in unprivileged containers. In Proxmox, you have to do a bunch of somewhat confusing ID mapping. In Incus, it's as simple as setting "shift=true". Also the profile system in Incus is really powerful and allowed me to deduplicate a ton of config.

udev4096 11mo ago
Profiles are really great. It's like cloud-init on steroids
aborsy 11mo ago
Incus is more comparable to LXD than proxmox. IncusOS is different though. LXD containers also are unprivileged by default.
- guipsp 11mo ago
  You might be mixing up LXC and LXD
  - gchamonlive 11mo ago
    Even I that worked for a long while with this tech would mix them up time and again, I think it's understandable.
  - madeforhnyo 11mo ago
    From Incus main page: > The Incus project was created by Aleksa Sarai as a community driven alternative to Canonical's LXD. Today, it's led and maintained by many of the same people that once created LXD. Thé confusion si real
  - aborsy 11mo ago
    No, LXD’s LXCs. I use it and it’s good. The UID mappings are correctly setup in Ubuntu so the containers run non-privileged by default. I hear Incus, a fork of LXD, is better. It’s used in truenas.

dizhn 11mo ago

In case there might be people who are not familiar with Incus, it was forked from LXD to keep it open source. It's very good software.

gchamonlive 11mo ago
AFAIK LXD is still opensource, as are most if not all products from Canonical. I think the fork is because LXD when it was moved to Canonical made the community uneasy because of the way that they would integrate with Ubuntu lifecycle and tooling. https://github.com/canonical/lxd https://github.com/canonical/lxd it's AGPL-V3
- dizhn 11mo ago
  Thanks for the correction. I kind of remember this as a more hostile thing done by Canonical at the time but this fork announcement from that time does not support that view either. Perhaps I am misremembering the little I do remember. https://discuss.linuxcontainers.org/t/introducing-incus/17781 https://discuss.linuxcontainers.org/t/introducing-incus/1778...
- stgraber 11mo ago
  It's indeed still open source, but was moved from Apache 2.0 to AGPLv3 and from not having any requirements to contributions to requiring all contributors sign a CLA. So it's definitely still open source, but the changes they made allows them to still look and import any change from Incus that they wish, whilst preventing us from looking at any LXD code without risk of tainting ourselves...
- qskousen 11mo ago
  My number one reason for moving away from using LXD in production after this change is that LXD is only available through snap, which caused multiple downtimes in the cluster because of the forced updates.
  - gchamonlive 11mo ago
    Exactly. And depending on whether you are installing it with snap or other package managers, like pacman in arch, it'll actually use differently folders for configs, so if you are writing automation for say automatically manage remotes without relying on the cli, you'll have to account for that. Better to just use Incus whenever possible.

octagons 11mo ago

I’ve been running an Incus cluster of 3 fairly beefy servers for about a year now. It’s my go-to recommendation for anyone wanting to setup a new virtualized environment. One of my favorite features is how you can tag different cluster members for different architectures. In the same cluster, I can have traditional dual-socket x86 servers with a dozen DIMM slots as well as Raspberry Pis. The architecture tagging lets me strategize execution of ARM-based container workloads to be only on the Pis, or opt to run them via QEMU on the x86 platforms if that makes more sense in a particular scenario. Since I deal with a lot of embedded firmware, this offers a nice, flexible platform. Stephen Graeber is also a long time contributor to the LXC project and his reasoning behind this fork and other changes are quite sound. I hope the project sees continued success. Stephen’s business model of offering consulting services for Incus systems also seems quite sound.

richardwhiuk 11mo ago
What hypervisior environments don't have this?
loloquwowndueo 11mo ago
* Stéphane Graber.

chaz6 11mo ago

It seems to suffer from a chicken and egg problem. To get an image you are supposed to run `incus remote get-client-certificate` to put into the "image customizer", and you cannot generate an image without it. So how do you get started?

stgraber 11mo ago
You can download the CLI client for Linux, Windows and MacOS from our Github releases: https://github.com/lxc/incus/releases/latest/ https://github.com/lxc/incus/releases/latest/ I've filed https://github.com/lxc/incus-os/issues/551 https://github.com/lxc/incus-os/issues/551 which we should be able to sort out later today.
- knowitnone3 11mo ago
  perhaps add installation instructions in the README? Most people already know they need the binary to run that command. For those who don't, I don't recommend you baby them because next thing you know, they've downloaded the wrong binary and it doesn't run.

Animats 11mo ago

Is there such a thing as DIMM modules with ROM chips? It would be useful for some applications to be able to burn the immutable OS into a read only memory as a form of tamper-resistance in key infrastructure.

amluto 11mo ago
There are CPUs with fuses that can store keys, e.g. Intel Boot Guard. The tooling to use it as an end user is not friendly to say the least. You can set your own Secure Boot keys. The history of outrageous security vulnerabilities that break it is long and storied. The underlying architecture is abysmal.

sklarsa 11mo ago

I've been using Incus containers (not VMs) for running tests against a "real" OS and it's been an absolute game changer for me. It's granted me the ability to simultaneously spin up-and-down a plethora of fresh OSes on my local dev machine, which I then use as testing targets for components of my codebase that require Docker or systemd. With traditional containers, it's tricky to mimic those capabilities as they would exist on a normal VM. Because both my project and Incus are written in Go, orchestrating Incus resources in my test code has been pretty seamless. And with "ephemeral" containers, if things start to get out of hand, I just need to stop the container to clean it up. Much easier than a 2-step process like it usually is. Looking forward to seeing what's to come in IncusOS!

acters 11mo ago

I use incus to pass a containerized kali os the Wayland and x11 sockets, and whatever else maybe in the /run/user/1000 folder and x11 socket folder, like pipewire. It isn't perfect, but it's really nice spawning a shell/bar/etc inside the container and it goes over the current Wayland desktop. Then I am able to use it to spawn other graphical apps. It works really well. Incus is amazing, or lxc and wayland in general.

emilfihlman 11mo ago

Incus is very nice and super featured, but suffers from a few issues, namely unintuitive/hard onboarding and bad defaults, which makes giving access to people annoying, as it requires teaching them first and that they can't just make a vm with a few clicks immediately, limited authentication and user control options, like if no external auth users must exist on underlying system, and with limited but very strict auth options requires a full domain and no proxying, currently (might get fixed partially later). And finally, it suffers from hardcore tracking upstream, ie canonical/lxd(-ui), meaning they won't really do any changes that lxd wouldn't do, and thus are slaved to them : (

loloquwowndueo 11mo ago
Sorry what? Lxd is NOT incus upstream. Incus was forked from lxd specifically to allow divergence and the licenses mean changes rather flow in the other direction. Not that canonical considers incus “upstream” - they’re just divergent forks at this point.
- emilfihlman 11mo ago
  But unfortunately it pretty much is, at least according to the maintainers themselves. They try to not diverge unless "necessary", and for most parts it's not necessary to them.
  - SirGiggles 11mo ago
    Can you substantiate this? The only time this held, vaguely to my recollection, true was prior to Incus 0.4 where both were cherry picking from each other but neither were upstream of each other
    - emilfihlman 11mo ago
      https://github.com/zabbly/incus/issues/89#issuecomment-3076413659 https://github.com/zabbly/incus/issues/89#issuecomment-30764... I mean sure, it's the UI component only, and not on the lxc repo but the Zabbly one, and maybe they treat things widely differently depending. But it's the same developer for both, and I'm willing to bet it's a similar mindset for projects. And I swear I had a similar experience with regards to some other incus issues where there was a similar response about changes.

mrbluecoat 11mo ago

Nice! Reminds me of Galos: https://github.com/ascension-association/galos https://github.com/ascension-association/galos

veidr 11mo ago

Interesting. I recommend Incus over other hypervisor-oriented OS distributions (Proxmox, etc.), for many reasons, but one minor reason is that you can install it on Debian or Arch or whatever Linux you like. It's a comprehensive management layer for VMs and Linux containers, with good CLI, web UI, cloning/moving/copying/migrating etc., but... why does that need to be an OS? However, I'm confident that if that is what you want, this is probably fantastic — Incus, including the old LXD (which was mainly built by the same core developers, until Canonical behaved in ways they didn't like, and they hard-forked LXD to create Incus) has been one of my favorite open-source projects for several years. Fantastic software, steady stream of reliable releases, helpful community... Incus is great.

_kb 11mo ago
> why does that need to be an OS? It doesn't. You can still run Incus on other platforms of choice. Sample size 1 here, but a big advantage of the 'full-stack' approach is things like network config, storage management, boot safety etc all work out of the box and you then get a single API (and nice client) for the whole machine. I get the benefits of cloud infra, like not having to care (too much) about sysadmin, from some hardware sitting in the corner. I can literally incus launch images:nixos/unstable foo -t aws:m1.large and start hacking. Previously I would still need to be maintaining that base layer too. That still makes sense for some environments, but particularly for home I just want my lights and music to work, and be able to play.
- veidr 11mo ago
  Yeah, I already maintain that base layer, and like being able to just run it on Debian, like I said. But, one of the awesome things about Incus is how easy it is to move instances around the LAN (or WAN, I suppose). I don't need super rigorous failovers for most things running at home, but just because it's so easy, I typically always do have a recent copy of every container I run (blogs, home automation servers, various web apps, etc) on a different machine, so when one machine goes down it's super easy to just start the equivalent instance on the other machine. I run instances I need to interact with (e.g., do development in containers via SSH and remote-editors, with occasional Remote Desktop) on my very-fast Linux workstation — that also does other stuff like local development, web browsing, etc., but most instances that don't need power run on my old 56-core Xeon enterprise server (used, they are roughly as cheap as a Mac Mini). Incus makes it super easy to move instances around, and from a skim of the announcement it looks like you could just put Incus OS on some machine you have lying around and drop it into an existing config like that with minimal effort. I look forward to trying it out, even if my "main" Incus will probably remain on my actual manually-curated Linux desktop.

cryptonector 11mo ago

There was a group at Sun that tried to make Solaris immutable based on ZFS like this back in the 00s, and... there were a lot of issues along the way, and they couldn't finish in time, so it didn't happen. So I can appreciate that Incus-OS must have been a difficult project.

AbuAssar 11mo ago

> What is Incus? > Incus is a next-generation system container, application container, and virtual > machine manager.