7 ms·
Incus – Next-generation system container, application container, and VM manager
- manosyja 1y agoWhat can this work with? It says „Containers and VMs“ - I guess that’s LXCs and QEMU VMs?
- nrabulinski 1y agoYes, it uses QEMU under the hood for VMs and runs LXC containers. But also, since recently, you can run docker images in it. Very handy, especially since it has 1st class remote support, meaning you can install only the incus client and when doing `incus launch` or whatever, it will transparently start the container/vm on your remote host
- Semaphor 1y agoSo it looks like a Proxmox alternative, this [0] goes into some reasons to switch. Main selling point seems to be fully OSS and no enterprise version. [0]: https://tadeubento.com/2024/replace-proxmox-with-incus-lxd/ https://tadeubento.com/2024/replace-proxmox-with-incus-lxd/
- hardwaresofton 1y agoIt’s more like a Kubernetes alternative
- loloquwowndueo 1y agoNot really, Kubernetes does a lot of different things that are out of scope for incus or lxd or docker compose for that matter or any hypervisor or …
- hardwaresofton 1y agolike what? I'd love to hear some examples of things Kubernetes does that incus doesn't at this point
- lifty 1y agoOne is for cluster orchestration the other is a single machine container/vm runtime.
- hardwaresofton 1y agohttps://linuxcontainers.org/incus/docs/main/explanation/clustering/ https://linuxcontainers.org/incus/docs/main/explanation/clus... https://linuxcontainers.org/incus/docs/main/explanation/clustering/ https://linuxcontainers.org/incus/docs/main/explanation/clus... You may have a point there that k8s is not meant for single machines but that’s not a hard rule, more like a “why would you want to” you can absolutely run single node Kubernetes. Also strictly speaking incus is not a container nor vm runtime, it’s an orchestrator of those things.
- ChocolateGod 1y agoService discovery? I'm sure you could probably manually hook up Incus to something like Consul, but it would be more effort than it's worth.
- hardwaresofton 1y agoHmmn... I think it really depends on what you call "service discovery", but I'd note that k8s doesn't give you something as complicated as Consul either -- by default, you get DNS & IP networking for service discovery, which Incus also supports. If you mean service discovery as in being able to use the k8s API (ex. to find service manually) from inside your application, then incus allows for that too.. Maybe the key difference you're pointing at here is that Incus does not give you one huge network (where by default everything is routable), so you have to set up your own bridge networks: https://linuxcontainers.org/incus/docs/main/explanation/networks/ https://linuxcontainers.org/incus/docs/main/explanation/netw... https://linuxcontainers.org/incus/docs/main/reference/network_bridge/ https://linuxcontainers.org/incus/docs/main/reference/networ... I could definitely agree here that Incus is more unencumbered in this respect
- moondev 1y agoProxmox feels like a more apt comparison, as they both act like a controlplane for KVM virtual-machines and LXC containers across one or multiple hosts. If you are interested in running kubernetes on top of incus, that is your kubernetes cluster nodes will be made up of KVM or LXC instances - I highly recommend the cluster-api provider incus https://github.com/lxc/cluster-api-provider-incus https://github.com/lxc/cluster-api-provider-incus This provider is really well done and maintained, including ClusterClass support and array of pre-built machine images for both KVM and LXC. It also supports pivoting the mgmt cluster on to a workload cluster, enabling the mgmt cluster to upgrade itself which is really cool. I was surprised to come across this provider by chance as for some reason it's not listed on the CAPI documentation provider list https://cluster-api.sigs.k8s.io/reference/providers https://cluster-api.sigs.k8s.io/reference/providers
- neoaggelos 1y agoHey! Long-time lurker, never really posted before, author of cluster-api-provider-incus here, did not really expect it to come up on hackernews. Thanks for the good comments! Indeed, adding to the list of CAPI providers is on the roadmap (did not want to do it before discussing with Stephan to move the project under the LXC org, but that is now complete). Also, I'm working on a few other niceties, like a "kind"-style script that would allow easily managing small k8s clusters without the full CAPI requirements (while, at the same time, documenting all it takes to run Kubernetes under LXC in general). You can expect more things about the project, and any feedback would be welcome!
- moondev 1y agoI've kicked the tires on many CAPI providers throughout the years, what you have here rivals CAPA and CAPV. Even CAPK (kubevirt) only recently fully implemented ClusterClass, and there are no classy templates in the release yet. I have actually learned quite a bit just reading your gitbook and workflows. This provider is also great because it sits in the space of fully on-prem and fully self-hosted. Kubevirt is also here but it needs an additional provider to be able to fully pivot and manage itself. I'm quite interested in your machine image pipeline and how you publish them on simplestreams. I'm working with MaaS and really want to implement the same pattern you have, pushing to a central location and let MaaS sync. It's very painful needing to manually import the images beforehand and handle garbage collection. Would your Incus and KVM images work with MaaS as well? If there is a better approach I am all ears. Thanks again for sharing your fantastic work with the community.
- snapplebobapple 1y agowhy would they think no enterprise version is a selling point? I can't learn stuff at work and apply it at home easily with this product. if anything proxmox needs a more enterprise option with faster support and it would be a better product for me. The caveat is there needs to be a credible way to keep the opensource open and available, which proxmox has done so far.
- 63stack 1y agoHow do you handle updating the machine that Incus itself runs on? I imagine you have to be super careful not to introduce any breakage, because then all the VMs/containers go down. What about kernel updates that require reboots? I have heard of ksplice/kexec, but I have never seen them used anywhere.
- dsr_ 1y agoAs with any such system, you need a spare box. Upgrade the spare, move the clients to it, upgrade the original.
- loloquwowndueo 1y agoBut then the clients have downtime while they’re being moved.
- pylotlight 1y agoIsn't that the exact problem that k8s workloads solve by scaling onto new nodes first etc? No downtime required.
- loloquwowndueo 1y agoRight but incus is not k8s. You can stand up spares and switch traffic, but it’s not built in functionality and requires extra orchestration.
- goku12 1y agoIt is a built-in functionality [1] and requires no extra orchestration. In a cluster setup, you would be using virtualized storage (ceph based) and virtualized network (ovn). You can replace a container/VM on one host with another on a different host with the same storage volumes, network and address. This is what k8s does with pod migrations too (edit: except the address). There are a couple of differences though. The first is the pet vs cattle treatment of containers by Incus and k8s respectively. Incus tries to resurrect dead containers as faithfully as possible. This means that Incus treats container crashes like system crashes, and its recovery involves systemd bootup inside the container (kernel too in case of VMs). This is what accounts for the delay. K8s on the other hand, doesn't care about dead containers/pods at all. It just creates another pod, likely with a different address and expects it to handle the interruption. Another difference is the orchestration mechanism behind this. K8s, as you may be aware, uses control loops on controller nodes to detect the crash and initiate the recovery. The recovery is mediated by the kubelets on the worker nodes. Incus seems to have the orchestrator on all nodes. They take decisions based on consensus and manage the recovery process themselves. [1] https://linuxcontainers.org/incus/docs/main/howto/cluster_manage/#evacuate-and-restore-cluster-members https://linuxcontainers.org/incus/docs/main/howto/cluster_ma...
- rascul 1y agoA little bit of context about where Incus came from: https://lwn.net/Articles/940684/ https://lwn.net/Articles/940684/
- Lightkey 1y agoNot to be confused with the cirrus7 incus[0], which are fanless PC models based on the ASRock DeskMini series that I'm using right now. [0] https://www.cirrus7.com/produkte/cirrus7-incus/ https://www.cirrus7.com/produkte/cirrus7-incus/
- mrbluecoat 1y agoor ear bone: https://en.m.wikipedia.org/wiki/Incus https://en.m.wikipedia.org/wiki/Incus
- burnt-resistor 1y agoNothing about resource (net, io, disk, cpu) isolation, limits, priorities, or guarantees. Not the same as a type 1 hypervisor. These qualities are needed to run things safely and predictably in the real world™, at scale. Also, accounting and multitenancy if it's going to be used as some sort VAR or VPS offering.
- loloquwowndueo 1y agoTook a few seconds of googling to find this: https://linuxcontainers.org/incus/docs/main/reference/instance_options/#instance-options-limits https://linuxcontainers.org/incus/docs/main/reference/instan...
- tok1 1y agoFun fact, Incus is being used as underlying infrastructure for the NorthSec CTF, i.e. in an "as hostile as it can get" environment. If you have close to a hundred teams of hackers on your systems trying to break stuff, I think it speaks for Incus and its capabilities regarding isolation and limits. In case you are interested, Zabbly has some interesting behind-the-scenes on Youtube (not affiliated).
- mdaniel 1y agoI would guess <https://www.youtube.com/watch?v=7A1yrLRNIp0 https://www.youtube.com/watch?v=7A1yrLRNIp0> is a good starting point "Looking at the NorthSec infrastructure" from April, 2024 The YT description also points to https://github.com/zabbly/incus https://github.com/zabbly/incus
- maple3142 1y agoIf being used in a CTF counts, then running latest docker with no extra privilege and non-root user on a reasonably up-to-date kernel meets the definition of secure I think. At least for what I have seen, this kind of infrastructure is pretty common in CTF.
- goku12 1y agoIncus supports Qemu/KVM VMs. And KVM is arguably a Type 1 hypervisor since it's part of the Linux kernel. So I guess it qualifies?
- danofsteel32 1y agoIncus is great when developing ansible playbooks. The main benefit for me over docker/podman is systemd works out of the box in incus containers.
- mekster 1y agoNot to mention the easy to use web UI.
- deleted 1y ago[deleted]
- anonfordays 1y agoWhat makes it better than Vagrant for this use-case?
- fuzzy2 1y agoDoesn't Vagrant spin up full VMs? Incus/LXD/LXC is about system containers. So like Docker, but with a full distro including init system running inside the container. They are much faster to spin up and have the best resource sharing possible.
- goku12 1y agoVagrant has a provider plugin for lxc. Please check my reply to the parent commenter for more explanation.
- mdaniel 1y agohttps://github.com/hashicorp/vagrant/blob/v2.4.7/LICENSE https://github.com/hashicorp/vagrant/blob/v2.4.7/LICENSE for one thing
- goku12 1y agoVagrant is not the right comparison against Incus for this use case. Vagrant is used to spin up VM or system container instances that are configured for software development and testing. But vagrant doesn't create those VMs or containers by itself. Instead, it depends on virtualization/container providers like VMware, Virtualbox, libvirt or lxc. In fact, you could create a provider plugin for vagrant to use Incus as its container/VM back end. (I couldn't find any, though.)
- actinium226 1y agoI went through the online tutorial, but I'm not really seeing how it's different from docker?
- skydhash 1y agoInstead of ephemeral containers, you have instances that are like VM (and incus can manage VM via qemu), so pretty much everything you would use a VM for, but if you do not need the kernel separation. It's more similar to FreeBSD jails than to docker.
- Levitating 1y agoIt's a difference between system containers and application containers. LXC containers used in incus run their own init, they act more like a VM. However incus can also execute actual VMs via libvirt and since recently even OCI containers like docker.
- CoolCold 1y agoif your question is genuine, then the simple answer would be full system inside container of LXD/Incus (I prefer to use term VE for Virtual Environment to distinguish from barebone containers of Docker style). The full system, means, all your standard stuff works in expected way - crons, systemd units, sshd, multiple users, your regular backup solutions and so on. As well, that "system" can be dumped/exported/backuped/snapshotted as a whole - very much similar like you do with your vSphere/Qemu or whatever you use in your datacenters as hypervisor. Foreseeing the question - yep, you can run Docker inside LXD/Incus VEs. In practical terms, that makes much simple when you need to give some dev team (who are of course not anywhere known to do sane things) access to environment with Docker access (which in 99% cases means that host's root level access is exposed).
- oulipo 1y agoIs there some kind of Terraform/Pulumi integration to make it easy to deploy stuff to some VM running Incus for my deployments? Or I'm missing the point of what Incus is for?
- Levitating 1y agoYou could use cloud-init
- goku12 1y agoIncus is like a cloud management software - especially in cluster mode. It has management API like many cloud services. So, yes, there's a terraform provider for Incus, which can be used to configure and provision instances. Guest setup can be managed using cloud-init. Ansible is also an alternative option for this.
- mdaniel 1y agoOr even one of each, since cloud-init has an ansible hook: https://cloudinit.readthedocs.io/en/latest/reference/modules.html#ansible https://cloudinit.readthedocs.io/en/latest/reference/modules... for something like ansible-pull behavior
- goku12 1y agoYes. Thanks! The entire cloud computing ecosystem is a hot mess of numerous tools with countless interdependencies.
- oulipo 1y agoVery clear! And would it make sense to run it on a GCP VM, and use it as a "nicer docker-compose" ? Or that would entirely miss the point?
- goku12 1y agoIncus is sort of like GCP's own resource management software, though GCP does a lot more than what Incus does. So you'd often be using Incus like a self-hosted alternative to GCP. Meanwhile, running Incus inside GCP VM(s) should be possible, though I haven't tried it and can't confirm it. Incus can manage system containers - containers that behave like VMs running full distros, except for the kernel. But keep in mind that Incus is more like docker than docker-compose. You will need a different tool to provision and configure all those containers over Incus's API (docker-compose does this for application containers over the docker API). As mentioned before, that could be Terraform/OpenTofu, cloud-init and Ansible. Or you could even write a script to do it over the API. I have done this using Python.
- tcfhgj 1y agoThe only tool I found which allows to easily spin up pre-configured VMs without any gui hassle
- pm2222 1y agoShould lxc user migrate to incus?
- Levitating 1y agoan lxd user should
- jenders 1y agoLXD is actually pronounced “lex-d” by the community and similarly, LXC is “lex-c”.
- goku12 1y agoShort answer: No. Long answer: Depends upon what you use lxc for. Incus is not a replacement for lxc. It's an alternative for LXD (LXD project is still active). Both Incus and LXD are built upon liblxc (the library version of lxc) and provide a higher level user interface than lxc (eg: projects, cloud-init support, etc). However, lxc gives you fine grained control over container options (this is sort of like flatpak and bubblewrap). So, if you don't need the fine grained control of lxc, Incus may be a more ergonomic solution. PS: Confusingly enough, LXD's CLI is also named lxc.
- gavinray 1y agoCan someone explain the usecase for this? Is this for people who want to run their own cloud provider, or that need to manage the infrastructure of org-owned VM's? When would you use this over k8s or serverless container runtimes like Fargate/Cloudrun?
- Levitating 1y agoThere's no particular usecase, though I do know of a company whose entire infrastructure is maintained within incus. I personally use it mostly for deploying a bunch of system containers and some OCI containers. But anyone who uses LXC, LXD, docker, libvirt, qemu etc. could potentially be interested in Incus. Incus is just an LXD fork btw, developed by Stephane Graber.
- clvx 1y agoWho also developed LXD and contributed to LXC. I wouldn’t say it’s just a fork but a continuation of the project without Canonical.
- Levitating 1y agoYou're right, I should've worded it differently.
- anonfordays 1y ago[flagged]
- anonfordays 1y ago[flagged]
- pxc 1y agoSystem container tech like Incus powers efficient Linux virtualization environments for developers, so that you can have only one VM but many "machines". OrbStack machines on macOS work like this, and the way WSL works is similar (one VM and one Linux kernel, many guests sharing that kernel via system containers).
- sigmonsays 1y agothe features worth mentioning imho are the different storage backends and their features. Using btrfs, lvm or zfs there is some level of support of thin copy provisioning and snapshotting. I believe btrfs/zfs have parity in terms of supported operations. Cheap snapshots and provisioning of both containers and VMs using the same tool is pretty awesome. I personally use lxd for running my homelab VMs and containers
- mdaniel 1y agoI first learned about this because colima supports it: https://github.com/abiosoft/colima#incus https://github.com/abiosoft/colima#incus
- mathfailure 1y agoTheir site doesn't open, that's a poor sign.
- mdaniel 1y agoOnly if they run their website on Incus
- darqis 1y agotried it, was buggy, removed and never looked back.