7 ms·

LXD (Canonical's daemon/API front end to lxc containers) is great -- as long as you aren't using the god awful snap package they insist on. The snap is probably

LXD (Canonical's daemon/API front end to lxc containers) is great -- as long as you aren't using the god awful snap package they insist on. The snap is probably fine for single dev machines, but it has zero place in anything production. This is because canonical insists on auto-updating and refreshing the snap at random intervals, even when you pin to a specific version channel. Three times I had to manually recover a cluster of lxd systems that broke during a snap refresh because the cluster couldn't cope with the snaps all refreshing at once. Going forward we built and installed lxd from source.

rambojazz 5y ago
I was bitten by LXD auto-updating as well. Server was down and I couldn't understand why since I hadn't changed anything.
alyandon 5y ago
I got so annoyed with snapd that I finally patched the auto-update functionality to provide control via environment variable. It's ridiculous that this is what I have to personally go through in order to maintain control of when updates are applied on my own systems. If enough people were to ever decide to get together and properly fork snapd and maintain the patched version I'd totally dedicate time to helping out. https://gist.github.com/alyandon/97813f577fe906497495439c37dacf37 https://gist.github.com/alyandon/97813f577fe906497495439c37d...
- bamboozled 5y ago
  Agreed, it's the stupidest thing around, especially because Linux is about having control of your own systems, then they force updates down your throat, it goes against a lot of the things I use Linux for. I just avoid snap and Ubuntu wherever possible now.
- agartner 5y ago
  We blocked the annoying snapd autoupdate behavior by setting a http proxy to a nonexistent server. Whenever we had a maintenance window we would unset the proxy, allow the update, then set the set the nonexistent proxy server again. Very annoying.
  - djbusby 5y ago
    this feels both clever and stupid at the same time - not you but the software games you have to play.
    - deleted 5y ago
      [deleted]
    - jethro_tell 5y ago
      theres a lot of wasted effort in the games we play to make software work in prod.
      - hkt 5y ago
        From being a production engineer for many years I can only agree: some of the brightest people of our generation spend most of their time compensating for the poor design choices of others. Not sure if this is better or worse than advertising.
  - alyandon 5y ago
    That certainly works too but with my approach you can run "snap refresh" manually whenever you feel like updating.
- tomrod 5y ago
  > If enough people were to ever decide to get together and properly fork snapd and maintain the patched version I'd totally dedicate time to helping out. Is that the gist of flatpak?
baggy_trough 5y ago
Yeah, it's truly terrible. I've had downtime from this as well.
whitepoplar 5y ago
Just curious--how do you use LXD in production? It always struck me as something very neat/useful for dev machines, but I had trouble imagining how it would improve production workloads.
- deleted 5y ago
  [deleted]
stingraycharles 5y ago
Makes you wonder whether Canonical has any idea about operating servers. Auto-updating packages is the last thing you want. Doing that for a container engine, without building in some jitter to avoid the scenario you described is absolutely insane. Who even uses snap in production? If I squint my eyes I can see the use for desktops, but why insist on it for server technologies as well?
- curt15 5y ago
  Canonical would gladly hand back full control of updates if you pay them for an "enterprise edition" snap store. https://ubuntu.com/core/docs/store-overview#:~:text=Brand%20stores&text=A%20brand%20store%20allows%20vendors,developers%20or%20a%20specific%20list https://ubuntu.com/core/docs/store-overview#:~:text=Brand%20....
  - stingraycharles 5y ago
    And even then, controlling the package versions is only one of the problems. The bigger problem which isn’t solved with this (as far as I can tell) is not having the machines automatically update due to how the snap software works.
pkulak 5y ago
Not even kidding, a huge part of what made me move to Arch was that it's one of the few distros that packages LXD. Apparently it's a pain, but I'm forever grateful!
- sreevisakh 5y ago
  Alpine is another distro that packages LXD. I have Arch on my workstation, but I'm not confident about securing Arch on a server. Alpine, on the other hand, is very much suited to be an LXD host. It's tiny, runs entirely from RAM and can tolerate disk access failures. Modifications to host filesystem won't persist after reboot, unless the admin 'commits' them. The modifications can be reviewed before a commit - so it's easy to notice any malicious modifications. I also heard a rumor that they are integrating ostree. The only gripe I have with Alpine is its installation experience. Like Arch, Alpine has a DIY type installation (but a completely different style). But unlike Arch, it isn't easy to properly install Alpine without a lot of trial and error. Alpine documentation felt like it neglects a lot of important edge cases that trip you up during installation. Arch wiki is excellent on that aspect - they are likely to cover every misstep or unexpected problem you may encounter during installation.
- rascul 5y ago
  openSUSE packages lxd
- hpb42 5y ago
  I did not know there were so few distros packaging LXD [0]. It does not come as a surprise that Canonical would not endorse non-snapded LXD though. And it is sad. [0] https://repology.org/project/lxd/versions https://repology.org/project/lxd/versions
rlpb 5y ago
> This is because canonical insists on auto-updating and refreshing the snap at random intervals, even when you pin to a specific version channel. You can control snap updates to match your maintenance windows, or just defer them. Documentation here: https://snapcraft.io/docs/keeping-snaps-up-to-date#heading--controlling-updates https://snapcraft.io/docs/keeping-snaps-up-to-date#heading--... What you cannot do without patching is defer an update for more than 90 days. [Edit: well, you sort of can, by bypassing the store and "sideloading" instead: https://forum.snapcraft.io/t/disabling-automatic-refresh-for-snap-from-store/707/269 https://forum.snapcraft.io/t/disabling-automatic-refresh-for...]
- alyandon 5y ago
  Why would any sysadmin that maintains a fleet of servers with varying maintenance windows (that may or may not be dependent on the packages installed) want to feed snapd a list of dates and times to basically ask it pretty please not to auto-update snaps? It is much easier to give sysadmins back the power they've always had to perform updates when it is reasonable to do so on their own schedule without having to inform snapd of what that schedule is. I really don't appreciate being told by Canonical that I have to jump through additional hoops and spend additional time satisfying snapd to maintain the systems under my control.
  - donmcronald 5y ago
    My initial reaction was that having schedules upgrades like that would be great, but then 5 seconds later I realized it's much better suited to Cron, SystemD, Ansible (etc.). I think the reason for auto-updates like this is because selling the control back to us is the business plan. It's the same thing Microsoft does.
khimaros 5y ago
it looks like there has been considerable progress packaging in debian bookworm: https://blog.calenhad.com/posts/2022/01/lxd-packaging-report-new-years/ https://blog.calenhad.com/posts/2022/01/lxd-packaging-report...
CSDude 5y ago
I had a huge argument in 2015 with a guy that wanted to move our every custom .deb package (100+) to Snap, because they had talked with Canonical and it would be the future, Docker would be obsolete. Main argument was to make distribution easier to worker/headless/server machines. Not that Docker is a direct replacement, but Snap is an abomination. They are mostly out of date, most of them requires system privilleges, unstable and the way they mount compressed rootfs is making starts very slow, even on a good machine. That all being said, LXD is great way to run non-ephemeral containers that behave more like a VM. Also checkout multipass, by Canonical that also makes spinning up Ubuntu VMs as easy as Docker.
- zibzab 5y ago
  pro tip: you can use lxc/lxd to run VMs under the same infrastructure. I belive it is something as easy as lxc launch -vm ubuntu/20.04 myvm
  - arjvik 5y ago
    What VM software does this make use of? KVM?
    - sreevisakh 5y ago
      Yes. LXD uses Qemu/KVM for VMs.
  - CSDude 5y ago
    Wow, cannot wait to try it! Thanks for the tip.
warent 5y ago
On my Ubuntu 20 server, I tried setting up microk8s with juju using LXD and my god the experience was horrendous. One bug after another after another after another after another. Then I upgraded my memory and somehow snap/LXD got perma stuck in an invalid state. The only solution was to wipe and purge everything related to snap/LXD. After that I setup minikube with a Docker backend. It all worked instantly, perfectly aligned with my mental model, zero bugs, zero hassle. Canonical builds a great OS, but their Snap/VM org is... not competitive.
- zibzab 5y ago
  Note that Canonical has special tools for installing and configuring that stuff (conjureup or something)
  - warent 5y ago
    That's what I was using, and is largely how Juju is used
- NorwegianDude 5y ago
  That's strange. I've had no issues using microk8s. Super easy to set up on Ubuntu 20.04.
conradfr 5y ago
Maybe two years ago I wanted to use LXD on a fresh Ubuntu server (after testing it locally). First they had just moved it to Snap which was not a great install experience compared to good old apt-get, and then all my containers had no IPv4 because of systemd for a reason I can't remember. After two or three tries I just gave up, installed CapRover (still in use today) and have not tried again since.
- NorwegianDude 5y ago
  The install experience should be the same. Using apt install lxd should give you snap. I've installed LXD quite a few times with different storage and network setups without any issues, so I do find it strange that it didn't work for you.
_448 5y ago
> The snap is probably fine for single dev machines It is not good even on single dev machines.
- curt15 5y ago
  I normally use podman for my dev machine. Unlike LXD, it can run rootless.
rajishx 5y ago
I am simple man with simple need, I am perfectly happy with a distro as long I have my editor my terminal and my browser. I could not bear the snaps on ubuntu always coming back and hard to disable on every update, I gave up and just switched to arch and happy to have control on my system again. I had a lot of crash running on Ubuntu when running huge rust based test suite doing a lot of IO (on btrfs), never had that issue on arch. not sure why, not sure how I can even debug it (full freeze, nothing in systemd logs) so I guess I just gave up.....
anothernewdude 5y ago
Canonical always backs the wrong horse. Unity, Snap, Mir, Upstart, etc. etc.

rambojazz 5y ago

I was bitten by LXD auto-updating as well. Server was down and I couldn't understand why since I hadn't changed anything.

alyandon 5y ago

I got so annoyed with snapd that I finally patched the auto-update functionality to provide control via environment variable. It's ridiculous that this is what I have to personally go through in order to maintain control of when updates are applied on my own systems. If enough people were to ever decide to get together and properly fork snapd and maintain the patched version I'd totally dedicate time to helping out. https://gist.github.com/alyandon/97813f577fe906497495439c37dacf37 https://gist.github.com/alyandon/97813f577fe906497495439c37d...

bamboozled 5y ago
Agreed, it's the stupidest thing around, especially because Linux is about having control of your own systems, then they force updates down your throat, it goes against a lot of the things I use Linux for. I just avoid snap and Ubuntu wherever possible now.
agartner 5y ago
We blocked the annoying snapd autoupdate behavior by setting a http proxy to a nonexistent server. Whenever we had a maintenance window we would unset the proxy, allow the update, then set the set the nonexistent proxy server again. Very annoying.
- djbusby 5y ago
  this feels both clever and stupid at the same time - not you but the software games you have to play.
  - deleted 5y ago
    [deleted]
  - jethro_tell 5y ago
    theres a lot of wasted effort in the games we play to make software work in prod.
    - hkt 5y ago
      From being a production engineer for many years I can only agree: some of the brightest people of our generation spend most of their time compensating for the poor design choices of others. Not sure if this is better or worse than advertising.
- alyandon 5y ago
  That certainly works too but with my approach you can run "snap refresh" manually whenever you feel like updating.
tomrod 5y ago
> If enough people were to ever decide to get together and properly fork snapd and maintain the patched version I'd totally dedicate time to helping out. Is that the gist of flatpak?

baggy_trough 5y ago

Yeah, it's truly terrible. I've had downtime from this as well.

whitepoplar 5y ago

Just curious--how do you use LXD in production? It always struck me as something very neat/useful for dev machines, but I had trouble imagining how it would improve production workloads.

deleted 5y ago
[deleted]

stingraycharles 5y ago

Makes you wonder whether Canonical has any idea about operating servers. Auto-updating packages is the last thing you want. Doing that for a container engine, without building in some jitter to avoid the scenario you described is absolutely insane. Who even uses snap in production? If I squint my eyes I can see the use for desktops, but why insist on it for server technologies as well?

curt15 5y ago
Canonical would gladly hand back full control of updates if you pay them for an "enterprise edition" snap store. https://ubuntu.com/core/docs/store-overview#:~:text=Brand%20stores&text=A%20brand%20store%20allows%20vendors,developers%20or%20a%20specific%20list https://ubuntu.com/core/docs/store-overview#:~:text=Brand%20....
- stingraycharles 5y ago
  And even then, controlling the package versions is only one of the problems. The bigger problem which isn’t solved with this (as far as I can tell) is not having the machines automatically update due to how the snap software works.

pkulak 5y ago

Not even kidding, a huge part of what made me move to Arch was that it's one of the few distros that packages LXD. Apparently it's a pain, but I'm forever grateful!

sreevisakh 5y ago
Alpine is another distro that packages LXD. I have Arch on my workstation, but I'm not confident about securing Arch on a server. Alpine, on the other hand, is very much suited to be an LXD host. It's tiny, runs entirely from RAM and can tolerate disk access failures. Modifications to host filesystem won't persist after reboot, unless the admin 'commits' them. The modifications can be reviewed before a commit - so it's easy to notice any malicious modifications. I also heard a rumor that they are integrating ostree. The only gripe I have with Alpine is its installation experience. Like Arch, Alpine has a DIY type installation (but a completely different style). But unlike Arch, it isn't easy to properly install Alpine without a lot of trial and error. Alpine documentation felt like it neglects a lot of important edge cases that trip you up during installation. Arch wiki is excellent on that aspect - they are likely to cover every misstep or unexpected problem you may encounter during installation.
rascul 5y ago
openSUSE packages lxd
hpb42 5y ago
I did not know there were so few distros packaging LXD [0]. It does not come as a surprise that Canonical would not endorse non-snapded LXD though. And it is sad. [0] https://repology.org/project/lxd/versions https://repology.org/project/lxd/versions

rlpb 5y ago

> This is because canonical insists on auto-updating and refreshing the snap at random intervals, even when you pin to a specific version channel. You can control snap updates to match your maintenance windows, or just defer them. Documentation here: https://snapcraft.io/docs/keeping-snaps-up-to-date#heading--controlling-updates https://snapcraft.io/docs/keeping-snaps-up-to-date#heading--... What you cannot do without patching is defer an update for more than 90 days. [Edit: well, you sort of can, by bypassing the store and "sideloading" instead: https://forum.snapcraft.io/t/disabling-automatic-refresh-for-snap-from-store/707/269 https://forum.snapcraft.io/t/disabling-automatic-refresh-for...]

alyandon 5y ago
Why would any sysadmin that maintains a fleet of servers with varying maintenance windows (that may or may not be dependent on the packages installed) want to feed snapd a list of dates and times to basically ask it pretty please not to auto-update snaps? It is much easier to give sysadmins back the power they've always had to perform updates when it is reasonable to do so on their own schedule without having to inform snapd of what that schedule is. I really don't appreciate being told by Canonical that I have to jump through additional hoops and spend additional time satisfying snapd to maintain the systems under my control.
- donmcronald 5y ago
  My initial reaction was that having schedules upgrades like that would be great, but then 5 seconds later I realized it's much better suited to Cron, SystemD, Ansible (etc.). I think the reason for auto-updates like this is because selling the control back to us is the business plan. It's the same thing Microsoft does.

khimaros 5y ago

it looks like there has been considerable progress packaging in debian bookworm: https://blog.calenhad.com/posts/2022/01/lxd-packaging-report-new-years/ https://blog.calenhad.com/posts/2022/01/lxd-packaging-report...

CSDude 5y ago

I had a huge argument in 2015 with a guy that wanted to move our every custom .deb package (100+) to Snap, because they had talked with Canonical and it would be the future, Docker would be obsolete. Main argument was to make distribution easier to worker/headless/server machines. Not that Docker is a direct replacement, but Snap is an abomination. They are mostly out of date, most of them requires system privilleges, unstable and the way they mount compressed rootfs is making starts very slow, even on a good machine. That all being said, LXD is great way to run non-ephemeral containers that behave more like a VM. Also checkout multipass, by Canonical that also makes spinning up Ubuntu VMs as easy as Docker.

zibzab 5y ago
pro tip: you can use lxc/lxd to run VMs under the same infrastructure. I belive it is something as easy as lxc launch -vm ubuntu/20.04 myvm
- arjvik 5y ago
  What VM software does this make use of? KVM?
  - sreevisakh 5y ago
    Yes. LXD uses Qemu/KVM for VMs.
- CSDude 5y ago
  Wow, cannot wait to try it! Thanks for the tip.

warent 5y ago

On my Ubuntu 20 server, I tried setting up microk8s with juju using LXD and my god the experience was horrendous. One bug after another after another after another after another. Then I upgraded my memory and somehow snap/LXD got perma stuck in an invalid state. The only solution was to wipe and purge everything related to snap/LXD. After that I setup minikube with a Docker backend. It all worked instantly, perfectly aligned with my mental model, zero bugs, zero hassle. Canonical builds a great OS, but their Snap/VM org is... not competitive.

zibzab 5y ago
Note that Canonical has special tools for installing and configuring that stuff (conjureup or something)
- warent 5y ago
  That's what I was using, and is largely how Juju is used
NorwegianDude 5y ago
That's strange. I've had no issues using microk8s. Super easy to set up on Ubuntu 20.04.

conradfr 5y ago

Maybe two years ago I wanted to use LXD on a fresh Ubuntu server (after testing it locally). First they had just moved it to Snap which was not a great install experience compared to good old apt-get, and then all my containers had no IPv4 because of systemd for a reason I can't remember. After two or three tries I just gave up, installed CapRover (still in use today) and have not tried again since.

NorwegianDude 5y ago
The install experience should be the same. Using apt install lxd should give you snap. I've installed LXD quite a few times with different storage and network setups without any issues, so I do find it strange that it didn't work for you.

_448 5y ago

> The snap is probably fine for single dev machines It is not good even on single dev machines.

curt15 5y ago
I normally use podman for my dev machine. Unlike LXD, it can run rootless.

rajishx 5y ago

I am simple man with simple need, I am perfectly happy with a distro as long I have my editor my terminal and my browser. I could not bear the snaps on ubuntu always coming back and hard to disable on every update, I gave up and just switched to arch and happy to have control on my system again. I had a lot of crash running on Ubuntu when running huge rust based test suite doing a lot of IO (on btrfs), never had that issue on arch. not sure why, not sure how I can even debug it (full freeze, nothing in systemd logs) so I guess I just gave up.....

anothernewdude 5y ago

Canonical always backs the wrong horse. Unity, Snap, Mir, Upstart, etc. etc.