8 ms·

When root on ZFS breaks on Arch Linux

65 points by LinuxBender 4y ago

pmarreck 4y ago
… it still continues to work on NixOS after rebooting into the previous generation. ;)
nullwarp 4y ago
Ah man been there, done that! I now specifically use the dated archive mirrors for arch mirrorlist as well as the zfs-linux AUR package over the DKMS. Been a lot less of a headache dealing with it that way. I just bump the date up to as far forward as the two packages were aligned and off ya go. I do wish the licensing would get sorted out because I love ZFS but dealing with always having to juggle that dependency is annoying. Also mkinitcpio should be much more aggressive on failing on errors.
- commandersaki 4y ago
  I'm on the zfs-linux AUR package in the archzfs repo and not DKMS. I'm still waiting for zfs packages for the latest kernel but they haven't been cut yet; I'm wondering if this has something to do with OPs problems that are being encountered, and so there's a bit of a delay on the next zfs release.
pm2222 4y ago
The part, where wireless nic is not allowed to be enslaved to a bridge, I believe that it was allowed at one point in the past. The root cause is that the wireless frame has 3-4 MAC addresses versus Ethernet just 2. But somehow software should be able to do something about it. Why not?
- fuzzy2 4y ago
  This can work with MAC address translation (NAT, but on a lower level) and connection tracking. Not exactly a great solution. Alternatively, you can have one association with the access point per device, which is what some off-the-shelf repeaters do. Drivers often limit these to very low numbers. This wasn't possible in the past either. There's WDS (4-address mode) and some drivers that did what I describe above under the hood. That's it.
- 3np 4y ago
  That part of the story was a plot hole for my. Why did the protagonist not just set up a forwarding proxy, or IP forwarding, on the laptop? Tor is convenient. https_proxy=socks://laptop:9050 curl ...
t344344 4y ago
I had similar issue with BTRFS thanks to new faulty DDR4. - CRC checksum for some blocks on fs was wrong - I discovered faulty DDR4, replaced it, without knowing FS corruption - system started randomly kernel panicking - no msg in logs (fs died with kernel) - eventually I experienced kernel panic while at computer, and seen msg on screen (broken checksum or something) - there was no way to repair filesystem. I had to recover as much as data, throw away some files and start over on fresh - I use BTRFS snapshot streaming in backups, wrong checksums propagated to backups. I had to restore from 1 week old backups - Since than I use RSYNC backups as well - And I test all PCs with memtest monthly overnight. It is also a good stress test for CPU.
- nicolaslem 4y ago
  > And I test all PCs with memtest monthly overnight. It is also a good stress test for CPU. I think that the consensus in the PC enthusiast community is that memtest is quite dated and not very good at finding instability in modern hardware.
  - tcrenshaw 4y ago
    what's the community recommendation these days?
  - Marsymars 4y ago
    I’m not particularly up-to-date on that subject, so someone else can probably provide more info, but FWIW PassMark’s MemTest86 has had ongoing development, while the open source MemTest86+ was effectively a dead project for nearly a decade before some recent activity - so there’s a fair amount of difference between the two.
  - t344344 4y ago
    I am not sure what consensus is, but I use this: https://memtest.org/ https://memtest.org/
imrehg 4y ago
Yup, this popped up for me about a month ago on a Raspberruy Pi system, fortunately not on the root, but still. On ARM the kernel_neon_begin / kernel_neon_end symbols were made GPL only[0] which is quite puzzling on many levels. The discussion of the reported issue on the OpenZFS GitHub is very educational, both on the maintainers side (useful, sensible), and the other commenters syde (kinda "thermonuclear")[1] In the meantime I just downgraded the kernel as well, fortunately still had it in the pacman cache; then pinned it for now. Sent an email to to the patch creator / signer-off too, to feed back on that "this is not expected to be disruptive to existing users." part of the commit (as in [0]). [0]: https://lore.kernel.org/all/20221107170747.276910-1-broonie@kernel.org/ https://lore.kernel.org/all/20221107170747.276910-1-broonie@... [1]: https://github.com/openzfs/zfs/issues/14555 https://github.com/openzfs/zfs/issues/14555
InvaderFizz 4y ago
> ERROR: modpost: GPL-incompatible module zfs.ko uses GPL-only symbol 'bio_start_io_acct' > ERROR: modpost: GPL-incompatible module zfs.ko uses GPL-only symbol 'bio_end_io_acct_remapped' I don't understand why there isn't a flag to ignore these and accept a tainted kernel so that this type of issue doesn't break things. Then again, pacman should have aborted the process as well.
- denotational 4y ago
  > ignore these and accept a tainted kernel That isn’t possible as I understand it. Loading any non-GPL module will mark the Kernel as tainted, but that’s not the issue here. The problem is that the ZFS module uses Kernel symbols that are marked as GPL-only, meaning that the Kernel will refuse to load the ZFS module as it isn’t marked as a GPL module. There isn’t any way to bypass this short of recompiling the Kernel to remove this restriction. The legal status of this is uncertain, since a number of questions naturally arise: 1. Is it acceptable to mark a Kernel module as a GPL module (this is done via a macro in the source) but release it under a license that is not GPL-compatible? 2. Is it acceptable to recompile the Kernel to remove this restriction? As far as I know neither of these have ever been tested, and there are differing opinions.
  - danarmak 4y ago
    If a kernel with the restriction removed is compiled locally and is never redistributed, why would it be illegal?
    - tomatocracy 4y ago
      It wouldn't. Moreover I can't see that there is any reason that redistribution of such a kernel would be against GPLv2, the licence under which the kernel is distributed.
      - denotational 4y ago
        > It wouldn't. Are you a lawyer? Asking in good faith, because if you are then this is the first qualified opinion I’ve seen on the matter. Everything else has been people claiming to have received legal opinions, see for example the email from Linus in my sibling comment.
dpwm 4y ago
I have the kernel and zfs modules in IgnorePkg in /etc/pacman.conf. This way I have to update them explicitly. I usually have to manually install a kernel from archive.archlinux.org/packages but I find this much less fragile than DKMS.
DvdGiessen 4y ago
A workaround that works locally (and only locally) is to build zfs-dkms (AUR) yourself and modify it so it doesn't break on these GPL-only symbols. While the licenses would forbid distributing it as such, you can do so on your own machine just fine. And if your root is on ZFS, make a snapshot before updating! I set up a pacman hook[0] which runs zfs_autobackup[1] which automatically manages snapshots, so I can always easily roll back to a non-broken state. The ZFSBootMenu[2] bootloader makes that extremely fast without even needing a bootable USB-drive. :) [0]: https://wiki.archlinux.org/title/Pacman#Hooks https://wiki.archlinux.org/title/Pacman#Hooks [1]: https://github.com/psy0rz/zfs_autobackup https://github.com/psy0rz/zfs_autobackup [2]: https://zfsbootmenu.org/ https://zfsbootmenu.org/
fuzzy2 4y ago
If you're running Linux with ZFS, make sure to have a ZFS-enabled live medium available. The Arch ISO is easy to modify. Various variants with ZFS exist around the net. I believe Ubuntu also ships with ZFS nowadays. I think NixOS too? Not entirely sure, it's been a while. If you go custom, you can also add drivers/firmware for whatever exotic hardware you have!
- tripdout 4y ago
  Yes, NixOS also ships ZFS in their ISO. It's been super easy coming from ZFS on Arch to NixOS, no hassles with adding custom repos, using DKMS, etc.
- e-v 4y ago
  The script available there: https://github.com/eoli3n/archiso-zfs https://github.com/eoli3n/archiso-zfs makes it extremely easy to add ZFS support to any Arch ISO after it has booted. You can copy any standard ISO to a USB drive, boot off it, then run `curl -s https://raw.githubusercontent.com/eoli3n/archiso-zfs/master/init https://raw.githubusercontent.com/eoli3n/archiso-zfs/master/... | bash` and you'll have ZFS support in a few seconds, without having anything to worry about.
  - rcthompson 4y ago
    > without having anything to worry about. Nothing to worry about except network access, of course, which was the issue in this case. Much better to have it not with ZFS support already enabled.
    - e-v 4y ago
      Indeed (I guess you meant "much better to have it with ZFS support already enabled"). I simply mentioned this for people unaware of the existence of this script who don't have network issues with the Arch ISOs.
ratorx 4y ago
This is one of the things NixOS is good for. There can be a special version of Linux that is only updated once some regression test on ZFS passes: `config.boot.zfs.package.latestCompatibleLinuxPackages` That way the main maintainer can still follow HEAD, whilst the ZFS maintainer can make sure that Linux is updated for ZFS once it’s been tested (or even automatically, but I don’t know if it’s that advanced yet).
- SuperSandro2000 4y ago
  And even if you don't use that, the build just fails early without consequences.
- aftbit 4y ago
  I see a lot of smart people recommending Nix, but whenever I look at it, it seems like there are a ton of sneaky config options that one must get right to get a working system...
  - ratorx 4y ago
    It depends (and not any more than applications that need sneaky config). Simple systems can have quite simple options. At the very least you can take a working copy and figure out exactly which configuration is necessary. Hard to do that without NixOS. Especially with nixos-option. And the options are all documented in a central place. The module definitions using the options are also “easy” to search through (Language is a bit complicated, but not in a way that grep doesn’t work).
  - assbuttbuttass 4y ago
    nixos-generate-config usually produces a working config out of the box. And if you need to change some obscure config options, everything is well documented with "man configuration.nix"
    - ElectricalUnion 4y ago
      I like https://search.nixos.org/options https://search.nixos.org/options , but while nicer it enters the "need somewhat working system to fix the system" bootstraping issues.
throw0101b 4y ago
Does Arch Linux have boot environments (or equivalent)? Boot Environments allows the system to be upgraded, while preserving the old system environment in a separate ZFS dataset. * https://man.freebsd.org/cgi/man.cgi?beadm https://man.freebsd.org/cgi/man.cgi?beadm The beadm command is the user interface for managing ZFS Boot Environments (BEs). This utility is intended to be used by system administrators who want to manage multiple Oracle Solaris instances on a single system. * https://docs.oracle.com/cd/E86824_01/html/E54764/beadm-1m.html https://docs.oracle.com/cd/E86824_01/html/E54764/beadm-1m.ht... > A ZFS boot environment is a bootable clone of the datasets needed to boot the operating system. Creating a BE before performing an upgrade provides a low-cost safeguard: if there is a problem with the update, the system can be rebooted back to the point in time before the upgrade. * https://klarasystems.com/articles/managing-boot-environments/ https://klarasystems.com/articles/managing-boot-environments... Or perhaps: > In essence, ZFSBootMenu is a small, self-contained Linux system that knows how to find other Linux kernels and initramfs images within ZFS filesystems. When a suitable kernel and initramfs are identified (either through an automatic process or direct user selection), ZFSBootMenu launches that kernel using the kexec command. * https://github.com/zbm-dev/zfsbootmenu https://github.com/zbm-dev/zfsbootmenu
- boomboomsubban 4y ago
  Yes, but like every part of ZFS on Arch, not by default. https://aur.archlinux.org/packages/zfsbootmenu https://aur.archlinux.org/packages/zfsbootmenu
- drewg123 4y ago
  Indeed. In fact, my main reason for ZFS on root is boot environments. If something breaks, its trivial to roll back the rootfs as it was in the old BE, before the upgrade. And I run ZFS on FreeBSD.
- nubinetwork 4y ago
  I believe only ubuntu half-supported boot environments. I don't know of any other distro that even bothered to support zfs outside of supplying the package for users to install.
frankjr 4y ago
The main problem here is Pacman's handling of "PostTransaction" hooks (which in this case builds the zfs kernel module). The whole installation should have failed but it didn't. Individual hooks can specify "AbortOnFail" but it's available to "PreTransaction" hooks only.
zokier 4y ago
Like many fun adventures in Linux, this begins with adding some 3rd party repos (and/or packages) into the system. In this case doubly so because: > I'd probably have caught the red error if it wasn't followed by tons of log output from building AUR packges :) Although I do agree that noticing important messages from even plain pacman output can occasionally be difficult, although I haven't seen many other package managers do it any better. > But I can't, because I can only build a live ISO with Linux 6.2.8 easily, which is the original problem That feels bit weird situation though. I'd imagine being able to build downgraded images would be useful in general, so interesting that apparently its not so easy.
blywi 4y ago
I just ran into the same issue yesterday when I was updating one of my machines. Furtunately I caught the DKMS error when mkinitcpio ran after the upgrade: ==> dkms install --no-depmod zfs/2.1.9 -k 6.2.8-arch1-1 Error! Bad return status for module build on kernel: 6.2.8-arch1-1 (x86_64) Consult /var/lib/dkms/zfs/2.1.9/build/make.log for more information. ==> WARNING: `dkms install --no-depmod zfs/2.1.9 -k 6.2.8-arch1-1' exited 10 Issues like these are when being on a rolling-release distro might bite you. To mitigate the risk a bit, I always have the linux-lts kernel package installed on all of my Arch Linux machines, which provides a fallback option when the mainline kernel fails, or some major kernel issue surfaces. In my case with the LTS kernel at 6.1.21-1-lts, DKMS build of ZFS on LTS succeeded, so it would have been a fallback option to boot even if I would have missed the error for the mainline kernel DKMS. ==> dkms install --no-depmod zfs/2.1.9 -k 6.1.21-1-lts ==> depmod 6.1.21-1-lts I don't have my root on ZFS, I only use it for data storage, but if I ever did,I would make damn sure to build myself a live ISO with ZFS support and all drivers necessary for network access, ahead of time, in case I need to do an emergency repair. It doesn't need to always be on the latest kernel, just recent enough that you are able to mount all partitions and to chroot into the system. For this issue a workaround with Kernel 6.2.8 is discussed in the AUR comments for zfs-dkms: https://aur.archlinux.org/packages/zfs-dkms https://aur.archlinux.org/packages/zfs-dkms This would involve editing the PKGBUILD for ZFS to patch the license for the symbols to GPL, though I'm not sure about the legality of this approach.
- tzs 4y ago
  > For this issue a workaround with Kernel 6.2.8 is discussed in the AUR comments for zfs-dkms: https://aur.archlinux.org/packages/zfs-dkms https://aur.archlinux.org/packages/zfs-dkms This would involve editing the PKGBUILD for ZFS to patch the license for the symbols to GPL, though I'm not sure about the legality of this approach In the US there is 17 USC 117(a) which might apply: > Making of Additional Copy or Adaptation by Owner of Copy.—Notwithstanding the provisions of section 106, it is not an infringement for the owner of a copy of a computer program to make or authorize the making of another copy or adaptation of that computer program provided: > (1) that such a new copy or adaptation is created as an essential step in the utilization of the computer program in conjunction with a machine and that it is used in no other manner, or (Section 106 is the section that says you need permission from the copyright owner to make copies or adaptations).
SuperSandro2000 4y ago
This can't happen with NixOS.
Volundr 4y ago
Can someone explain the purpose of GPL only symbols to me, and why existing long standing APIs keep getting changed to GPL only? From the outside it certainly feels like all it does is (intentionally?) break things.
- drewg123 4y ago
  I think that is the point.. the kernel devs want to break 3rd party modules, because they don't like stuff distributed outside the kernel.
  - cesarb 4y ago
    > because they don't like stuff distributed outside the kernel. This mechanism doesn't prevent stuff from being distributed outside of the kernel at all, a GPL-licensed module could stay out of tree for a long time. They even have a documented mode in the kernel Makefile to compile out-of-tree modules, as described at "make help": "make -C path/to/kernel/src M=$PWD modules modules_install"
- georgyo 4y ago
  That is indeed the goal, they explicitly want to break compatibility. Their responses on the mailing list even say that. GPL or GTFO. It's a definitely annoying as a user. OpenZFS is much nicer than BTRFS, but you have to accept that breakage of the kernel devs may cause you issues.
  - Volundr 4y ago
    That's the inference I was trying to avoid making. It's frustrating because a big reason I use Linux is it gives me the freedom to build the system I want exactly how I want it, but a couple times a year I get this reminder that the devs are actually against me in this.
    - blibble 4y ago
      personally I will abandon Linux before I abandon ZFS
mst 4y ago
These sorts of events are why I know quite a number of people who do ZFS-for-everything on their FreeBSD systems but ZFS-for-everything-except-root on their Linux boxen. Seems like in a lot of cases having all of your data on ZFS gets you almost all of the awesomeness of having ZFS around with fairly minimal downsides. (deciding if this is true for any particular system is left as an exercise to the person designing the deployment)
- mustache_kimono 4y ago
  This is somewhat understandable. But to be clear, this is also an extraordinary set of circumstances -- a very recent, but marked stable kernel, on rolling release distro, with a back ported set of changes, breaking a dkms ZFS build. The user was notified and was able to take remedial action. My only issue would have been -- I'm not sure Arch made this very easy. FWIW I use Ubuntu LTS and ZFS on root works great. IMHO every distro should just ship ZFS. Ubuntu has proven this is just not a practical difficulty anymore. Let's quit the charade of making users compile their own kernel modules for each kernel release.
  - mhio 4y ago
    It's not a charade, it's the looming shadow of Oracle lawyers.
    - mustache_kimono 4y ago
      Who I'd note haven't done anything so far? This is what is so mysterious to me -- if this is the blockbuster case some on Reddit have made it out to be, why hasn't there been a suit against Canonical?
      - mhio 4y ago
        Does the "blockbuster case" line of thought you've seen suggest it should be easy for Oracle? From the differing legal opinions I've read it seems a legal quagmire. That's a quagmire that _both_ sides of the argument have to wade in to, just Oracle gets to choose if and when that happens. They might just be a bit gun shy after their last 10 year long open source case.
Asdrubalini 4y ago
This is one of the reason I use NixOS. I think it has the best ZFS support possible (at least on Linux)
Caligatio 4y ago
I dual booted my machine Win10 and Arch on ZFS Root. Admittedly I used Win10 like 99% so there was always a laundry list of updates when I booted into Arch. It seemed ZFS broke 20% of the time I updated. It was either the silly GRUB config fix that wasn't upstreamed getting overwritten or the ZFS module build somehow failed. I got so frustrated that I just nuked the whole install. I realize that ZFS is a weird bolt-on but the fact that kernel modules failing to build doesn't cause significant error messages and/or failure is silly. Arch is great but Arch on ZFS felt like my Gentoo days: I want to have fun using my computer, not constantly fiddling to make my computer usable.
- nubinetwork 4y ago
  It's not worth your time to use grub if you run zfs... the zfs module in grub only supports the ancient feature flag set from Solaris 9/10, its a miracle that it even works on Linux at all... It also doesn't help that grub is spearheaded by Oracle employees, and as such, they couldn't give a rats ass about maintaining the module. (And if they do, it'll be to add more Oracle Solaris features rather than zfs on linux/bsd/illumos)
mustache_kimono 4y ago
Oh would it be nice if ZFS wasn't CDDL or if the CDDL/GPL compatibility would be settled. For all practical purposes, it doesn't appear to be a problem. Ubuntu has shipped ZFS for years. As much bellyaching as we hear from some quarters about the hypothetical harms this has caused, no one has brought a suit, because my guess is -- it would be a pretty terrible case for a copyright holder from the Linux side. See [0] in addition to all the other ink that has been spilled re: this matter. Linus may not feel comfortable with ZFS included in mainline. I'm not sure distro maintainers need to feel the same way. [0]: https://www.networkworld.com/article/2301697/encouraging-closed-source-modules-part-1--copyright-and-software.html https://www.networkworld.com/article/2301697/encouraging-clo...
- nubinetwork 4y ago
  To be fair, Larry has the deepest pockets, and it takes some balls to stand up to the likelihood of a lawsuit.
  - mustache_kimono 4y ago
    > it takes some balls to stand up to the likelihood of a lawsuit. Likelihood? It's been years Canonical is doing what I've told be the Reddit-rati is a blatant copyright violation and ... crickets.
    - nubinetwork 4y ago
      Larry knows if he were to sue Shuttleworth/Canonical it could potentially bankrupt them both.
      - mustache_kimono 4y ago
        My guess is Larry could just buy Canonical and turn it into a parking lot. But maybe this is the long game -- Larry is hoping to trap IBM/Red Hat in a suit when they add ZFS to the kernel, but I'm not sure that's how copyright litigation works. If you forgo litigation for years and now decide to sue, I'm not sure you get to assert copyright now.
AviationAtom 4y ago
Root on ZFS just feels too much like playing with fire to me. Fixing an unbootable ext4 filesystem seems tricky enough already. Am I just in the dark on how to easily repair boot issues, especially on more modern filesystems? I feel like the OP probably had to put countless hours into working through all these issues to finally get back to a fully functioning environment.
- Gordonjcp 4y ago
  Why would you bother? It'll be quicker to just blow it away and reinstall.
  - AviationAtom 4y ago
    You're not concerned about all your system settings, installed applications, and any other data worthy of preservation on your root disk? Or am I misunderstanding how Root on ZFS works/boots?
    - Gordonjcp 4y ago
      No, why would I be? In what world would you even lose all that?
      - AviationAtom 4y ago
        What specifically would a reinstall cover then? Is it just a boot partition/stub (FAT?) that loads the root ZFS pool? It all seems like a drastically different architecture than I'm used to seeing on a system.
        Gordonjcp 4y ago
        Why would you keep /home on the same partition? Why would you be manually adding in apps after a reinstall? You just script this shit.
- AviationAtom 4y ago
nubinetwork 4y ago
Honestly, arch seems like a fragile distro. My experience with arch is: install the distro files, set up efistub booting, reboot... install updates a couple days later and wind up with a system that can't reboot, and if you force reboot it, it won't ever mount the root fs again. I've never had that happen on Gentoo, or debian/ubuntu or fedora, or suse. Only arch could be so temperamental to break itself on kernel upgrades.
- sgtnoodle 4y ago
  I've been running it for 8 years or so, and it's been fine. I have yet to need to do a reinstall on any of my computers. The rolling release has been great from a maintenance standpoint. There is the occasional problem, of course. Most recently, pipewire snuck its way in as a dependency. Not having been configured sanely, it caused some minor irritating bugs as it clashed with pulseaudio. My newest desktop's bios do tend to mess up my boot configuration after a firmware update. Windows boots directly instead of grub. It's a 10 second fix from the command line, whether in Windows or in Linux.
  - nubinetwork 4y ago
    I'm sure part of it for me has to be the efistub. I don't understand why anyone would want to bother with a bootloader in modern times because efi will let you pick and choose what to boot just fine without one. Clobbered bootloader is also why your system goes back to windows after an update (windows has always had issues cohabiting with other operating systems that weren't windows)
    - sgtnoodle 4y ago
      I'm not too concerned about the details of how the kernel gets loaded into memory, whether it's the firmware or efistub or grub. I just like grub because it's familiar and has a nice menu that makes it easy to tweak options.
vermaden 4y ago
This: https://i.imgur.com/W3edB3t.png https://i.imgur.com/W3edB3t.png Seems nothing changed for the part 5 years ... Moar details here: https://is.gd/BECTL https://is.gd/BECTL In other words - when it comes to #ZFS - just use #FreeBSD - you will thank me later.
soupbowl 4y ago
I prefer FreeBSD if I am going to use ZFS, never had a problem and I always use zroot mirrors. On linux I have had many issues, I won't use it on linux anymore.
_emacsomancer_ 4y ago
The Arch derivative CachyOS[1] has ZFS-enabled kernels by default. And their repos can be added to existing Arch installs[2], so I've found this a nice way of working around the non-ideal state of ZFS integration in Arch. (Plus their kernel choices have other interesting features.) [1]: https://cachyos.org/ https://cachyos.org/ [2]: https://github.com/CachyOS/linux-cachyos#cachyos-repositories https://github.com/CachyOS/linux-cachyos#cachyos-repositorie...
quags 4y ago
Not an arch user but a zfs user. I skip zfs on root , except on FreeBSD because that is the most solid. For Linux info a basic raid 1 for root and boot and create a zfs mirror on the other drives. Normally I have 6 drives (mirror plus the 2 os drives). I then take some space on the os drives to create a slog mirror for zfs. The heavy usage comes on the zfs mirror anyway the os is really just there to power vms.
ashleyis 4y ago
My solution for problems I've hit similar to this though not ZFS related (and by no means a catch-all) was to install linux-lts alongside linux and have boot entries for both

65 points by LinuxBender 4y ago

… it still continues to work on NixOS after rebooting into the previous generation. ;)

Ah man been there, done that! I now specifically use the dated archive mirrors for arch mirrorlist as well as the zfs-linux AUR package over the DKMS. Been a lot less of a headache dealing with it that way. I just bump the date up to as far forward as the two packages were aligned and off ya go. I do wish the licensing would get sorted out because I love ZFS but dealing with always having to juggle that dependency is annoying. Also mkinitcpio should be much more aggressive on failing on errors.

commandersaki 4y ago
I'm on the zfs-linux AUR package in the archzfs repo and not DKMS. I'm still waiting for zfs packages for the latest kernel but they haven't been cut yet; I'm wondering if this has something to do with OPs problems that are being encountered, and so there's a bit of a delay on the next zfs release.

pm2222 4y ago

The part, where wireless nic is not allowed to be enslaved to a bridge, I believe that it was allowed at one point in the past. The root cause is that the wireless frame has 3-4 MAC addresses versus Ethernet just 2. But somehow software should be able to do something about it. Why not?

fuzzy2 4y ago
This can work with MAC address translation (NAT, but on a lower level) and connection tracking. Not exactly a great solution. Alternatively, you can have one association with the access point per device, which is what some off-the-shelf repeaters do. Drivers often limit these to very low numbers. This wasn't possible in the past either. There's WDS (4-address mode) and some drivers that did what I describe above under the hood. That's it.
3np 4y ago
That part of the story was a plot hole for my. Why did the protagonist not just set up a forwarding proxy, or IP forwarding, on the laptop? Tor is convenient. https_proxy=socks://laptop:9050 curl ...

t344344 4y ago

I had similar issue with BTRFS thanks to new faulty DDR4. - CRC checksum for some blocks on fs was wrong - I discovered faulty DDR4, replaced it, without knowing FS corruption - system started randomly kernel panicking - no msg in logs (fs died with kernel) - eventually I experienced kernel panic while at computer, and seen msg on screen (broken checksum or something) - there was no way to repair filesystem. I had to recover as much as data, throw away some files and start over on fresh - I use BTRFS snapshot streaming in backups, wrong checksums propagated to backups. I had to restore from 1 week old backups - Since than I use RSYNC backups as well - And I test all PCs with memtest monthly overnight. It is also a good stress test for CPU.

nicolaslem 4y ago
> And I test all PCs with memtest monthly overnight. It is also a good stress test for CPU. I think that the consensus in the PC enthusiast community is that memtest is quite dated and not very good at finding instability in modern hardware.
- tcrenshaw 4y ago
  what's the community recommendation these days?
- Marsymars 4y ago
  I’m not particularly up-to-date on that subject, so someone else can probably provide more info, but FWIW PassMark’s MemTest86 has had ongoing development, while the open source MemTest86+ was effectively a dead project for nearly a decade before some recent activity - so there’s a fair amount of difference between the two.
- t344344 4y ago
  I am not sure what consensus is, but I use this: https://memtest.org/ https://memtest.org/

imrehg 4y ago

Yup, this popped up for me about a month ago on a Raspberruy Pi system, fortunately not on the root, but still. On ARM the kernel_neon_begin / kernel_neon_end symbols were made GPL only[0] which is quite puzzling on many levels. The discussion of the reported issue on the OpenZFS GitHub is very educational, both on the maintainers side (useful, sensible), and the other commenters syde (kinda "thermonuclear")[1] In the meantime I just downgraded the kernel as well, fortunately still had it in the pacman cache; then pinned it for now. Sent an email to to the patch creator / signer-off too, to feed back on that "this is not expected to be disruptive to existing users." part of the commit (as in [0]). [0]: https://lore.kernel.org/all/20221107170747.276910-1-broonie@kernel.org/ https://lore.kernel.org/all/20221107170747.276910-1-broonie@... [1]: https://github.com/openzfs/zfs/issues/14555 https://github.com/openzfs/zfs/issues/14555

InvaderFizz 4y ago

> ERROR: modpost: GPL-incompatible module zfs.ko uses GPL-only symbol 'bio_start_io_acct' > ERROR: modpost: GPL-incompatible module zfs.ko uses GPL-only symbol 'bio_end_io_acct_remapped' I don't understand why there isn't a flag to ignore these and accept a tainted kernel so that this type of issue doesn't break things. Then again, pacman should have aborted the process as well.

denotational 4y ago
> ignore these and accept a tainted kernel That isn’t possible as I understand it. Loading any non-GPL module will mark the Kernel as tainted, but that’s not the issue here. The problem is that the ZFS module uses Kernel symbols that are marked as GPL-only, meaning that the Kernel will refuse to load the ZFS module as it isn’t marked as a GPL module. There isn’t any way to bypass this short of recompiling the Kernel to remove this restriction. The legal status of this is uncertain, since a number of questions naturally arise: 1. Is it acceptable to mark a Kernel module as a GPL module (this is done via a macro in the source) but release it under a license that is not GPL-compatible? 2. Is it acceptable to recompile the Kernel to remove this restriction? As far as I know neither of these have ever been tested, and there are differing opinions.
- danarmak 4y ago
  If a kernel with the restriction removed is compiled locally and is never redistributed, why would it be illegal?
  - tomatocracy 4y ago
    It wouldn't. Moreover I can't see that there is any reason that redistribution of such a kernel would be against GPLv2, the licence under which the kernel is distributed.
    - denotational 4y ago
      > It wouldn't. Are you a lawyer? Asking in good faith, because if you are then this is the first qualified opinion I’ve seen on the matter. Everything else has been people claiming to have received legal opinions, see for example the email from Linus in my sibling comment.

dpwm 4y ago

I have the kernel and zfs modules in IgnorePkg in /etc/pacman.conf. This way I have to update them explicitly. I usually have to manually install a kernel from archive.archlinux.org/packages but I find this much less fragile than DKMS.

DvdGiessen 4y ago

A workaround that works locally (and only locally) is to build zfs-dkms (AUR) yourself and modify it so it doesn't break on these GPL-only symbols. While the licenses would forbid distributing it as such, you can do so on your own machine just fine. And if your root is on ZFS, make a snapshot before updating! I set up a pacman hook[0] which runs zfs_autobackup[1] which automatically manages snapshots, so I can always easily roll back to a non-broken state. The ZFSBootMenu[2] bootloader makes that extremely fast without even needing a bootable USB-drive. :) [0]: https://wiki.archlinux.org/title/Pacman#Hooks https://wiki.archlinux.org/title/Pacman#Hooks [1]: https://github.com/psy0rz/zfs_autobackup https://github.com/psy0rz/zfs_autobackup [2]: https://zfsbootmenu.org/ https://zfsbootmenu.org/

fuzzy2 4y ago

If you're running Linux with ZFS, make sure to have a ZFS-enabled live medium available. The Arch ISO is easy to modify. Various variants with ZFS exist around the net. I believe Ubuntu also ships with ZFS nowadays. I think NixOS too? Not entirely sure, it's been a while. If you go custom, you can also add drivers/firmware for whatever exotic hardware you have!

tripdout 4y ago
Yes, NixOS also ships ZFS in their ISO. It's been super easy coming from ZFS on Arch to NixOS, no hassles with adding custom repos, using DKMS, etc.
e-v 4y ago
The script available there: https://github.com/eoli3n/archiso-zfs https://github.com/eoli3n/archiso-zfs makes it extremely easy to add ZFS support to any Arch ISO after it has booted. You can copy any standard ISO to a USB drive, boot off it, then run `curl -s https://raw.githubusercontent.com/eoli3n/archiso-zfs/master/init https://raw.githubusercontent.com/eoli3n/archiso-zfs/master/... | bash` and you'll have ZFS support in a few seconds, without having anything to worry about.
- rcthompson 4y ago
  > without having anything to worry about. Nothing to worry about except network access, of course, which was the issue in this case. Much better to have it not with ZFS support already enabled.
  - e-v 4y ago
    Indeed (I guess you meant "much better to have it with ZFS support already enabled"). I simply mentioned this for people unaware of the existence of this script who don't have network issues with the Arch ISOs.

ratorx 4y ago

This is one of the things NixOS is good for. There can be a special version of Linux that is only updated once some regression test on ZFS passes: `config.boot.zfs.package.latestCompatibleLinuxPackages` That way the main maintainer can still follow HEAD, whilst the ZFS maintainer can make sure that Linux is updated for ZFS once it’s been tested (or even automatically, but I don’t know if it’s that advanced yet).

SuperSandro2000 4y ago
And even if you don't use that, the build just fails early without consequences.
aftbit 4y ago
I see a lot of smart people recommending Nix, but whenever I look at it, it seems like there are a ton of sneaky config options that one must get right to get a working system...
- ratorx 4y ago
  It depends (and not any more than applications that need sneaky config). Simple systems can have quite simple options. At the very least you can take a working copy and figure out exactly which configuration is necessary. Hard to do that without NixOS. Especially with nixos-option. And the options are all documented in a central place. The module definitions using the options are also “easy” to search through (Language is a bit complicated, but not in a way that grep doesn’t work).
- assbuttbuttass 4y ago
  nixos-generate-config usually produces a working config out of the box. And if you need to change some obscure config options, everything is well documented with "man configuration.nix"
  - ElectricalUnion 4y ago
    I like https://search.nixos.org/options https://search.nixos.org/options , but while nicer it enters the "need somewhat working system to fix the system" bootstraping issues.

throw0101b 4y ago

Does Arch Linux have boot environments (or equivalent)? Boot Environments allows the system to be upgraded, while preserving the old system environment in a separate ZFS dataset. * https://man.freebsd.org/cgi/man.cgi?beadm https://man.freebsd.org/cgi/man.cgi?beadm The beadm command is the user interface for managing ZFS Boot Environments (BEs). This utility is intended to be used by system administrators who want to manage multiple Oracle Solaris instances on a single system. * https://docs.oracle.com/cd/E86824_01/html/E54764/beadm-1m.html https://docs.oracle.com/cd/E86824_01/html/E54764/beadm-1m.ht... > A ZFS boot environment is a bootable clone of the datasets needed to boot the operating system. Creating a BE before performing an upgrade provides a low-cost safeguard: if there is a problem with the update, the system can be rebooted back to the point in time before the upgrade. * https://klarasystems.com/articles/managing-boot-environments/ https://klarasystems.com/articles/managing-boot-environments... Or perhaps: > In essence, ZFSBootMenu is a small, self-contained Linux system that knows how to find other Linux kernels and initramfs images within ZFS filesystems. When a suitable kernel and initramfs are identified (either through an automatic process or direct user selection), ZFSBootMenu launches that kernel using the kexec command. * https://github.com/zbm-dev/zfsbootmenu https://github.com/zbm-dev/zfsbootmenu

boomboomsubban 4y ago
Yes, but like every part of ZFS on Arch, not by default. https://aur.archlinux.org/packages/zfsbootmenu https://aur.archlinux.org/packages/zfsbootmenu
drewg123 4y ago
Indeed. In fact, my main reason for ZFS on root is boot environments. If something breaks, its trivial to roll back the rootfs as it was in the old BE, before the upgrade. And I run ZFS on FreeBSD.
nubinetwork 4y ago
I believe only ubuntu half-supported boot environments. I don't know of any other distro that even bothered to support zfs outside of supplying the package for users to install.

frankjr 4y ago

The main problem here is Pacman's handling of "PostTransaction" hooks (which in this case builds the zfs kernel module). The whole installation should have failed but it didn't. Individual hooks can specify "AbortOnFail" but it's available to "PreTransaction" hooks only.

zokier 4y ago

Like many fun adventures in Linux, this begins with adding some 3rd party repos (and/or packages) into the system. In this case doubly so because: > I'd probably have caught the red error if it wasn't followed by tons of log output from building AUR packges :) Although I do agree that noticing important messages from even plain pacman output can occasionally be difficult, although I haven't seen many other package managers do it any better. > But I can't, because I can only build a live ISO with Linux 6.2.8 easily, which is the original problem That feels bit weird situation though. I'd imagine being able to build downgraded images would be useful in general, so interesting that apparently its not so easy.

blywi 4y ago

I just ran into the same issue yesterday when I was updating one of my machines. Furtunately I caught the DKMS error when mkinitcpio ran after the upgrade: ==> dkms install --no-depmod zfs/2.1.9 -k 6.2.8-arch1-1 Error! Bad return status for module build on kernel: 6.2.8-arch1-1 (x86_64) Consult /var/lib/dkms/zfs/2.1.9/build/make.log for more information. ==> WARNING: `dkms install --no-depmod zfs/2.1.9 -k 6.2.8-arch1-1' exited 10 Issues like these are when being on a rolling-release distro might bite you. To mitigate the risk a bit, I always have the linux-lts kernel package installed on all of my Arch Linux machines, which provides a fallback option when the mainline kernel fails, or some major kernel issue surfaces. In my case with the LTS kernel at 6.1.21-1-lts, DKMS build of ZFS on LTS succeeded, so it would have been a fallback option to boot even if I would have missed the error for the mainline kernel DKMS. ==> dkms install --no-depmod zfs/2.1.9 -k 6.1.21-1-lts ==> depmod 6.1.21-1-lts I don't have my root on ZFS, I only use it for data storage, but if I ever did,I would make damn sure to build myself a live ISO with ZFS support and all drivers necessary for network access, ahead of time, in case I need to do an emergency repair. It doesn't need to always be on the latest kernel, just recent enough that you are able to mount all partitions and to chroot into the system. For this issue a workaround with Kernel 6.2.8 is discussed in the AUR comments for zfs-dkms: https://aur.archlinux.org/packages/zfs-dkms https://aur.archlinux.org/packages/zfs-dkms This would involve editing the PKGBUILD for ZFS to patch the license for the symbols to GPL, though I'm not sure about the legality of this approach.

tzs 4y ago
> For this issue a workaround with Kernel 6.2.8 is discussed in the AUR comments for zfs-dkms: https://aur.archlinux.org/packages/zfs-dkms https://aur.archlinux.org/packages/zfs-dkms This would involve editing the PKGBUILD for ZFS to patch the license for the symbols to GPL, though I'm not sure about the legality of this approach In the US there is 17 USC 117(a) which might apply: > Making of Additional Copy or Adaptation by Owner of Copy.—Notwithstanding the provisions of section 106, it is not an infringement for the owner of a copy of a computer program to make or authorize the making of another copy or adaptation of that computer program provided: > (1) that such a new copy or adaptation is created as an essential step in the utilization of the computer program in conjunction with a machine and that it is used in no other manner, or (Section 106 is the section that says you need permission from the copyright owner to make copies or adaptations).

SuperSandro2000 4y ago

This can't happen with NixOS.

Volundr 4y ago

Can someone explain the purpose of GPL only symbols to me, and why existing long standing APIs keep getting changed to GPL only? From the outside it certainly feels like all it does is (intentionally?) break things.

drewg123 4y ago
I think that is the point.. the kernel devs want to break 3rd party modules, because they don't like stuff distributed outside the kernel.
- cesarb 4y ago
  > because they don't like stuff distributed outside the kernel. This mechanism doesn't prevent stuff from being distributed outside of the kernel at all, a GPL-licensed module could stay out of tree for a long time. They even have a documented mode in the kernel Makefile to compile out-of-tree modules, as described at "make help": "make -C path/to/kernel/src M=$PWD modules modules_install"
georgyo 4y ago
That is indeed the goal, they explicitly want to break compatibility. Their responses on the mailing list even say that. GPL or GTFO. It's a definitely annoying as a user. OpenZFS is much nicer than BTRFS, but you have to accept that breakage of the kernel devs may cause you issues.
- Volundr 4y ago
  That's the inference I was trying to avoid making. It's frustrating because a big reason I use Linux is it gives me the freedom to build the system I want exactly how I want it, but a couple times a year I get this reminder that the devs are actually against me in this.
  - blibble 4y ago
    personally I will abandon Linux before I abandon ZFS

mst 4y ago

These sorts of events are why I know quite a number of people who do ZFS-for-everything on their FreeBSD systems but ZFS-for-everything-except-root on their Linux boxen. Seems like in a lot of cases having all of your data on ZFS gets you almost all of the awesomeness of having ZFS around with fairly minimal downsides. (deciding if this is true for any particular system is left as an exercise to the person designing the deployment)

mustache_kimono 4y ago
This is somewhat understandable. But to be clear, this is also an extraordinary set of circumstances -- a very recent, but marked stable kernel, on rolling release distro, with a back ported set of changes, breaking a dkms ZFS build. The user was notified and was able to take remedial action. My only issue would have been -- I'm not sure Arch made this very easy. FWIW I use Ubuntu LTS and ZFS on root works great. IMHO every distro should just ship ZFS. Ubuntu has proven this is just not a practical difficulty anymore. Let's quit the charade of making users compile their own kernel modules for each kernel release.
- mhio 4y ago
  It's not a charade, it's the looming shadow of Oracle lawyers.
  - mustache_kimono 4y ago
    Who I'd note haven't done anything so far? This is what is so mysterious to me -- if this is the blockbuster case some on Reddit have made it out to be, why hasn't there been a suit against Canonical?
    - mhio 4y ago
      Does the "blockbuster case" line of thought you've seen suggest it should be easy for Oracle? From the differing legal opinions I've read it seems a legal quagmire. That's a quagmire that _both_ sides of the argument have to wade in to, just Oracle gets to choose if and when that happens. They might just be a bit gun shy after their last 10 year long open source case.

Asdrubalini 4y ago

This is one of the reason I use NixOS. I think it has the best ZFS support possible (at least on Linux)

Caligatio 4y ago

I dual booted my machine Win10 and Arch on ZFS Root. Admittedly I used Win10 like 99% so there was always a laundry list of updates when I booted into Arch. It seemed ZFS broke 20% of the time I updated. It was either the silly GRUB config fix that wasn't upstreamed getting overwritten or the ZFS module build somehow failed. I got so frustrated that I just nuked the whole install. I realize that ZFS is a weird bolt-on but the fact that kernel modules failing to build doesn't cause significant error messages and/or failure is silly. Arch is great but Arch on ZFS felt like my Gentoo days: I want to have fun using my computer, not constantly fiddling to make my computer usable.

nubinetwork 4y ago
It's not worth your time to use grub if you run zfs... the zfs module in grub only supports the ancient feature flag set from Solaris 9/10, its a miracle that it even works on Linux at all... It also doesn't help that grub is spearheaded by Oracle employees, and as such, they couldn't give a rats ass about maintaining the module. (And if they do, it'll be to add more Oracle Solaris features rather than zfs on linux/bsd/illumos)

mustache_kimono 4y ago

Oh would it be nice if ZFS wasn't CDDL or if the CDDL/GPL compatibility would be settled. For all practical purposes, it doesn't appear to be a problem. Ubuntu has shipped ZFS for years. As much bellyaching as we hear from some quarters about the hypothetical harms this has caused, no one has brought a suit, because my guess is -- it would be a pretty terrible case for a copyright holder from the Linux side. See [0] in addition to all the other ink that has been spilled re: this matter. Linus may not feel comfortable with ZFS included in mainline. I'm not sure distro maintainers need to feel the same way. [0]: https://www.networkworld.com/article/2301697/encouraging-closed-source-modules-part-1--copyright-and-software.html https://www.networkworld.com/article/2301697/encouraging-clo...

nubinetwork 4y ago
To be fair, Larry has the deepest pockets, and it takes some balls to stand up to the likelihood of a lawsuit.
- mustache_kimono 4y ago
  > it takes some balls to stand up to the likelihood of a lawsuit. Likelihood? It's been years Canonical is doing what I've told be the Reddit-rati is a blatant copyright violation and ... crickets.
  - nubinetwork 4y ago
    Larry knows if he were to sue Shuttleworth/Canonical it could potentially bankrupt them both.
    - mustache_kimono 4y ago
      My guess is Larry could just buy Canonical and turn it into a parking lot. But maybe this is the long game -- Larry is hoping to trap IBM/Red Hat in a suit when they add ZFS to the kernel, but I'm not sure that's how copyright litigation works. If you forgo litigation for years and now decide to sue, I'm not sure you get to assert copyright now.

AviationAtom 4y ago

Root on ZFS just feels too much like playing with fire to me. Fixing an unbootable ext4 filesystem seems tricky enough already. Am I just in the dark on how to easily repair boot issues, especially on more modern filesystems? I feel like the OP probably had to put countless hours into working through all these issues to finally get back to a fully functioning environment.

Gordonjcp 4y ago
Why would you bother? It'll be quicker to just blow it away and reinstall.
- AviationAtom 4y ago
  You're not concerned about all your system settings, installed applications, and any other data worthy of preservation on your root disk? Or am I misunderstanding how Root on ZFS works/boots?
  - Gordonjcp 4y ago
    No, why would I be? In what world would you even lose all that?
    - AviationAtom 4y ago
      What specifically would a reinstall cover then? Is it just a boot partition/stub (FAT?) that loads the root ZFS pool? It all seems like a drastically different architecture than I'm used to seeing on a system.
      - Gordonjcp 4y ago
        Why would you keep /home on the same partition? Why would you be manually adding in apps after a reinstall? You just script this shit.
AviationAtom 4y ago

nubinetwork 4y ago

Honestly, arch seems like a fragile distro. My experience with arch is: install the distro files, set up efistub booting, reboot... install updates a couple days later and wind up with a system that can't reboot, and if you force reboot it, it won't ever mount the root fs again. I've never had that happen on Gentoo, or debian/ubuntu or fedora, or suse. Only arch could be so temperamental to break itself on kernel upgrades.

sgtnoodle 4y ago
I've been running it for 8 years or so, and it's been fine. I have yet to need to do a reinstall on any of my computers. The rolling release has been great from a maintenance standpoint. There is the occasional problem, of course. Most recently, pipewire snuck its way in as a dependency. Not having been configured sanely, it caused some minor irritating bugs as it clashed with pulseaudio. My newest desktop's bios do tend to mess up my boot configuration after a firmware update. Windows boots directly instead of grub. It's a 10 second fix from the command line, whether in Windows or in Linux.
- nubinetwork 4y ago
  I'm sure part of it for me has to be the efistub. I don't understand why anyone would want to bother with a bootloader in modern times because efi will let you pick and choose what to boot just fine without one. Clobbered bootloader is also why your system goes back to windows after an update (windows has always had issues cohabiting with other operating systems that weren't windows)
  - sgtnoodle 4y ago
    I'm not too concerned about the details of how the kernel gets loaded into memory, whether it's the firmware or efistub or grub. I just like grub because it's familiar and has a nice menu that makes it easy to tweak options.

vermaden 4y ago

This: https://i.imgur.com/W3edB3t.png https://i.imgur.com/W3edB3t.png Seems nothing changed for the part 5 years ... Moar details here: https://is.gd/BECTL https://is.gd/BECTL In other words - when it comes to #ZFS - just use #FreeBSD - you will thank me later.

soupbowl 4y ago

I prefer FreeBSD if I am going to use ZFS, never had a problem and I always use zroot mirrors. On linux I have had many issues, I won't use it on linux anymore.

_emacsomancer_ 4y ago

The Arch derivative CachyOS[1] has ZFS-enabled kernels by default. And their repos can be added to existing Arch installs[2], so I've found this a nice way of working around the non-ideal state of ZFS integration in Arch. (Plus their kernel choices have other interesting features.) [1]: https://cachyos.org/ https://cachyos.org/ [2]: https://github.com/CachyOS/linux-cachyos#cachyos-repositories https://github.com/CachyOS/linux-cachyos#cachyos-repositorie...

quags 4y ago

Not an arch user but a zfs user. I skip zfs on root , except on FreeBSD because that is the most solid. For Linux info a basic raid 1 for root and boot and create a zfs mirror on the other drives. Normally I have 6 drives (mirror plus the 2 os drives). I then take some space on the os drives to create a slog mirror for zfs. The heavy usage comes on the zfs mirror anyway the os is really just there to power vms.

ashleyis 4y ago

My solution for problems I've hit similar to this though not ZFS related (and by no means a catch-all) was to install linux-lts alongside linux and have boot entries for both