10 ms·

I personally think they're moving in the wrong direction. I'd rather have "NMIRFS" (no more initramfs). Eg, a smarter bootloader that understands all bootable

I personally think they're moving in the wrong direction. I'd rather have "NMIRFS" (no more initramfs). Eg, a smarter bootloader that understands all bootable filesystems and cooperates with the kernel to pre-load modules needed for boot and obviates the need for initramfs. FreeBSD's loader does this, and its so much easier to deal with. Eg, it understands ZFS, and can pre-load storage driver modules and zfs.ko for the kernel, so that the kernel has everything it needs to boot up. It also understands module dependencies, and will preload all modules that are needed for a module you specify (similar to modprobe).

sim7c00 2y ago
this exactly. freebsd's loader is one of the only sane ones ive seen. grub is an amazing piece of software but its really a mess to work with.
ta8645 2y ago
The Linux kernel does not require an initramfs. You can build a kernel with everything compiled in; with no modules needed at all. Initramfs is used for generic kernels where you don't know beforehand which features will be required. This allows you to avoid wasting RAM on features you don't use. But it is optional.
- Muromec 2y ago
  I think the idea is, since modules map to device ids statically, bootloader could have enough information to read them from the filestem one by one. I don’t see the point of doing so however.
- jolmg 2y ago
  > Initramfs is used for generic kernels where you don't know beforehand which features will be required. And also for e.g. cases where you've got some custom stack of block devices that you need to set up before the root FS and other devices can be mounted. It's not just about loading kernel modules.
- drewg123 2y ago
  I realize that. But every distro I've used uses an initramfs, so unless you want to build your own kernels, you're stuck with it, and the painfully slow initramfs updates when you update packages, and dkms (or similar) updates the initramfs with the newer version of your out-of-tree modules.
  - kbolino 2y ago
    Given the reason why "out-of-tree modules" exist, there's really no way to eliminate initramfs or something like it entirely in the general case. It might be possible to speed up the process of building the image (as long as the results are not "redistributed"), but this is a licensing and legal problem, not a technical one. FreeBSD is under a much more permissive non-copyleft license and so can legally bundle things that Linux cannot.
    - medstrom 2y ago
      You're talking about something like ZFS, and I get that they can't just compile it in, but a distro can still ship the module, if I'm not mistaken. ...But to load it at boot time it absolutely must be done through an initramfs. Is that right?
      - aaronmdjones 2y ago
        > But to load it at boot time it absolutely must be done through an initramfs. Is that right? Yes, because it cannot be part of the kernel image, or it would be illegal (a violation of the GPL license) to distribute that kernel. Therefore, it must be a module, and that module has to live somewhere and be loaded by something. If root is on ZFS, this must therefore live in an initramfs and be loaded by it so that the initramfs can mount the real root filesystem on the kernel's behalf.
        megous 2y ago
        It doesn't have to be a module if you're building the kernel for yourself. No violation in that.
- linsomniac 2y ago
  Is anyone really wanting to get back into the business of building their own kernels? I started using Linux heavily in '92, and I've built a lot of kernels, and am quite happy to not be building them anymore.
  - ssl-3 2y ago
    I kind of liked compiling my own kernels. I felt I was better-connected to the state of things, and it was fun to see it all evolve from the vantage point of "make menuconfig". But initramfs isn't so bad, and it allows things like ZFS root to have a modicum of smoothness and integration.
  - megous 2y ago
    It's easy (2-3 commands), takes like a minute on a modern machine with trimmed down kernel configuration, and you can customize the kernel to your liking (write/patch drivers, embed firmware blobs, fix things that are broken or missing). What's to hate? :) Though I only do it for my ARM based devices currently. And if you're not throwing away build artifacts after each build, then getting stable updates is just a `git pull` and incremental make, which is usually very quick.
  - teo_zero 2y ago
    I build my own kernel. I did invest some time to select the right configuration, but now it's just a question of copying over the old .config and running "make". What's annoying about that?
  - adamomada 2y ago
    https://www.funtoo.org/Support_Matrix#Kernels https://www.funtoo.org/Support_Matrix#Kernels
  - prmoustache 2y ago
    kernel compilation is easily automated. I don't want to do that and like the initramfs approach mostly because I like the fact I can take a hard drive out of a computer and boot the system on another one in case of a hardware failure. That is a lot faster than recovering from a backup.
- arp242 2y ago
  What's the reason it doesn't load those modules from the regular filesystem? That's what FreeBSD does, and seems to work well enough?
  - ta8645 2y ago
    Because there are a lot of different types of filesystems supported. And you'd have to compile them all into the kernel. Which of course you can do, that is supported by the build system today. But Distros typically prefer to keep their kernels small, and not waste the RAM that would be taken up by compiling it all into the kernel.
    - mixmastamyk 2y ago
      It must already have vfat and the ESP, so why not just copy a basic set of modules to a subfolder there?
Aurornis 2y ago
> I'd rather have "NMIRFS" (no more initramfs). In many cases, you don't need initramfs. I rarely use one in embedded systems.
- fullstop 2y ago
  I use them in embedded systems because they allow me to mount encrypted volumes without exposing the keys.
  - account42 2y ago
    How does that work? The keys have to be loaded from somewhere.
    - ptsneves 2y ago
      I guess from a hardware cryptography module or OPTEE[1] [1] https://optee.readthedocs.io/en/latest/general/about.html https://optee.readthedocs.io/en/latest/general/about.html
    - fullstop 2y ago
      The keys to decrypt the kernel are in u-boot. u-boot's keys are in the low level boot loader, and the keys for that are sometimes burned in write-only fuses on the microcontroller itself. Other chips have OP-TEE or similar frameworks, and you just chain the keys all the way down to the initramfs and that data is wiped when you start init. You're reliant on the capabilities of the chip that you're working with, and a flaw in that can unravel everything that you've done. In one case, I had to disable the on-chip boot agent once things were provisioned because of flaws in their implementation. In short, a signed applet could be sent to the chip to do things like read/write NAND or NOR, set fuse bits, etc. When an unsigned applet was sent, it was rejected as expected but they neglected to clear the memory contents in this case. So you could send a malicious applet, let it be rejected, and then just tell it to execute. It's kind of a fascinating writeup if you want to know more [1]. 1. https://labs.withsecure.com/advisories/microchip-atsama5-soc-multiple-vulnerabilities-full-release https://labs.withsecure.com/advisories/microchip-atsama5-soc...
Arnavion 2y ago
Linux has multiple choices for filesystems for root, even if you only count the most popular ones. And on top of that they could be encrypted by LUKS. Duplicating all that into the bootloader is what GRUB does, and poorly. Putting the kernel into the ESP is much better in that regard.
m463 2y ago
> I personally think they're moving in the wrong direction the other direction is to put everything in systemd. :)
- markhahn 2y ago
  bad thing to joke about.
aaronmdjones 2y ago
As other sibling comments have explained, an initramfs is usually optional for booting Linux. If you build the drivers for your storage media and filesystem into the kernel (not as a module), and the filesystem is visible to the kernel without any userland setup required beforehand (e.g. the filesystem is not in an LVM volume, not on an MD-RAID array, not encrypted), it is fully capable of mounting the real root filesystem and booting init directly from it. The only point of consideration is that it doesn't understand filesystem UUIDs or labels (this is part of libuuid which is used by userland tools like mount and blkid), so you have to specify partition UUIDs or labels instead (if you want to use UUIDs or labels). For GPT disks, this is natively available (e.g. root=PARTUUID=4F68BCE3-E8CD-4DB1-96E7-FBCAF984B709 or root=PARTLABEL=Root). For MS-DOS disks, this is emulated for UUIDs only by using the disk ID and partition number (e.g. root=PARTUUID=11223344-02). You can also specify the device name directly (e.g. root=/dev/sda2) or the major:minor directly (e.g. root=08:02), but this is prone to enumeration order upset. If you can guarantee that this is the only disk it will ever see, or that it will always see that disk first, this is often the most simple approach, but these days I use GPT partition UUIDs.
- StillBored 2y ago
  Not entirely, it is possible for simple boot solutions, but to enable LUKS and various other rootfs, it usually requires some userspace probing, authentication, etc. I'm not so sure it would be easy to convince the kernel maintainers to add a user prompt for pin code to the kernel just to avoid having an initrd.
  - aaronmdjones 2y ago
    > it is possible for simple boot solutions, but to enable LUKS and various other rootfs, it usually requires some userspace probing, authentication, etc I did mention that the root filesystem had to be both not encrypted and not require any userland setup. > I'm not so sure it would be easy to convince the kernel maintainers to add a user prompt for pin code to the kernel That wouldn't help, as the password is not usually used directly as a volume encryption key. In LUKS2 for example, a memory-hard KDF (Argon2) can be used to transform the password or PIN into a key-slot encryption key which is then used to decrypt that slot to obtain the volume encryption key (which is usually completely random and not at all related to or derived from any kind of password) in order to set up the dm-crypt mapping. > just to avoid having an initrd. No distribution has used an initrd in the last several years; initramfs reigns supreme now. They are not the same thing; an initrd is an image of a real filesystem (e.g. squashfs or ext2) that is read-only mounted onto / by the kernel during its startup, while an initramfs is (an optionally-compressed) cpio archive that is extracted to a read-write tmpfs (or ramfs if tmpfs is not available) mounted on / by the kernel during its startup and then the memory occupied by that cpio archive is freed because it is no longer necessary. An initramfs can also be built into the kernel image; an initrd cannot, and must be provided by a bootloader.
zdw 2y ago
FWIW, Grub has a read-only ZFS implementation to allow booting: https://git.savannah.gnu.org/cgit/grub.git/tree/include/grub/zfs https://git.savannah.gnu.org/cgit/grub.git/tree/include/grub...
- nubinetwork 2y ago
  Grub uses an ancient version of zfs code, it's tied to Oracle's zfs and they refuse to update it to current openzfs.
  - UnlockedSecrets 2y ago
    Are there any instances of features being utilized that the old version of the code is unable to cope with well enough to be able to boot the system?
    - upon_drumhead 2y ago
      Native ZFS encryption is the major one I'm aware of
    - nubinetwork 2y ago
      https://github.com/openzfs/zfs/tree/master/cmd/zpool/compatibility.d https://github.com/openzfs/zfs/tree/master/cmd/zpool/compati...
  - yjftsjthsd-h 2y ago
    Refuse, or legally can't? Oracle doesn't own the copyrights on commits made after illumos forked from the corpse of opensolaris.
    - nubinetwork 2y ago
      A little of both. Everyone know about Linus' refusal to touch CDDL code, but grub isn't the kernel. There have been several attempts to add features to the grub zfs code over the years, but there are several maintainers of grub who happen to be employees of Oracle, and typically the attempts go nowhere. I personally can't recommend using grub anymore. The whole "just make 2 pools" solution is unacceptable, and until Oracle stops gatekeeping, their code becomes more obsolete in my eyes.
mjg59 2y ago
This requires a bunch of additional logic in the bootloader (eg, providing disk encryption keys), and since you're not doing this in a fully-featured OS environment (as you are in the initramfs case) it's going to be hard providing a consistent experience through the entire boot process. Having the pre-boot environment be a (slightly) cut-down version of the actual OS means all your tooling can be shared between the two environments.
zauguin 2y ago
This is a step in that direction. What they are proposing is not so much "no bootloader" but using a small Linux as bootloader. I'm using a similar setup for some time and it gives some of these advantages. Especially you get support for all relevant filesystems (you can support everything Linux supports because it is Linux), it can dynamically build a minimal initramfs with only the needed drivers if you want to and understands module dependencies (e.g. it can just dump the list of modules it uses itself) and is generally much more flexible.
rcxdude 2y ago
Does FreeBSD's loader share code with the kernel? It does seem like a lot of duplication of systems to make it work in comparison to just using the same code.
fooker 2y ago
The more smartness you put here, the more it makes life difficult for non-standard operating systems. And if this bit is closed source, and something doesn't work, you don't have a recourse.
dexen 2y ago
Seconding this. Having lucked into using Lilo and no initramfs for several years now, I'm very happy with robustness and straightforwardness of the solution. In contrast, on the rare occasion I've dealt with somebody elses' GRUB and initramfs setups, they turn out brittle and complex.

sim7c00 2y ago

this exactly. freebsd's loader is one of the only sane ones ive seen. grub is an amazing piece of software but its really a mess to work with.

ta8645 2y ago

The Linux kernel does not require an initramfs. You can build a kernel with everything compiled in; with no modules needed at all. Initramfs is used for generic kernels where you don't know beforehand which features will be required. This allows you to avoid wasting RAM on features you don't use. But it is optional.

Muromec 2y ago
I think the idea is, since modules map to device ids statically, bootloader could have enough information to read them from the filestem one by one. I don’t see the point of doing so however.
jolmg 2y ago
> Initramfs is used for generic kernels where you don't know beforehand which features will be required. And also for e.g. cases where you've got some custom stack of block devices that you need to set up before the root FS and other devices can be mounted. It's not just about loading kernel modules.
drewg123 2y ago
I realize that. But every distro I've used uses an initramfs, so unless you want to build your own kernels, you're stuck with it, and the painfully slow initramfs updates when you update packages, and dkms (or similar) updates the initramfs with the newer version of your out-of-tree modules.
- kbolino 2y ago
  Given the reason why "out-of-tree modules" exist, there's really no way to eliminate initramfs or something like it entirely in the general case. It might be possible to speed up the process of building the image (as long as the results are not "redistributed"), but this is a licensing and legal problem, not a technical one. FreeBSD is under a much more permissive non-copyleft license and so can legally bundle things that Linux cannot.
  - medstrom 2y ago
    You're talking about something like ZFS, and I get that they can't just compile it in, but a distro can still ship the module, if I'm not mistaken. ...But to load it at boot time it absolutely must be done through an initramfs. Is that right?
    - aaronmdjones 2y ago
      > But to load it at boot time it absolutely must be done through an initramfs. Is that right? Yes, because it cannot be part of the kernel image, or it would be illegal (a violation of the GPL license) to distribute that kernel. Therefore, it must be a module, and that module has to live somewhere and be loaded by something. If root is on ZFS, this must therefore live in an initramfs and be loaded by it so that the initramfs can mount the real root filesystem on the kernel's behalf.
      - megous 2y ago
        It doesn't have to be a module if you're building the kernel for yourself. No violation in that.
linsomniac 2y ago
Is anyone really wanting to get back into the business of building their own kernels? I started using Linux heavily in '92, and I've built a lot of kernels, and am quite happy to not be building them anymore.
- ssl-3 2y ago
  I kind of liked compiling my own kernels. I felt I was better-connected to the state of things, and it was fun to see it all evolve from the vantage point of "make menuconfig". But initramfs isn't so bad, and it allows things like ZFS root to have a modicum of smoothness and integration.
- megous 2y ago
  It's easy (2-3 commands), takes like a minute on a modern machine with trimmed down kernel configuration, and you can customize the kernel to your liking (write/patch drivers, embed firmware blobs, fix things that are broken or missing). What's to hate? :) Though I only do it for my ARM based devices currently. And if you're not throwing away build artifacts after each build, then getting stable updates is just a `git pull` and incremental make, which is usually very quick.
- teo_zero 2y ago
  I build my own kernel. I did invest some time to select the right configuration, but now it's just a question of copying over the old .config and running "make". What's annoying about that?
- adamomada 2y ago
  https://www.funtoo.org/Support_Matrix#Kernels https://www.funtoo.org/Support_Matrix#Kernels
- prmoustache 2y ago
  kernel compilation is easily automated. I don't want to do that and like the initramfs approach mostly because I like the fact I can take a hard drive out of a computer and boot the system on another one in case of a hardware failure. That is a lot faster than recovering from a backup.
arp242 2y ago
What's the reason it doesn't load those modules from the regular filesystem? That's what FreeBSD does, and seems to work well enough?
- ta8645 2y ago
  Because there are a lot of different types of filesystems supported. And you'd have to compile them all into the kernel. Which of course you can do, that is supported by the build system today. But Distros typically prefer to keep their kernels small, and not waste the RAM that would be taken up by compiling it all into the kernel.
  - mixmastamyk 2y ago
    It must already have vfat and the ESP, so why not just copy a basic set of modules to a subfolder there?

Aurornis 2y ago

> I'd rather have "NMIRFS" (no more initramfs). In many cases, you don't need initramfs. I rarely use one in embedded systems.

fullstop 2y ago
I use them in embedded systems because they allow me to mount encrypted volumes without exposing the keys.
- account42 2y ago
  How does that work? The keys have to be loaded from somewhere.
  - ptsneves 2y ago
    I guess from a hardware cryptography module or OPTEE[1] [1] https://optee.readthedocs.io/en/latest/general/about.html https://optee.readthedocs.io/en/latest/general/about.html
  - fullstop 2y ago
    The keys to decrypt the kernel are in u-boot. u-boot's keys are in the low level boot loader, and the keys for that are sometimes burned in write-only fuses on the microcontroller itself. Other chips have OP-TEE or similar frameworks, and you just chain the keys all the way down to the initramfs and that data is wiped when you start init. You're reliant on the capabilities of the chip that you're working with, and a flaw in that can unravel everything that you've done. In one case, I had to disable the on-chip boot agent once things were provisioned because of flaws in their implementation. In short, a signed applet could be sent to the chip to do things like read/write NAND or NOR, set fuse bits, etc. When an unsigned applet was sent, it was rejected as expected but they neglected to clear the memory contents in this case. So you could send a malicious applet, let it be rejected, and then just tell it to execute. It's kind of a fascinating writeup if you want to know more [1]. 1. https://labs.withsecure.com/advisories/microchip-atsama5-soc-multiple-vulnerabilities-full-release https://labs.withsecure.com/advisories/microchip-atsama5-soc...

Arnavion 2y ago

Linux has multiple choices for filesystems for root, even if you only count the most popular ones. And on top of that they could be encrypted by LUKS. Duplicating all that into the bootloader is what GRUB does, and poorly. Putting the kernel into the ESP is much better in that regard.

m463 2y ago

> I personally think they're moving in the wrong direction the other direction is to put everything in systemd. :)

markhahn 2y ago
bad thing to joke about.

aaronmdjones 2y ago

As other sibling comments have explained, an initramfs is usually optional for booting Linux. If you build the drivers for your storage media and filesystem into the kernel (not as a module), and the filesystem is visible to the kernel without any userland setup required beforehand (e.g. the filesystem is not in an LVM volume, not on an MD-RAID array, not encrypted), it is fully capable of mounting the real root filesystem and booting init directly from it. The only point of consideration is that it doesn't understand filesystem UUIDs or labels (this is part of libuuid which is used by userland tools like mount and blkid), so you have to specify partition UUIDs or labels instead (if you want to use UUIDs or labels). For GPT disks, this is natively available (e.g. root=PARTUUID=4F68BCE3-E8CD-4DB1-96E7-FBCAF984B709 or root=PARTLABEL=Root). For MS-DOS disks, this is emulated for UUIDs only by using the disk ID and partition number (e.g. root=PARTUUID=11223344-02). You can also specify the device name directly (e.g. root=/dev/sda2) or the major:minor directly (e.g. root=08:02), but this is prone to enumeration order upset. If you can guarantee that this is the only disk it will ever see, or that it will always see that disk first, this is often the most simple approach, but these days I use GPT partition UUIDs.

StillBored 2y ago
Not entirely, it is possible for simple boot solutions, but to enable LUKS and various other rootfs, it usually requires some userspace probing, authentication, etc. I'm not so sure it would be easy to convince the kernel maintainers to add a user prompt for pin code to the kernel just to avoid having an initrd.
- aaronmdjones 2y ago
  > it is possible for simple boot solutions, but to enable LUKS and various other rootfs, it usually requires some userspace probing, authentication, etc I did mention that the root filesystem had to be both not encrypted and not require any userland setup. > I'm not so sure it would be easy to convince the kernel maintainers to add a user prompt for pin code to the kernel That wouldn't help, as the password is not usually used directly as a volume encryption key. In LUKS2 for example, a memory-hard KDF (Argon2) can be used to transform the password or PIN into a key-slot encryption key which is then used to decrypt that slot to obtain the volume encryption key (which is usually completely random and not at all related to or derived from any kind of password) in order to set up the dm-crypt mapping. > just to avoid having an initrd. No distribution has used an initrd in the last several years; initramfs reigns supreme now. They are not the same thing; an initrd is an image of a real filesystem (e.g. squashfs or ext2) that is read-only mounted onto / by the kernel during its startup, while an initramfs is (an optionally-compressed) cpio archive that is extracted to a read-write tmpfs (or ramfs if tmpfs is not available) mounted on / by the kernel during its startup and then the memory occupied by that cpio archive is freed because it is no longer necessary. An initramfs can also be built into the kernel image; an initrd cannot, and must be provided by a bootloader.

zdw 2y ago

FWIW, Grub has a read-only ZFS implementation to allow booting: https://git.savannah.gnu.org/cgit/grub.git/tree/include/grub/zfs https://git.savannah.gnu.org/cgit/grub.git/tree/include/grub...

nubinetwork 2y ago
Grub uses an ancient version of zfs code, it's tied to Oracle's zfs and they refuse to update it to current openzfs.
- UnlockedSecrets 2y ago
  Are there any instances of features being utilized that the old version of the code is unable to cope with well enough to be able to boot the system?
  - upon_drumhead 2y ago
    Native ZFS encryption is the major one I'm aware of
  - nubinetwork 2y ago
    https://github.com/openzfs/zfs/tree/master/cmd/zpool/compatibility.d https://github.com/openzfs/zfs/tree/master/cmd/zpool/compati...
- yjftsjthsd-h 2y ago
  Refuse, or legally can't? Oracle doesn't own the copyrights on commits made after illumos forked from the corpse of opensolaris.
  - nubinetwork 2y ago
    A little of both. Everyone know about Linus' refusal to touch CDDL code, but grub isn't the kernel. There have been several attempts to add features to the grub zfs code over the years, but there are several maintainers of grub who happen to be employees of Oracle, and typically the attempts go nowhere. I personally can't recommend using grub anymore. The whole "just make 2 pools" solution is unacceptable, and until Oracle stops gatekeeping, their code becomes more obsolete in my eyes.

mjg59 2y ago

This requires a bunch of additional logic in the bootloader (eg, providing disk encryption keys), and since you're not doing this in a fully-featured OS environment (as you are in the initramfs case) it's going to be hard providing a consistent experience through the entire boot process. Having the pre-boot environment be a (slightly) cut-down version of the actual OS means all your tooling can be shared between the two environments.

zauguin 2y ago

This is a step in that direction. What they are proposing is not so much "no bootloader" but using a small Linux as bootloader. I'm using a similar setup for some time and it gives some of these advantages. Especially you get support for all relevant filesystems (you can support everything Linux supports because it is Linux), it can dynamically build a minimal initramfs with only the needed drivers if you want to and understands module dependencies (e.g. it can just dump the list of modules it uses itself) and is generally much more flexible.

rcxdude 2y ago

Does FreeBSD's loader share code with the kernel? It does seem like a lot of duplication of systems to make it work in comparison to just using the same code.

fooker 2y ago

The more smartness you put here, the more it makes life difficult for non-standard operating systems. And if this bit is closed source, and something doesn't work, you don't have a recourse.

dexen 2y ago

Seconding this. Having lucked into using Lilo and no initramfs for several years now, I'm very happy with robustness and straightforwardness of the solution. In contrast, on the rare occasion I've dealt with somebody elses' GRUB and initramfs setups, they turn out brittle and complex.