7 ms·

Red Hat and CentOS systems aren’t booting due to BootHole patches

113 points by thg 6y ago

nullc 6y ago
Number of users saved from attack by secure boot: 0 Number of systems bricked by fixes to secure boot vulnerabilities: Many Sad to see TSA in software form.
fomine3 6y ago
It seems that it's really critical issue. How does this patch pass Red Hat's test?
- chousuke 6y ago
  Bugs get through testing quite frequently, and this one was probably somewhat fast-tracked because it's a security fix; it happens. FWIW, I tested upgrading a few (test) CentOS virtual machines at work to see if I can trigger this bug, but they worked fine, so perhaps the bug only triggers with a configuration they happen to have not tested.
  - cesarb 6y ago
    > I tested upgrading a few (test) CentOS virtual machines at work Were the virtual machines using EFI? Most virtual machines I've seen boot through legacy BIOS, not EFI.
    - chousuke 6y ago
      they boot using BIOS, yes. I suppose some of our EFI-booting hardware hosts might be affected, but all of our virtual machines boot using BIOS since it happens to be the default and there's not much point in switching them to EFI.
fideloper 6y ago
Ubuntu had this as well, but got a release out quickly so it seems to have not been too huge an issue. Bug report: https://bugs.launchpad.net/cloud-init/+bug/1877491 https://bugs.launchpad.net/cloud-init/+bug/1877491 Description/remediation: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/GRUB2SecureBootBypass?_ga=2.62301103.2044889230.1596151413-1800768090.1596151413 https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/GRUB2Secu...
- gruez 6y ago
  >Currently, we populate the debconf database variable grub-pc/install_devices by checking to see if a device is present in a hardcoded list [1] of directories: Is there a reason why this patch was silently applied? For something as risky as breaking the boot process, you'd think you want user confirmation before proceeding. It can obviously be done, eg. https://d11a6trkgmumsb.cloudfront.net/original/3X/f/5/f55e369ca3bd66b980e00a0f56ae0f332371fc52.png https://d11a6trkgmumsb.cloudfront.net/original/3X/f/5/f55e36.... Also, recovering from this might be easy if you're technically inclined, but it could be worse if you have FDE enabled with the boot keys sealed in TPM. Changing the boot loader or the secure boot settings in that case might lead to the TPM refusing to release the disk encryption keys, which could lead to permanent loss of data.
cvwilliams 6y ago
As usual, there are prophetic warnings from Linus: https://yarchive.net/comp/linux/efi.html https://yarchive.net/comp/linux/efi.html BIOS should be simple, because it is buggy anyway. Handing over to a bootloader in the MBR is all that a BIOS should do. Now one is at the mercy of NVRAM, grub2 and loads of gratuitous complexity.
- bzb3 6y ago
  "New software will eventually contain bugs", and then bringing it up 14 years later, seems to bring nothing to the table.
  - chris_wot 6y ago
    Hence the “prophetic” bit.
- paol 6y ago
  EFI really was a step in completely the wrong direction. Massive complexity & feature set for no good reason. Legacy BIOS had to go, it was designed at the time of 8086 and DOS and was really out of step with modern HW and OS needs. The replacement could have been even simpler though, now that OSes really want to take over everything themselves and no longer lean on the BIOS that way that DOS used to. Instead EFI created a monster and now we're stuck with it.
  - vbezhenar 6y ago
    BIOS is a good idea. It should have been expanded. For example I should be able to download Nvidia driver, install it into BIOS and then talk to it via standard interface (something like Vulkan). Now every operating system can just utilize that interface and use well tested driver from manufacturer, rather than ask manufacturer to write that driver for every operating system.
    - cptroot 6y ago
      "Via Standard Interface" I don't think you can gloss over the differences between say DirectX and Metal that easily. Maybe you're arguing they should be gotten rid of, but I think that's rather pie in the sky.
Jonnax 6y ago
Is Secure Boot on Linux actually secure? I remember reading it was like a signed loader and that's it. But I presume that's incorrect?
- deleted 6y ago
  [deleted]
- rst 6y ago
  "Is it secure?" is an incomplete question -- it has to be "Is it secure against...?". And then the answers will depend on whatever threat model winds up in the dots. That said, all secure boot even tries to assure is that the software that's booting is the same thing you thought you installed. If that is, say, a Linux distribution running a webapp which has problems, well... the boot mechanism can't save you from those.
  - Jonnax 6y ago
    Ah yes. The question I should have asked is: Is it more secure than not using it, and in what way? Like for example, the Linux kernel isn't signed, right?
    - henryfjordan 6y ago
      The code for the kernel is signed through git, but you or your distro maintainer need to then compile it. The resulting binary could be signed, but it wouldn't be by Linus/The Linux Foundation. Given that there are a million options specific to your use-case and hardware, you can't really rely on reproducible builds in the general case. Secureboot is one step in a chain of verification you'd need to do to make sure you're only running the binaries you've approved: > Secure Boot is a technology where the system firmware checks that the system boot loader is signed with a cryptographic key authorized by a database contained in the firmware. With adequate signature verification in the next-stage boot loader(s), kernel, and, potentially, user space, it is possible to prevent the execution of unsigned code. https://docs.fedoraproject.org/en-US/Fedora/18/html/UEFI_Secure_Boot_Guide/chap-UEFI_Secure_Boot_Guide-What_is_Secure_Boot.html https://docs.fedoraproject.org/en-US/Fedora/18/html/UEFI_Sec...
cpncrunch 6y ago
We have a centos 8 server that has been bricked, and there doesn't seem to be any solution right now. Downgrading the packages shows "lowest version already installed, cannot downgrade it.", and I can't manually install the old packages because there doesn't seem to be any way of getting them. Someone says to use http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/Packages/ http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/... but that seems to have the very latest packages and even after updating the affected ones it still fails to boot.
- asicsp 6y ago
  See if this helps: https://www.reddit.com/r/linux/comments/i1hlay/red_hat_and_centos_systems_arent_booting_due_to/fzxu4xs/ https://www.reddit.com/r/linux/comments/i1hlay/red_hat_and_c...
  - cpncrunch 6y ago
    It just links to the redhat info which doesn't work on centos 8.2 (error message is that those packages are already at lowest versions).
- cesarb 6y ago
  > Someone says to use http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/Packages/ http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/... but that seems to have the very latest packages At least for me, that page seems to have both shim-x64-15-13.el8.x86_64.rpm from 2020-07-29 22:10 and the older shim-x64-15-11.el8.x86_64.rpm from 2020-05-07 19:53; the older one should work. Worst case, you could manually copy the shim executables from a working server to the EFI partition of the broken server (from what I have read at https://bugzilla.redhat.com/show_bug.cgi?id=1861977 https://bugzilla.redhat.com/show_bug.cgi?id=1861977, in the RHEL/CentOS case it's the shim executable which is broken, so you don't have to do anything to the grub executables).
  - cpncrunch 6y ago
    Yes, I see I needed to look for the 81 version numbers. I have now installed those packages (and confirmed I have the older grubx64.efi in /boot/efi/EFI/). However it is still giving the same error on reboot. I installed these rpms (which are the affected ones for my system): grub2-common-2.02-81.el8.noarch.rpm grub2-efi-x64-2.02-81.el8.x86_64.rpm grub2-pc-2.02-81.el8.x86_64.rpm grub2-pc-modules-2.02-81.el8.noarch.rpm grub2-tools-2.02-81.el8.x86_64.rpm grub2-tools-efi-2.02-81.el8.x86_64.rpm grub2-tools-extra-2.02-81.el8.x86_64.rpm grub2-tools-minimal-2.02-81.el8.x86_64.rpm Is anything else needed? I'm thinking the easiest solution is for me just to wait a few days and do a "yum update" in rescue mode once an update fix is available. Luckily this is a non-critical server.
ginko 6y ago
I still don't understand why this even needed to be fixed. Finding a way to circumvent UEFI DRM seems to be a good thing.
- Wowfunhappy 6y ago
  Secure boot can usually be turned off if its unwanted. I agree its not necessary on every device—but if it is enabled, the OS should assume the user wants security at that level of the chain. It certainly shouldn't just circumvent it as a matter of course.
  - cesarb 6y ago
    > Secure boot can usually be turned off if its unwanted. Emphasis mine. The main objection to secure boot is that, some time in the future, it will be mandatory; that already is the case for Windows devices with ARM CPUs (https://www.softwarefreedom.org/blog/2012/jan/12/microsoft-confirms-UEFI-fears-locks-down-ARM/ https://www.softwarefreedom.org/blog/2012/jan/12/microsoft-c...).
    - my123 6y ago
      Hello, Windows Client (laptops/tablets/...) devices with 64-bit Arm CPUs have Secure Boot unlockable. That article applied to Windows RT and Windows Phone, which were earlier projects.
      - Lammy 6y ago
        My objection to Secure Boot is right in the name. What's it designed to be secure from? From me, the user.
        Wowfunhappy 6y ago
        But it's not, it's meant to be secure from rootkits! A lot of secure boot implementations let you add your own keys. Some don't, and that's bad, but it's not the fault of secure boot!
fortran77 6y ago
That's the great thing about open source, though. You guys can fix it yourself. Me, I'm stuck on Windows 10.
- overcast 6y ago
  There's probably more people capable of fixing Windows 10 at Microsoft, then there are people capable of fixing bootloaders in Linux in the world. Just because something is open source, doesn't mean anyone can just jump in there and change the oil.
  - fortran77 6y ago
    Exactly my point.
qalmakka 6y ago
I'm kind of out of the loop - is BootHole a UEFI or GRUB2 vulnerability?
- enzanki_ars 6y ago
  BootHole is a GRUB2 vulnerability that could allow a local attacker to bypass Secure Boot protections. The patches for RHEL/CentOS lead to failures in booting due to a crash in the UEFI loader for GRUB2.
noahbliss 6y ago
Hey all, just wanted to make you aware of the mortar project here: https://github.com/noahbliss/mortar https://github.com/noahbliss/mortar It takes a comprehensive approach rather than piecemeal like a lot of these patches, leveraging technology already in your system to build a conceptually airtight and fully audited system. Happy to get some of your opinions on it, constructive criticism, and pull requests!
benttoothpaste 6y ago
The most secure boot is the one that does not take place ;) So maybe there is a method in this madness?
cpncrunch 6y ago
Fixes now available for RHEL, still waiting for CentOS.

113 points by thg 6y ago

Number of users saved from attack by secure boot: 0 Number of systems bricked by fixes to secure boot vulnerabilities: Many Sad to see TSA in software form.

fomine3 6y ago

It seems that it's really critical issue. How does this patch pass Red Hat's test?

chousuke 6y ago
Bugs get through testing quite frequently, and this one was probably somewhat fast-tracked because it's a security fix; it happens. FWIW, I tested upgrading a few (test) CentOS virtual machines at work to see if I can trigger this bug, but they worked fine, so perhaps the bug only triggers with a configuration they happen to have not tested.
- cesarb 6y ago
  > I tested upgrading a few (test) CentOS virtual machines at work Were the virtual machines using EFI? Most virtual machines I've seen boot through legacy BIOS, not EFI.
  - chousuke 6y ago
    they boot using BIOS, yes. I suppose some of our EFI-booting hardware hosts might be affected, but all of our virtual machines boot using BIOS since it happens to be the default and there's not much point in switching them to EFI.

fideloper 6y ago

Ubuntu had this as well, but got a release out quickly so it seems to have not been too huge an issue. Bug report: https://bugs.launchpad.net/cloud-init/+bug/1877491 https://bugs.launchpad.net/cloud-init/+bug/1877491 Description/remediation: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/GRUB2SecureBootBypass?_ga=2.62301103.2044889230.1596151413-1800768090.1596151413 https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/GRUB2Secu...

gruez 6y ago
>Currently, we populate the debconf database variable grub-pc/install_devices by checking to see if a device is present in a hardcoded list [1] of directories: Is there a reason why this patch was silently applied? For something as risky as breaking the boot process, you'd think you want user confirmation before proceeding. It can obviously be done, eg. https://d11a6trkgmumsb.cloudfront.net/original/3X/f/5/f55e369ca3bd66b980e00a0f56ae0f332371fc52.png https://d11a6trkgmumsb.cloudfront.net/original/3X/f/5/f55e36.... Also, recovering from this might be easy if you're technically inclined, but it could be worse if you have FDE enabled with the boot keys sealed in TPM. Changing the boot loader or the secure boot settings in that case might lead to the TPM refusing to release the disk encryption keys, which could lead to permanent loss of data.

cvwilliams 6y ago

As usual, there are prophetic warnings from Linus: https://yarchive.net/comp/linux/efi.html https://yarchive.net/comp/linux/efi.html BIOS should be simple, because it is buggy anyway. Handing over to a bootloader in the MBR is all that a BIOS should do. Now one is at the mercy of NVRAM, grub2 and loads of gratuitous complexity.

bzb3 6y ago
"New software will eventually contain bugs", and then bringing it up 14 years later, seems to bring nothing to the table.
- chris_wot 6y ago
  Hence the “prophetic” bit.
paol 6y ago
EFI really was a step in completely the wrong direction. Massive complexity & feature set for no good reason. Legacy BIOS had to go, it was designed at the time of 8086 and DOS and was really out of step with modern HW and OS needs. The replacement could have been even simpler though, now that OSes really want to take over everything themselves and no longer lean on the BIOS that way that DOS used to. Instead EFI created a monster and now we're stuck with it.
- vbezhenar 6y ago
  BIOS is a good idea. It should have been expanded. For example I should be able to download Nvidia driver, install it into BIOS and then talk to it via standard interface (something like Vulkan). Now every operating system can just utilize that interface and use well tested driver from manufacturer, rather than ask manufacturer to write that driver for every operating system.
  - cptroot 6y ago
    "Via Standard Interface" I don't think you can gloss over the differences between say DirectX and Metal that easily. Maybe you're arguing they should be gotten rid of, but I think that's rather pie in the sky.

Jonnax 6y ago

Is Secure Boot on Linux actually secure? I remember reading it was like a signed loader and that's it. But I presume that's incorrect?

deleted 6y ago
[deleted]
rst 6y ago
"Is it secure?" is an incomplete question -- it has to be "Is it secure against...?". And then the answers will depend on whatever threat model winds up in the dots. That said, all secure boot even tries to assure is that the software that's booting is the same thing you thought you installed. If that is, say, a Linux distribution running a webapp which has problems, well... the boot mechanism can't save you from those.
- Jonnax 6y ago
  Ah yes. The question I should have asked is: Is it more secure than not using it, and in what way? Like for example, the Linux kernel isn't signed, right?
  - henryfjordan 6y ago
    The code for the kernel is signed through git, but you or your distro maintainer need to then compile it. The resulting binary could be signed, but it wouldn't be by Linus/The Linux Foundation. Given that there are a million options specific to your use-case and hardware, you can't really rely on reproducible builds in the general case. Secureboot is one step in a chain of verification you'd need to do to make sure you're only running the binaries you've approved: > Secure Boot is a technology where the system firmware checks that the system boot loader is signed with a cryptographic key authorized by a database contained in the firmware. With adequate signature verification in the next-stage boot loader(s), kernel, and, potentially, user space, it is possible to prevent the execution of unsigned code. https://docs.fedoraproject.org/en-US/Fedora/18/html/UEFI_Secure_Boot_Guide/chap-UEFI_Secure_Boot_Guide-What_is_Secure_Boot.html https://docs.fedoraproject.org/en-US/Fedora/18/html/UEFI_Sec...

cpncrunch 6y ago

We have a centos 8 server that has been bricked, and there doesn't seem to be any solution right now. Downgrading the packages shows "lowest version already installed, cannot downgrade it.", and I can't manually install the old packages because there doesn't seem to be any way of getting them. Someone says to use http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/Packages/ http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/... but that seems to have the very latest packages and even after updating the affected ones it still fails to boot.

asicsp 6y ago
See if this helps: https://www.reddit.com/r/linux/comments/i1hlay/red_hat_and_centos_systems_arent_booting_due_to/fzxu4xs/ https://www.reddit.com/r/linux/comments/i1hlay/red_hat_and_c...
- cpncrunch 6y ago
  It just links to the redhat info which doesn't work on centos 8.2 (error message is that those packages are already at lowest versions).
cesarb 6y ago
> Someone says to use http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/Packages/ http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/... but that seems to have the very latest packages At least for me, that page seems to have both shim-x64-15-13.el8.x86_64.rpm from 2020-07-29 22:10 and the older shim-x64-15-11.el8.x86_64.rpm from 2020-05-07 19:53; the older one should work. Worst case, you could manually copy the shim executables from a working server to the EFI partition of the broken server (from what I have read at https://bugzilla.redhat.com/show_bug.cgi?id=1861977 https://bugzilla.redhat.com/show_bug.cgi?id=1861977, in the RHEL/CentOS case it's the shim executable which is broken, so you don't have to do anything to the grub executables).
- cpncrunch 6y ago
  Yes, I see I needed to look for the 81 version numbers. I have now installed those packages (and confirmed I have the older grubx64.efi in /boot/efi/EFI/). However it is still giving the same error on reboot. I installed these rpms (which are the affected ones for my system): grub2-common-2.02-81.el8.noarch.rpm grub2-efi-x64-2.02-81.el8.x86_64.rpm grub2-pc-2.02-81.el8.x86_64.rpm grub2-pc-modules-2.02-81.el8.noarch.rpm grub2-tools-2.02-81.el8.x86_64.rpm grub2-tools-efi-2.02-81.el8.x86_64.rpm grub2-tools-extra-2.02-81.el8.x86_64.rpm grub2-tools-minimal-2.02-81.el8.x86_64.rpm Is anything else needed? I'm thinking the easiest solution is for me just to wait a few days and do a "yum update" in rescue mode once an update fix is available. Luckily this is a non-critical server.

ginko 6y ago

I still don't understand why this even needed to be fixed. Finding a way to circumvent UEFI DRM seems to be a good thing.

Wowfunhappy 6y ago
Secure boot can usually be turned off if its unwanted. I agree its not necessary on every device—but if it is enabled, the OS should assume the user wants security at that level of the chain. It certainly shouldn't just circumvent it as a matter of course.
- cesarb 6y ago
  > Secure boot can usually be turned off if its unwanted. Emphasis mine. The main objection to secure boot is that, some time in the future, it will be mandatory; that already is the case for Windows devices with ARM CPUs (https://www.softwarefreedom.org/blog/2012/jan/12/microsoft-confirms-UEFI-fears-locks-down-ARM/ https://www.softwarefreedom.org/blog/2012/jan/12/microsoft-c...).
  - my123 6y ago
    Hello, Windows Client (laptops/tablets/...) devices with 64-bit Arm CPUs have Secure Boot unlockable. That article applied to Windows RT and Windows Phone, which were earlier projects.
    - Lammy 6y ago
      My objection to Secure Boot is right in the name. What's it designed to be secure from? From me, the user.
      - Wowfunhappy 6y ago
        But it's not, it's meant to be secure from rootkits! A lot of secure boot implementations let you add your own keys. Some don't, and that's bad, but it's not the fault of secure boot!

fortran77 6y ago

That's the great thing about open source, though. You guys can fix it yourself. Me, I'm stuck on Windows 10.

overcast 6y ago
There's probably more people capable of fixing Windows 10 at Microsoft, then there are people capable of fixing bootloaders in Linux in the world. Just because something is open source, doesn't mean anyone can just jump in there and change the oil.
- fortran77 6y ago
  Exactly my point.

qalmakka 6y ago

I'm kind of out of the loop - is BootHole a UEFI or GRUB2 vulnerability?

enzanki_ars 6y ago
BootHole is a GRUB2 vulnerability that could allow a local attacker to bypass Secure Boot protections. The patches for RHEL/CentOS lead to failures in booting due to a crash in the UEFI loader for GRUB2.

noahbliss 6y ago

Hey all, just wanted to make you aware of the mortar project here: https://github.com/noahbliss/mortar https://github.com/noahbliss/mortar It takes a comprehensive approach rather than piecemeal like a lot of these patches, leveraging technology already in your system to build a conceptually airtight and fully audited system. Happy to get some of your opinions on it, constructive criticism, and pull requests!

benttoothpaste 6y ago

The most secure boot is the one that does not take place ;) So maybe there is a method in this madness?

cpncrunch 6y ago

Fixes now available for RHEL, still waiting for CentOS.