3 ms·

That would explain why my PXE installs broke yet again (with that oh so familiar error message about a mismatch between the kernel and modules versions)! Fortu

That would explain why my PXE installs broke yet again (with that oh so familiar error message about a mismatch between the kernel and modules versions)! Fortunately, it only takes a minute to grab the latest mini.iso, mount it, grab the kernel and initial ramdisk images (and "cat" the non-free/unofficial firmware to the end of the ramdisk), push 'em up to the TFTP server, and be right back in business.

mkesper 7y ago
This happens every time a new kernel is pushed out (security fixes...). Debian's suggested way is to use the debian-installer-netboot packages for updating, e.g. https://packages.debian.org/buster/debian-installer-10-netboot-amd64 https://packages.debian.org/buster/debian-installer-10-netbo...
- jlgaddis 7y ago
  Yep, but the TFTP server doesn't run Debian so this is the easiest "solution" for me. When it does break, it literally takes less than five minutes to get it working again. Looks like I'd been using the previous kernel and initramfs since 2019-09-05 and, well, I can deal with spending five minutes on an issue once every five months -- especially considering all the time that it saves me! (At one point, I insisted -- for some unknown reason! -- on using the weekly "unstable" images so the breakage was much, much more frequent. Fortunately for my sanity, I finally gave up and went back to just using the latest "stable" mini.iso.)
mattl 7y ago
https://fiendish.github.io/The-Debian-Gotham-Needs/ https://fiendish.github.io/The-Debian-Gotham-Needs/ might help?
- jlgaddis 7y ago
  Short answer: Thanks for the suggestion but, no, this won't work for me. --- Long answer: Unfortunately, no, that doesn't help me as I don't really even need an ISO image at all. What I really need is just the new version of the kernel ("linux") and initial ramdisk ("initrd.gz") files. I don't need the regular ("cdrom") ones (like you might find on a regular ISO image), though. I need the "netboot" ones -- yes, they're different! Well, the kernel is usually the same but the initrd.gz is different. When a new version of the kernel and modules show up on the servers (and once my local "cache/proxy" notices them and downloads the newer version of the modules), my network installs will start failing. See, a host PXE boots the existing version of the kernel but when debian-installer launches and starts downloading the modules it needs, it finds that they were built for a different version of the kernel than what it booted from. This causes "that oh so familiar error message" to be thrown [0] and the installation to fail. The "fix", obviously, is to boot the new kernel instead of the old one. As I mentioned, all that I really need to do is replace the existing versions of the linux and initrd.gz files on my TFTP server with the new ones. In theory, I could just SSH into the TFTP server, "cd" to the appropriate directory, fire off a single "wget" (the URLs for those two files don't even change!), and be done; real life is, however, (only slightly) more difficult! Now, once in a very great while, I will actually do an installation on a VM where I do use an ISO image instead of performing a network install. That's why I just go ahead and grab the ("netboot") "mini.iso" file and extract the linux and initrd.gz files out of it instead of downloading just those files in the first place. The other thing has to do with the "non-free" firmware. Although they provide ISO images that include it, it was easier for me to just do it myself. Once I've got the linux and initrd.gz files downloaded, all that is needed is to download the "firmware.cpio.gz" file if I don't already have it (or if a newer version is available or something) and (literally) "cat" initrd.gz and firmware.cpio.gz together. Then, I replace the existing linux and initrd.gz files on my TFTP server with these two new files and reboot the host so that it restarts the network install! Depending on the hardware, it sometimes takes the server longer to POST and get to the point where it actually starts booting than it takes for the actual OS installation (I'm looking at you, Dell PowerEdge servers!). It may sound like a fustercluck but it's actually pretty simple (just a few commands) and from "uh oh, the install failed" to "okay, it's working again" is almost always less than five minutes (well, it is now that I switched back to using the "stable" mini.iso images instead of the daily or weekly "unstable" ones!). I could write a script to automate it, I suppose, but it doesn't seem worth it as it only happens two or three times a year. (That's a rather long way of saying "no, that doesn't help" but since you offered a suggestion I wanted to explain why and not simply dismiss it with a "no, that doesn't help"!) --- [0]: https://i.imgur.com/3cB5LC3g.jpg https://i.imgur.com/3cB5LC3g.jpg

mkesper 7y ago

This happens every time a new kernel is pushed out (security fixes...). Debian's suggested way is to use the debian-installer-netboot packages for updating, e.g. https://packages.debian.org/buster/debian-installer-10-netboot-amd64 https://packages.debian.org/buster/debian-installer-10-netbo...

jlgaddis 7y ago
Yep, but the TFTP server doesn't run Debian so this is the easiest "solution" for me. When it does break, it literally takes less than five minutes to get it working again. Looks like I'd been using the previous kernel and initramfs since 2019-09-05 and, well, I can deal with spending five minutes on an issue once every five months -- especially considering all the time that it saves me! (At one point, I insisted -- for some unknown reason! -- on using the weekly "unstable" images so the breakage was much, much more frequent. Fortunately for my sanity, I finally gave up and went back to just using the latest "stable" mini.iso.)

mattl 7y ago

https://fiendish.github.io/The-Debian-Gotham-Needs/ https://fiendish.github.io/The-Debian-Gotham-Needs/ might help?

jlgaddis 7y ago
Short answer: Thanks for the suggestion but, no, this won't work for me. --- Long answer: Unfortunately, no, that doesn't help me as I don't really even need an ISO image at all. What I really need is just the new version of the kernel ("linux") and initial ramdisk ("initrd.gz") files. I don't need the regular ("cdrom") ones (like you might find on a regular ISO image), though. I need the "netboot" ones -- yes, they're different! Well, the kernel is usually the same but the initrd.gz is different. When a new version of the kernel and modules show up on the servers (and once my local "cache/proxy" notices them and downloads the newer version of the modules), my network installs will start failing. See, a host PXE boots the existing version of the kernel but when debian-installer launches and starts downloading the modules it needs, it finds that they were built for a different version of the kernel than what it booted from. This causes "that oh so familiar error message" to be thrown [0] and the installation to fail. The "fix", obviously, is to boot the new kernel instead of the old one. As I mentioned, all that I really need to do is replace the existing versions of the linux and initrd.gz files on my TFTP server with the new ones. In theory, I could just SSH into the TFTP server, "cd" to the appropriate directory, fire off a single "wget" (the URLs for those two files don't even change!), and be done; real life is, however, (only slightly) more difficult! Now, once in a very great while, I will actually do an installation on a VM where I do use an ISO image instead of performing a network install. That's why I just go ahead and grab the ("netboot") "mini.iso" file and extract the linux and initrd.gz files out of it instead of downloading just those files in the first place. The other thing has to do with the "non-free" firmware. Although they provide ISO images that include it, it was easier for me to just do it myself. Once I've got the linux and initrd.gz files downloaded, all that is needed is to download the "firmware.cpio.gz" file if I don't already have it (or if a newer version is available or something) and (literally) "cat" initrd.gz and firmware.cpio.gz together. Then, I replace the existing linux and initrd.gz files on my TFTP server with these two new files and reboot the host so that it restarts the network install! Depending on the hardware, it sometimes takes the server longer to POST and get to the point where it actually starts booting than it takes for the actual OS installation (I'm looking at you, Dell PowerEdge servers!). It may sound like a fustercluck but it's actually pretty simple (just a few commands) and from "uh oh, the install failed" to "okay, it's working again" is almost always less than five minutes (well, it is now that I switched back to using the "stable" mini.iso images instead of the daily or weekly "unstable" ones!). I could write a script to automate it, I suppose, but it doesn't seem worth it as it only happens two or three times a year. (That's a rather long way of saying "no, that doesn't help" but since you offered a suggestion I wanted to explain why and not simply dismiss it with a "no, that doesn't help"!) --- [0]: https://i.imgur.com/3cB5LC3g.jpg https://i.imgur.com/3cB5LC3g.jpg