6 ms·

How to optimize boot time in user space on a Raspberry Pi 4 / Boot2Qt

37 points by jandeboevrie 2y ago

cvilgan 2y ago
I wonder how this even works, as PID 1 should exit right away, no? And I thought this would lead to an immediate kernel panic?
- megous 2y ago
  Yeah, I'd say it simply doesn't work as described.
- halayli 2y ago
  what do you mean pid 1 should exit right away? pid 1 is the init process and is the first and last process in the OS life cycle. All other processes run as child processes of init.
  - cvilgan 2y ago
    Ah, sorry, that was badly written. What I meant was that it looks like the shell script exits almost immediately, i.e. pid 1 forks the app process but then exits itself...
    - halayli 2y ago
      got it. The confusion then is spawning(no pun) from the assumption that pid 1 script forks while it doesn't. It calls a program and wait for it to exit. Only then pid 1 exits and the system shuts down.
megous 2y ago
This just shows how proprietary boot flow of RPi is slow. On other platforms you can get bootloader to hand off to kernel in subsecond timeframes, sometimes as fast as in 0.3s. Then kernel can be done in another 0.3s or faster, and then userspace can still be optimized more than what's described in the article, too. Instead of shell and executing programs, which requires filesystem access, loading them, loading shared libraries, etc. you can have a specialized, tiny statically built init binary which just does all initialization/mounts via syscalls and then runs your app (which should be built statically, too). This way you can achieve ~1-2s boot to UI on even way slower platforms than Rpi4. I did something like this on Pinephone a few years ago and recently again on Luckfox Pico Mini, which is a slow Cortex-A7 with DDR2 memory. There's no justification for Rpi to be booting for 12s. Even unoptimized standard U-Boot on common platforms often runs for just ~1s before handing off to the kernel.
- deleted 2y ago
  [deleted]
- PhilipRoman 2y ago
  >On other platforms you can get bootloader to hand off to kernel in subsecond timeframes, sometimes as fast as in 0.3s. Any good resources about this? I absolutely loathe the pre-kernel boot times we get at $WORK, even when using a supposedly minimal bootloader. Just decompressing the kernel takes like 10 seconds.
  - megous 2y ago
    For 0.3s you need specialized bootloader, like eg. levinboot https://gitlab.com/DeltaGem/levinboot https://gitlab.com/DeltaGem/levinboot for Rockchip RK3399 or https://xnux.eu/p-boot/ https://xnux.eu/p-boot/ for Allwinner A64. Or make something similar for your chosen platform with U-Boot's falcon mode and making the kernel as small as possible, so that not much needs to be loaded for the initial UI (put in modules what can wait, like wifi driver, etc.). For around 1-2s you need a platform that has good SD or eMMC host implementation in U-Boot, and doesn't spend too much time initializing DRAM, or in U-Boot SPL code before MMU/CPU data cache is enabled. If the CPU is slow or not upclocked in U-Boot, not doing decompression in bootloader is also a tradeoff option that may end up being faster. Some newer Rockchip RK35xx SoCs have pretty fast boot times in this regard. (RK3399 interstingly doesn't) It's all necessarily SoC specific. On some platforms (RV1103/6) I've also experimented with using bootrom functions that normally load U-Boot to load the kernel directly, so with that approach, DTB, and kernel is being loaded right away, without re-initializing eMMC or SD card or having to load and execute a complicated DT driven bootloader like U-Boot at all. But that required quite bit of work and some reverse engineering of the Boot ROM code, so that's not just a optimization.
dividuum 2y ago
Even if your userspace is pretty fast, some necessary initialization happening before the kernel is even loaded means the you probably never get lower than 6-8 seconds. There is a lengthy EEPROM issue thread about that for the Pi4: https://github.com/raspberrypi/firmware/issues/1375 https://github.com/raspberrypi/firmware/issues/1375
joezydeco 2y ago
"Instead of finding and optimizing tasks, we just turned systemd off". Okay. The folks at Bootlin have way more interesting approaches and techniques to review. I'd look to them. https://bootlin.com/blog/tag/boot-time/ https://bootlin.com/blog/tag/boot-time/
- moffkalast 2y ago
  I mean systemd-analyze blame usually shows only a few services taking up 90% of the boot time (looking at you snap daemon and cloud init taking a whole 20 seconds lmao) so turning off those yields most of the gain without any real usability hit for general use. But in practical terms if you really need boot speed, you need a Pi 5 that only takes a few seconds at worst, not a Pi 4 which uses its slow ass GPU to run the boot process for absolute maximum sluggishness possible.
  - arghwhat 2y ago
    s/boot process/initial boot phase/ A few seconds is still very long - in automotive you’re dealing with a few hundred milliseconds from power is applied till the system is presenting an interactive GUI.
    - joezydeco 2y ago
      Automotive HMIs with Linux tend to boot long, then hibernate while the car is off. The only thing that needs to come up quickly (at least in the US) is the backup camera and there tends to be hardware to inject that video quicker than the HMI can.
  - notpushkin 2y ago
    This just prompted me to finally look into boot time on my Asahi Linux setup. Turns out Docker was eating 5 seconds (about half of the boot time!) for some reason, so I disabled it (and enabled docker.socket instead, so that it starts up automatically when I need it).
    - 2y ago

37 points by jandeboevrie 2y ago

I wonder how this even works, as PID 1 should exit right away, no? And I thought this would lead to an immediate kernel panic?

megous 2y ago
Yeah, I'd say it simply doesn't work as described.
halayli 2y ago
what do you mean pid 1 should exit right away? pid 1 is the init process and is the first and last process in the OS life cycle. All other processes run as child processes of init.
- cvilgan 2y ago
  Ah, sorry, that was badly written. What I meant was that it looks like the shell script exits almost immediately, i.e. pid 1 forks the app process but then exits itself...
  - halayli 2y ago
    got it. The confusion then is spawning(no pun) from the assumption that pid 1 script forks while it doesn't. It calls a program and wait for it to exit. Only then pid 1 exits and the system shuts down.

megous 2y ago

This just shows how proprietary boot flow of RPi is slow. On other platforms you can get bootloader to hand off to kernel in subsecond timeframes, sometimes as fast as in 0.3s. Then kernel can be done in another 0.3s or faster, and then userspace can still be optimized more than what's described in the article, too. Instead of shell and executing programs, which requires filesystem access, loading them, loading shared libraries, etc. you can have a specialized, tiny statically built init binary which just does all initialization/mounts via syscalls and then runs your app (which should be built statically, too). This way you can achieve ~1-2s boot to UI on even way slower platforms than Rpi4. I did something like this on Pinephone a few years ago and recently again on Luckfox Pico Mini, which is a slow Cortex-A7 with DDR2 memory. There's no justification for Rpi to be booting for 12s. Even unoptimized standard U-Boot on common platforms often runs for just ~1s before handing off to the kernel.

deleted 2y ago
[deleted]
PhilipRoman 2y ago
>On other platforms you can get bootloader to hand off to kernel in subsecond timeframes, sometimes as fast as in 0.3s. Any good resources about this? I absolutely loathe the pre-kernel boot times we get at $WORK, even when using a supposedly minimal bootloader. Just decompressing the kernel takes like 10 seconds.
- megous 2y ago
  For 0.3s you need specialized bootloader, like eg. levinboot https://gitlab.com/DeltaGem/levinboot https://gitlab.com/DeltaGem/levinboot for Rockchip RK3399 or https://xnux.eu/p-boot/ https://xnux.eu/p-boot/ for Allwinner A64. Or make something similar for your chosen platform with U-Boot's falcon mode and making the kernel as small as possible, so that not much needs to be loaded for the initial UI (put in modules what can wait, like wifi driver, etc.). For around 1-2s you need a platform that has good SD or eMMC host implementation in U-Boot, and doesn't spend too much time initializing DRAM, or in U-Boot SPL code before MMU/CPU data cache is enabled. If the CPU is slow or not upclocked in U-Boot, not doing decompression in bootloader is also a tradeoff option that may end up being faster. Some newer Rockchip RK35xx SoCs have pretty fast boot times in this regard. (RK3399 interstingly doesn't) It's all necessarily SoC specific. On some platforms (RV1103/6) I've also experimented with using bootrom functions that normally load U-Boot to load the kernel directly, so with that approach, DTB, and kernel is being loaded right away, without re-initializing eMMC or SD card or having to load and execute a complicated DT driven bootloader like U-Boot at all. But that required quite bit of work and some reverse engineering of the Boot ROM code, so that's not just a optimization.

dividuum 2y ago

Even if your userspace is pretty fast, some necessary initialization happening before the kernel is even loaded means the you probably never get lower than 6-8 seconds. There is a lengthy EEPROM issue thread about that for the Pi4: https://github.com/raspberrypi/firmware/issues/1375 https://github.com/raspberrypi/firmware/issues/1375

joezydeco 2y ago

"Instead of finding and optimizing tasks, we just turned systemd off". Okay. The folks at Bootlin have way more interesting approaches and techniques to review. I'd look to them. https://bootlin.com/blog/tag/boot-time/ https://bootlin.com/blog/tag/boot-time/

moffkalast 2y ago
I mean systemd-analyze blame usually shows only a few services taking up 90% of the boot time (looking at you snap daemon and cloud init taking a whole 20 seconds lmao) so turning off those yields most of the gain without any real usability hit for general use. But in practical terms if you really need boot speed, you need a Pi 5 that only takes a few seconds at worst, not a Pi 4 which uses its slow ass GPU to run the boot process for absolute maximum sluggishness possible.
- arghwhat 2y ago
  s/boot process/initial boot phase/ A few seconds is still very long - in automotive you’re dealing with a few hundred milliseconds from power is applied till the system is presenting an interactive GUI.
  - joezydeco 2y ago
    Automotive HMIs with Linux tend to boot long, then hibernate while the car is off. The only thing that needs to come up quickly (at least in the US) is the backup camera and there tends to be hardware to inject that video quicker than the HMI can.
- notpushkin 2y ago
  This just prompted me to finally look into boot time on my Asahi Linux setup. Turns out Docker was eating 5 seconds (about half of the boot time!) for some reason, so I disabled it (and enabled docker.socket instead, so that it starts up automatically when I need it).
  - 2y ago