10 ms·

Booting embedded Linux in 0.37 seconds on an ARMv7-A CPU at 528 MHz

187 points by eerimoq 6y ago

yjftsjthsd-h 6y ago
> A reboot is even faster, only 0.26 seconds from issuing the reboot to entering user space. Curious; it doesn't say how that works. Could be kexec, but if it's a real reboot then I'd be interested to know why it's faster. Can you still skip some hardware initialization somehow?
- aappleby 6y ago
  Soft boot doesn't have to read the bootloader from ROM again.
- eerimoq 6y ago
  I'm curious as well, and yes, by reboot I mean calling "reboot(RB_AUTOBOOT)". I was under the impression that the ROM code has to start over from the beginning, which I think it does. Maybe it can skip some initialization since the SoC is already up and running. Maybe there is some initial hardware setup that is only done at power on. It's hard to tell as this part of the boot sequence runs closed source NXP code.
ndesaulniers 6y ago
I'm impressed. I don't know of too much work that's going on upstream for optimizing boot times, other than some of the clear linux stuff: https://www.phoronix.com/scan.php?page=news_item&px=Clear-Linux-Ubuntu-Eoan-Boot https://www.phoronix.com/scan.php?page=news_item&px=Clear-Li... There are folks looking into improving boot times on Android; turns out init and kernel drivers are a tangly mess of {dependency} spaghetti. Loading kernel modules can induce delays in processing relocations. The kernel patches disable a bunch of stuff, including ethernet it looks like? Most of the kernel changes comment out blocks of code, or trade long delays for shorter delays with more iterations.
- yjftsjthsd-h 6y ago
  Yeah, this appears to take advantage of the fact that if you know exactly what hardware you're working with, you can skip a whole lot of detection and general support, which helps quite a bit for embedded but less so on ex. your laptop. Honestly, while there may well be dependency issues, I was under the impression that at least the kernel side of things generally is pretty well optimized, and in most cases you're paying for flexibility.
  - Twirrim 6y ago
    dracut on RHEL7+ builds the initramfs in "host_only" mode, which attempts to strip it down to just the kernel modules you need for boot time. Which is also somewhat annoying because it completely trashes portability, which can be really irritating in cloud environments. Ubuntu has similar capability (and I'm guessing Debian upstream?), but you have to specifically enable it, by default it ships the full modules set in the initramfs. I would imagine most places outside of embedded world and maybe microvms, this stuff isn't that valuable.
    - dvdkhlng 6y ago
      Slight correction: by default Debian/Ubuntu put all modules potentially required for initial boot into the initramfs. That's still a very small percentage of all modules. I.e. you only need those modules that will get you so far into the boot process that you have access to the root partition with all the remaining modules. E.g. if you want to do network-boot over wifi, you'll have to add a initramfs-hook script to add the wifi modules for your hardware into the initramfs [1]. They are not included by default. [1] http://www.marcfargas.com/posts/enable-wireless-debian-initramfs/ http://www.marcfargas.com/posts/enable-wireless-debian-initr...
      - deleted 6y ago
        [deleted]
      - Twirrim 6y ago
        You're right, I misinterpreted what "most" meant in the mkinitramfs config. Interesting. I've not seen any difficulties with porting Ubuntu between different hardware configurations, so it seems to include a reasonable amount of them. Every now and then I'm tempted to try "dep" instead of "most", but then I realise there just isn't enough benefit!
WatchDog 6y ago
> Networking takes by far the longest time to get ready. The main reason is that Ethernet auto-negotiation takes a significant amount of time, about 1 to 3 seconds. Is this a fundamental limitation of how auto-negotiation works? Is there a way to speed it up?
- mschuster91 6y ago
  Set fixed values for the speed (i.e. force 1 Gbit/s if you know there will be a capable cable). I'm not sure on if the feature that detects crossover cables can be switched off in userspace. As for the timing, as long as you don't specify speed / crossover detection you will always have some sort of physical link training and negotiation...
  - derefr 6y ago
    > Physical link training and negotiation That still shouldn't take that long, though, should it? 3s sounds like some O(N^2) process is happening. Keep in mind that this stuff is happening close to the metal, on a nowadays-unshared medium (no Ethernet hubs around any more), with negligible speed-of-light delays because the nearest switch is probably ~100ft away at most. If some high-level protocol like Steam Link can have no perceivable latency, then certainly PHY negotiation shouldn't. My naive guess would be that the medium is speed-tested in order, first seeing if it works at 1Mbps, then 10Mbps, then 100Mbps, and finally 1Gbps; and alternating in the crossover-cable versions of those tests; satisficing with the last-achieved line rate when the next up-clocking fails. If that's the case, then I have a feeling that modern hardware could get a bit of an advantage just from doing things in the opposite order: 1. optimistically assuming everything is set up for 1Gbps, and then, if not, ratcheting down the link-speed until the link starts working; and 2. only doing the crossover-cable tests after all the non-crossover tests fail. You'd still have the same worst-case performance (3s) as before, but now that worst-case would be for old 1Mbps crossover cables: not a common case!
    - icedchai 6y ago
      1 Megabit ethernet was never a thing.
Polylactic_acid 6y ago
Does anyone know why x86 systems are so slow to start up? I got a x570 mobo recently and it still takes about 5 seconds to get to grub.
- kube-system 6y ago
  Before grub you might have POST and possibly an intentional delay to allow for user input to enter EFI configuation.
  - csunbird 6y ago
    Yes, most of the bios programs have the setting of "POST Boot Delay", which intentionally waits a couple of seconds to give user to interact with the bios, e.g. entering settings page or to allow user to change the boot drive.
- deleted 6y ago
  [deleted]
- p1necone 6y ago
  I know that this is part of it: https://blog.asset-intertech.com/test_data_out/2014/11/memory-training-testing-and-margining.html https://blog.asset-intertech.com/test_data_out/2014/11/memor...
  - rasz 6y ago
    memory controller training takes fraction of a second
    - hexmiles 6y ago
      Not on some amd board. It's a common complain, my previous board would have a pre-bios delay of 5 to 10 second with xmp enable and still up to 5 without xmp. I changed board and it's now a lot better, but still slower than intel. ps: xmp is probably an intel only name, but i cant remember the amd one.
nimish 6y ago
One interesting reason there's a been a bunch of progress in this space is that automotive systems are required to show the rear view camera in a certain amount of time. Progress driven by the oddest of things.
- abdulmuhaimin 6y ago
  My car takes around 10 seconds(I think more im not sure) for the reverse camera to work after starting the engine. Its annoying having to wait, thats for sure
  - PudgePacket 6y ago
    Is it pulling a fresh docker image each time?
    - saltedonion 6y ago
      Gold
      - sushshshsh 6y ago
        I've been deep in the interview cycle lately and I just had a dream last night that I was asked, in a non-technical interview with a product manager, "what are the 5 ways to dockerize an application?" I then said that I could only think of one way, and he responded "how do you not know docker if you are applying for a java developer job?" Thankfully I woke up
        merb 6y ago
        well most of the time, my company deals with docker aswell. but my coworkers have nothing to do with it, it just is fully automated. the only problem we had, when introducing long running jobs that can be run while clicking a button inside our ui, which runs a k8s job. that was hairy for my coworker, but with enough shell scripts it started to be easier and easier.
eximius 6y ago
Why can't network boot up be done asynchronously?
- eerimoq 6y ago
  You are right, it probably could. I should try compiling the fec-driver into the kernel and make it asynchronous. Should save a couple of milliseconds, and hopefully not delay entering user space.
  - mindentropy 6y ago
    Isn't network bring up done asynchronously in distros these days? I see my network interface not up and running even after the system is fully booted.
    - eerimoq 6y ago
      I'm just referring to the fec driver kernel module, not the user space software. But I might of course be wrong. I've done lots of iterations trying to optimize the boot time on this tiny embedded system, and it's not always easy to remember all details. It could be that the fec driver is asynchronously probed. I guess I have to try it again at some point. =)
- bluesign 6y ago
  “Async MMC and FEC (Ethernet) driver probes to do other initialization in parallel.” I think it is already is
Yanu-3452 6y ago
Are super-quick boots vulnerable to having a (presumably?) lower entropy pool exploited or do the steps taken to mitigate low entropy across freshly minted cloud images also help here?
- HPsquared 6y ago
  Isn't that more a question of how repeatable the process is?
dmitrygr 6y ago
> Start with MMC clock frequency at 52 MHz instaed of 400 kHz. Whoa there! Spec violation. Not guaranteed to work. Might work some days but not others.
- eerimoq 6y ago
  The goal is to not configure the MMC at all in Linux, but just rely on the bootloader has already configured it. Btw, where can I find the spec? And where in the spec can I read about this? Is this true even if we know the MMC supports 52 MHz?
  - dmitrygr 6y ago
    Spec says initialization shall be done at 400KHz or less. You can Google for sd spec
    - eerimoq 6y ago
      Will fix it at some point =)
  - OnACoffeeBreak 6y ago
    https://www.jedec.org/sites/default/files/docs/JESD84-B451.pdf https://www.jedec.org/sites/default/files/docs/JESD84-B451.p... You can use http://bugmenot.com/view/jedec.org http://bugmenot.com/view/jedec.org to get free-to-use credentials. The section in point is "A.6.1Bus initialization". I'd say that if your MMC works fine going full-speed out of the gate and the IC on the board supports this speed, then you unlikely to encounter any issues.
    - eerimoq 6y ago
      Thank you very much. Very helpful answer. I'll continue to use 52 MHz until I encounter problems (if any).
pcdoodle 6y ago
The hardware looks awesome! Looks like it hasn't been updated in 2 years though, has anyone produced these PCBs based on the Jiffy? Is that an EMMC Socket on there?
m463 6y ago
I have an intel system that takes 20x as long just to beep that there's no keyboard.

187 points by eerimoq 6y ago

> A reboot is even faster, only 0.26 seconds from issuing the reboot to entering user space. Curious; it doesn't say how that works. Could be kexec, but if it's a real reboot then I'd be interested to know why it's faster. Can you still skip some hardware initialization somehow?

aappleby 6y ago
Soft boot doesn't have to read the bootloader from ROM again.
eerimoq 6y ago
I'm curious as well, and yes, by reboot I mean calling "reboot(RB_AUTOBOOT)". I was under the impression that the ROM code has to start over from the beginning, which I think it does. Maybe it can skip some initialization since the SoC is already up and running. Maybe there is some initial hardware setup that is only done at power on. It's hard to tell as this part of the boot sequence runs closed source NXP code.

ndesaulniers 6y ago

I'm impressed. I don't know of too much work that's going on upstream for optimizing boot times, other than some of the clear linux stuff: https://www.phoronix.com/scan.php?page=news_item&px=Clear-Linux-Ubuntu-Eoan-Boot https://www.phoronix.com/scan.php?page=news_item&px=Clear-Li... There are folks looking into improving boot times on Android; turns out init and kernel drivers are a tangly mess of {dependency} spaghetti. Loading kernel modules can induce delays in processing relocations. The kernel patches disable a bunch of stuff, including ethernet it looks like? Most of the kernel changes comment out blocks of code, or trade long delays for shorter delays with more iterations.

yjftsjthsd-h 6y ago
Yeah, this appears to take advantage of the fact that if you know exactly what hardware you're working with, you can skip a whole lot of detection and general support, which helps quite a bit for embedded but less so on ex. your laptop. Honestly, while there may well be dependency issues, I was under the impression that at least the kernel side of things generally is pretty well optimized, and in most cases you're paying for flexibility.
- Twirrim 6y ago
  dracut on RHEL7+ builds the initramfs in "host_only" mode, which attempts to strip it down to just the kernel modules you need for boot time. Which is also somewhat annoying because it completely trashes portability, which can be really irritating in cloud environments. Ubuntu has similar capability (and I'm guessing Debian upstream?), but you have to specifically enable it, by default it ships the full modules set in the initramfs. I would imagine most places outside of embedded world and maybe microvms, this stuff isn't that valuable.
  - dvdkhlng 6y ago
    Slight correction: by default Debian/Ubuntu put all modules potentially required for initial boot into the initramfs. That's still a very small percentage of all modules. I.e. you only need those modules that will get you so far into the boot process that you have access to the root partition with all the remaining modules. E.g. if you want to do network-boot over wifi, you'll have to add a initramfs-hook script to add the wifi modules for your hardware into the initramfs [1]. They are not included by default. [1] http://www.marcfargas.com/posts/enable-wireless-debian-initramfs/ http://www.marcfargas.com/posts/enable-wireless-debian-initr...
    - deleted 6y ago
      [deleted]
    - Twirrim 6y ago
      You're right, I misinterpreted what "most" meant in the mkinitramfs config. Interesting. I've not seen any difficulties with porting Ubuntu between different hardware configurations, so it seems to include a reasonable amount of them. Every now and then I'm tempted to try "dep" instead of "most", but then I realise there just isn't enough benefit!

WatchDog 6y ago

> Networking takes by far the longest time to get ready. The main reason is that Ethernet auto-negotiation takes a significant amount of time, about 1 to 3 seconds. Is this a fundamental limitation of how auto-negotiation works? Is there a way to speed it up?

mschuster91 6y ago
Set fixed values for the speed (i.e. force 1 Gbit/s if you know there will be a capable cable). I'm not sure on if the feature that detects crossover cables can be switched off in userspace. As for the timing, as long as you don't specify speed / crossover detection you will always have some sort of physical link training and negotiation...
- derefr 6y ago
  > Physical link training and negotiation That still shouldn't take that long, though, should it? 3s sounds like some O(N^2) process is happening. Keep in mind that this stuff is happening close to the metal, on a nowadays-unshared medium (no Ethernet hubs around any more), with negligible speed-of-light delays because the nearest switch is probably ~100ft away at most. If some high-level protocol like Steam Link can have no perceivable latency, then certainly PHY negotiation shouldn't. My naive guess would be that the medium is speed-tested in order, first seeing if it works at 1Mbps, then 10Mbps, then 100Mbps, and finally 1Gbps; and alternating in the crossover-cable versions of those tests; satisficing with the last-achieved line rate when the next up-clocking fails. If that's the case, then I have a feeling that modern hardware could get a bit of an advantage just from doing things in the opposite order: 1. optimistically assuming everything is set up for 1Gbps, and then, if not, ratcheting down the link-speed until the link starts working; and 2. only doing the crossover-cable tests after all the non-crossover tests fail. You'd still have the same worst-case performance (3s) as before, but now that worst-case would be for old 1Mbps crossover cables: not a common case!
  - icedchai 6y ago
    1 Megabit ethernet was never a thing.

Polylactic_acid 6y ago

Does anyone know why x86 systems are so slow to start up? I got a x570 mobo recently and it still takes about 5 seconds to get to grub.

kube-system 6y ago
Before grub you might have POST and possibly an intentional delay to allow for user input to enter EFI configuation.
- csunbird 6y ago
  Yes, most of the bios programs have the setting of "POST Boot Delay", which intentionally waits a couple of seconds to give user to interact with the bios, e.g. entering settings page or to allow user to change the boot drive.
deleted 6y ago
[deleted]
p1necone 6y ago
I know that this is part of it: https://blog.asset-intertech.com/test_data_out/2014/11/memory-training-testing-and-margining.html https://blog.asset-intertech.com/test_data_out/2014/11/memor...
- rasz 6y ago
  memory controller training takes fraction of a second
  - hexmiles 6y ago
    Not on some amd board. It's a common complain, my previous board would have a pre-bios delay of 5 to 10 second with xmp enable and still up to 5 without xmp. I changed board and it's now a lot better, but still slower than intel. ps: xmp is probably an intel only name, but i cant remember the amd one.

nimish 6y ago

One interesting reason there's a been a bunch of progress in this space is that automotive systems are required to show the rear view camera in a certain amount of time. Progress driven by the oddest of things.

abdulmuhaimin 6y ago
My car takes around 10 seconds(I think more im not sure) for the reverse camera to work after starting the engine. Its annoying having to wait, thats for sure
- PudgePacket 6y ago
  Is it pulling a fresh docker image each time?
  - saltedonion 6y ago
    Gold
    - sushshshsh 6y ago
      I've been deep in the interview cycle lately and I just had a dream last night that I was asked, in a non-technical interview with a product manager, "what are the 5 ways to dockerize an application?" I then said that I could only think of one way, and he responded "how do you not know docker if you are applying for a java developer job?" Thankfully I woke up
      - merb 6y ago
        well most of the time, my company deals with docker aswell. but my coworkers have nothing to do with it, it just is fully automated. the only problem we had, when introducing long running jobs that can be run while clicking a button inside our ui, which runs a k8s job. that was hairy for my coworker, but with enough shell scripts it started to be easier and easier.

eximius 6y ago

Why can't network boot up be done asynchronously?

eerimoq 6y ago
You are right, it probably could. I should try compiling the fec-driver into the kernel and make it asynchronous. Should save a couple of milliseconds, and hopefully not delay entering user space.
- mindentropy 6y ago
  Isn't network bring up done asynchronously in distros these days? I see my network interface not up and running even after the system is fully booted.
  - eerimoq 6y ago
    I'm just referring to the fec driver kernel module, not the user space software. But I might of course be wrong. I've done lots of iterations trying to optimize the boot time on this tiny embedded system, and it's not always easy to remember all details. It could be that the fec driver is asynchronously probed. I guess I have to try it again at some point. =)
bluesign 6y ago
“Async MMC and FEC (Ethernet) driver probes to do other initialization in parallel.” I think it is already is

Yanu-3452 6y ago

Are super-quick boots vulnerable to having a (presumably?) lower entropy pool exploited or do the steps taken to mitigate low entropy across freshly minted cloud images also help here?

HPsquared 6y ago
Isn't that more a question of how repeatable the process is?

dmitrygr 6y ago

> Start with MMC clock frequency at 52 MHz instaed of 400 kHz. Whoa there! Spec violation. Not guaranteed to work. Might work some days but not others.

eerimoq 6y ago
The goal is to not configure the MMC at all in Linux, but just rely on the bootloader has already configured it. Btw, where can I find the spec? And where in the spec can I read about this? Is this true even if we know the MMC supports 52 MHz?
- dmitrygr 6y ago
  Spec says initialization shall be done at 400KHz or less. You can Google for sd spec
  - eerimoq 6y ago
    Will fix it at some point =)
- OnACoffeeBreak 6y ago
  https://www.jedec.org/sites/default/files/docs/JESD84-B451.pdf https://www.jedec.org/sites/default/files/docs/JESD84-B451.p... You can use http://bugmenot.com/view/jedec.org http://bugmenot.com/view/jedec.org to get free-to-use credentials. The section in point is "A.6.1Bus initialization". I'd say that if your MMC works fine going full-speed out of the gate and the IC on the board supports this speed, then you unlikely to encounter any issues.
  - eerimoq 6y ago
    Thank you very much. Very helpful answer. I'll continue to use 52 MHz until I encounter problems (if any).

pcdoodle 6y ago

The hardware looks awesome! Looks like it hasn't been updated in 2 years though, has anyone produced these PCBs based on the Jiffy? Is that an EMMC Socket on there?

m463 6y ago

I have an intel system that takes 20x as long just to beep that there's no keyboard.