11 ms·
Linux Kernel Fastboot [pdf]
- NelsonMinar 7y agoI love the bit on slide 16 where initializing 8GB of RAM takes 100MS, so they boot with only 2GB and then hotplug in more memory later.
- anon1m0us 7y agoWe've been progressively loading webpages for years. It's like the 7th layer of the stack is working its way down.
- satanspastaroll 7y agoLazy loading is not unique for the web or in anyway a new idea. The web is in fact one of the last places it has appeared in.
- diroussel 7y agoIndeed. In apps lazy loading is something you add when you need it. But for kernels, it’s the first problem you tackle: how to pick yourself up with your bootstraps. Booting is tricky now, and it always has been. Right back to the early machines that were booted by manually entering machine code via switches. Interestingly AWS virtual machines still boot in 16-bit mode, then bootstrap up through 32 and then 64 bit modes, last I heard.
- freeone3000 7y agoAll Intel-compatible and emulated Intel-compatible processors do this. You'd have to run ARM to avoid it.
- yjftsjthsd-h 7y agoI thought x86 started in 64-bit mode when using UEFI?
- vertex-four 7y agoMore like the UEFI puts it into 32-bit/64-bit mode before it gets to the kernel, the hardware didn’t change.
- w56rjrtyu6ru 7y agoYes, if you call "the 7th layer of the stack" the people who started with it.
- aidenn0 7y agoA lot of embedded systems have been doing this for years.
- cperciva 7y agoJust last weekend I was talking to one of FreeBSD's VM gurus about doing exactly this in FreeBSD too. You have to be careful about it though -- some data structures are autosized at boot, and if you're not careful you can end up with systems "running out of memory" because they have too much memory.
- throwaway8941 7y agoReminds me of Windows 95 "running out of memory" on systems with too much RAM. https://devblogs.microsoft.com/oldnewthing/20030814-00/?p=42903 https://devblogs.microsoft.com/oldnewthing/20030814-00/?p=42...
- toast0 7y ago> some data structures are autosized at boot It's probably a good idea to audit all the things that are autosized at boot by ram size; some of them are probably much too large on a system with 128 GB or so, given some of the sizing was written when 128 MB was big. I know I've run into the size of IP fragment buffers being way too big, but not sure what else.
- hansendc 7y agoOn Linux, most (maybe even all) of the data structures that must grow with memory size can be carved out of the memory being added. The main culprit is 'struct page' which typically needs 64b of metadata for each 4k physical page (on x86 at least). Much of the code to do this stuff is in Linux's "sparsemem" infrastructure: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/sparse.c https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- Twirrim 7y agoThere are some drawbacks to that. Memory flagged as hotplug can't be used by the kernel itself (because otherwise unplugging the RAM would risk crashing the system). That includes things like network buffers and a number of things which use the total RAM size as a guideline. There was a bug in a proposed kernel patch a while back, that we were testing. It ended up designating a good chunk of the machine's RAM as hotpluggable. Test suite was passing, but the benchmarks showed a drastic performance drop. Took a while to track down why that was happening.
- madez 7y agoIsn't the solution introducing another type of memory abstraction in the kernel next to "available since boot" and "hot pluggable", namely "later added but won't be removed"?
- hansendc 7y agoThis solution didn't introduce it. The concepts being used have existed in Linux for quite some time. Here's a patch of mine that added (some of) it from 2005: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3947be1969a9 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- hansendc 7y agoI think you may be confusing some of the aspects of hot-add with hot-remove. Memory which you want to later remove should be onlined as "movable" and it has some content limitations. But, not all hot-added memory must be onlined that way. You can very easily just online it as "normal", which is (believe it or not) normal and can have any allocation type placed there.
- cperciva 7y agoThose of you who are interested in such things might also like to watch my BSDCan 2018 talk: https://www.youtube.com/watch?v=HMywejyXB9k https://www.youtube.com/watch?v=HMywejyXB9k
- jmccorm 7y agoHow nostalgic! Mind you, I say that in a good way! This brings me back to the heydays of Sun Microsystems where they controlled both the hardware (SPARC) and the operating system (Solaris). Sun didn't hesitate to use this to their advantage. With Clear Linux, I see Intel doing some very similar things. Yes, this was just a story about boot time, but it led me to take a look at Clear Linux and it looks like it could be a star. I'm interested in how much more performance their distribtuion is able to squeeze out. I'm also wondering how they're dividing their focus here and what high notes they're trying to achieve. At a strategic level, I'm assuming that they're wanting to distinguish themselves from ARM? Are they looking to create a well-tuned reference for other Linux distributions on Intel to adopt? At a tactical level, I see they're focusing on performance, availability (which boot time would be a function of)... but are they tweaking for other things such as reliability? Fault tolerance? Until now, Clear Linux wasn't even on my radar. Today, it is. Anyone have some experiences with this distro (for better or worse) that they'd like to share?
- Filligree 7y ago> Anyone have some experiences with this distro (for better or worse) that they'd like to share? Not much of one, but for a while I was playing with multiple distros trying to find the best possible power management for my laptop. (XPS 13 9380.) Clear did better than the Ubuntu installation that came with it -- but a dead octopus would have done better than that. It was sitting at 6W for non-CPU-intensive work, and was also noticeably snapppier. Windows on the same hardware sits at 8W when idling; same as Ubuntu. More recently I've tried Arch, though. Which somehow got that same workload down to 3.3W, albeit with a lot of adjustments... that didn't seem to affect performance, but did affect power use, clearly. It's just too bad that all of this optimization goes out the window if I fire up a browser, but I've had a fun time finding ways to avoid that. Apart from that, obviously Clear's software repository is far more limited than Arch's, and it also seems harder to contribute to. I don't think that problem will be going away soon.
- nobleach 7y agoI'd love to hear more about the tweaks you've done to Arch. I've gone little beyond a few power management things myself (On a Thinkpad X1C Gen 6). I get decent battery life, but I know it could be better. And giving up Arch would be a tough sell to myself.
- doener 7y agoDiscussion on Reddit: https://www.reddit.com/r/linux/comments/d34rvg/how_intels_clear_linux_team_cut_the_kernel_boot/ https://www.reddit.com/r/linux/comments/d34rvg/how_intels_cl...
- craftyguy 7y ago> Just remove the sleep(2700). Such high quality discussion too!
- why-oh-why 7y ago> SATA driver init takes 100 to 200 ms even without a real disk While it’s a joke, they’re not particularly far from the truth, as noted in a different comment here. They do indeed wait for some parts to respond.
- msclrhd 7y agoWould it be possible on multithreaded architectures to have a thread listening for the response, then setting a flag in /proc or somewhere else to indicate that the part is now active? That way, init code that does not depend on that device can continue to run.
- craftyguy 7y ago2700 ms is 27x further from the truth than 100 ms.
- newnewpdro 7y ago> Systemd is ~1.5MB - the loading time for emmc is 100ms Surely not all of systemd is faulted in immediately, is this really an issue?
- amluto 7y agoIf you want just the important parts faulted in, you need to get the important parts into their own pages, preferably in order. I don’t know whether profile-guided linking is a thing.
- newnewpdro 7y agoOn my long-running debian box, /lib/systemd/systemd is 1.1MiB in size. In /proc/1/smaps, I find the following executable mapping: 55aa05f89000-55aa06076000 r-xp 00000000 fd:01 1823 /lib/systemd/systemd Size: 948 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 752 kB Pss: 489 kB Shared_Clean: 504 kB Shared_Dirty: 0 kB Private_Clean: 248 kB Private_Dirty: 0 kB Referenced: 740 kB Anonymous: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: rd ex mr mw me dw So it appears 740KiB has been referenced, 752KiB resident. There's also the read-only mapping: 55aa06076000-55aa0609b000 r--p 000ec000 fd:01 1823 /lib/systemd/systemd Size: 148 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 148 kB Pss: 74 kB Shared_Clean: 0 kB Shared_Dirty: 148 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 148 kB Anonymous: 148 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: rd mr mw me dw ac Another 148KiB, so 900KiB, a good chunk of the 1.1MiB - though this is with a substantial uptime. The really disgusting part is all the dependencies; `pmap 1` says 57MiB total, and there's not much anonymous memory to blame.
- jtchang 7y ago> SATA driver init takes 100 to 200 ms even without a real disk That's pretty interesting. Wonder what it is doing?
- mises 7y agoProbably checking for a real disk? I guess it's got to check all the bus nodes for a disk, even if there's not one.
- ComputerGuru 7y agoSATA controller firmware has a hard-coded “drive predelay” that waits before enumerating attached devices, a relic from the days long before SSDs. Even when an option to set a custom amount is available in the BIOS, it is in addition to and not instead of, as there’s no bus master to tell it all attached disks have init and the controller may speak to them. It wouldn’t necessarily have to be done synchronously up front at all, except when the BIOS hands off control of the system to the SATA controller it expects that afterward it has a definitive list of available boot devices and their properties, for selection/display in the boot menu and BIOS configuration. Theoretically with UEFI and it’s NVRAM-resident settings, a boot device could be expressed as a namespace and path, and the firmware need not enumerate devices until it has determined that there’s a need to search for alternative/fallback boot devices, but the stack hasn’t been updated with that in mind.
- wtallis 7y agoThe delay in question is in initializing the Linux driver, not in the system firmware scanning for bootable devices before running Linux. The Linux driver may be taking a while for similar reasons as the firmware delay you describe. The hard drive predelay is also not as obsolete as you may think; I've encountered high-capacity SATA SSDs that take a surprisingly long time to come up after being hotplugged, to the point that Linux doesn't always succeed in bringing up the link on the first try.
- ComputerGuru 7y agoSorry, I missed that in the discussion. Thanks. I’m willing to bet if the stack enforced tighter tolerances for modern non-rotating disks, manufacturers wouldn’t be able to slip such shoddy firmwares into production.
- mises 7y agoI've seriously got to look into running arch with a clear linux kernel. I knew they did some pretty serious optimization work, but this is definitely impressive.
- ComputerGuru 7y agoMost of the bloat isn’t from the kernel. That’s the last thing you really optimize after getting rid of all the background services, startup tasks, etc.
- mises 7y agoClear kernel performed better in every single test run by phoronix, except one (because it has retpoline and others did not). https://www.phoronix.com/scan.php?page=article&item=arch-antergos-clear&num=4 https://www.phoronix.com/scan.php?page=article&item=arch-ant...
- ComputerGuru 7y agoI think you misunderstood. I was saying in a typical stack, time from POST to desktop is significantly overshadowed by userland bloat, not that Clear kernel is not worth the optimization.
- big_chungus 7y agoYeah, but a stock arch system is significantly less slowed than a typical stack. Getting to bootloader is still significantly longer than actual OS boot, though. I'm not usually a supporter of the "all software has to be open source" thing, but between spying issues and horrible code, it seems like coreboot is a serious benefit. Wish I could get it on my machine.
- l3s2d 7y agoSome relevant links: https://aur.archlinux.org/packages/linux-clear/ https://aur.archlinux.org/packages/linux-clear/ https://www.reddit.com/r/archlinux/comments/7rcvxc/clear_linux_kernel_for_archlinux/ https://www.reddit.com/r/archlinux/comments/7rcvxc/clear_lin...
- zamadatix 7y ago> “mem=4096m” in cmdline to only init 2 GB What is going on that makes this 2:1?
- sedatk 7y ago2gb user + 2gb kernel?
- trdtaylor1 7y ago2 gb user + 1 gb swap + 1 gb rootfs unpacked in ram perhaps. They discuss a decompression operation and the only portion of the sequence that would occur in normally is unpacking a rootfs in RAM.
- wtallis 7y agoThe kernel itself is compressed, and among the first things that happen after the bootloader hands off to the kernel is that the stub at the beginning of the kernel decompresses the rest of the kernel. And in this case they don't appear to be using any initrd for the root filesystem, just hurrying to bring up access to the full filesystem over eMMC.
- naner 7y agoJust a mistake, should be 2048.
- hansendc 7y agomem= doesn't control the amount of RAM. It actually just specifies the largest physical address the kernel should try to use. The first 4GB of physical memory typically has some non-RAM things in it like the part of the physical address space that's reserved for talking to PCI devices. If you have 2GB of RAM, 2GB of "other stuff" in the first 4GB, and specify mem=4096m, you'll end up with a kernel that sees 2GB of RAM.
- hungra 7y agoAwesome! I'm seriously thinking about running Gentoo with Linux Clear kernel!
- jjuhl 7y ago"running Gentoo" - Sure, waste your life compiling code for next to nothing performance benefits that will never add up to the extra time it took to compile the code..
- bmurphy1976 7y agoI'd really like to get my raspberry pi booting in a couple seconds. I have a few ideas for things I'd like to try but the boot time is too long. Don't need X11 but I would like to use raspbian.
- voltagex_ 7y agoThe Pi 4 is definitely much faster on boot. I wonder how much of your bottleneck is the SD read speed, though.
- mallets 7y agoYes, I'm getting around 10-15 secs boot time for a headless raspbian. I'm sure it could be made even faster with a lightweight application specific distro.
- fmntf 7y agoIf you do not need the graphical stack, there is room for optimization both in kernel and user space. On a similar board (4 ARM cores) with Ubuntu I managed to start user space applications in <2 seconds (from power on) and the complete boot (including Xorg/MATE) in <10 seconds. If you are interested, on Medium I have some write-up (same username, last post).
- akx 7y agoA link to said write-up: https://medium.com/@fmntf/adapting-ubuntu-for-the-automotive-quick-boot-and-power-loss-resilience-74d25915250c https://medium.com/@fmntf/adapting-ubuntu-for-the-automotive...
- joezydeco 7y agoThink like an embedded developer. They’ve done a lot of work in this area. https://elinux.org/Boot_Time https://elinux.org/Boot_Time
- KKKKkkkk1 7y agoDo I understand correctly that the motivation for this is that the rear-view camera on the Chery Exeed LX runs Linux? Is it a good idea to ship a car with safety components running Linux?
- kchamplewski 7y agoAs opposed to running what? Linux (as in, the kernel) is extremely stable and well tested. If the components in question utilise any significant features of the kernel, I strongly doubt some homegrown kernel-esque project will achieve more stability and reliability than Linux. Sure, if it can be done with a (minimal) amount of embedded code then it quite possibly should be, but if a whole kernel is actually needed then Linux is a fine choice.
- detaro 7y agoQNX is still fairly popular in cars. Sometimes also QNX for the more critical things and Linux for the rest.
- izacus 7y agoA huge percent of cars are already using Linux on their head units. The ones showing the backup camera. E.g. new Mazdas have a Linux distro with DBus communicating between binaries and Opera browser to show UI. The actual UI is written in JavaScript and HTML.
- Jonnax 7y agoBetter to use a proprietary OS?
- curlypaul924 7y agoI miss the days when I could reverse without the aid of a computer.
- katzeilla 7y agoP10 > Rootfs Mouting -> Rootfs Mounting ?
- baybal2 7y agoI have hard time understanding why do they have to have a supervisor there to begin with
- ahmtokta 7y agoCan friends with more information and experience write their recommendations? Looks satisfactory
- maika72 7y agoSite seems Not Found, is there any backup?