11 ms·

I love the bit on slide 16 where initializing 8GB of RAM takes 100MS, so they boot with only 2GB and then hotplug in more memory later.

anon1m0us 7y ago
We've been progressively loading webpages for years. It's like the 7th layer of the stack is working its way down.
- satanspastaroll 7y ago
  Lazy loading is not unique for the web or in anyway a new idea. The web is in fact one of the last places it has appeared in.
  - diroussel 7y ago
    Indeed. In apps lazy loading is something you add when you need it. But for kernels, it’s the first problem you tackle: how to pick yourself up with your bootstraps. Booting is tricky now, and it always has been. Right back to the early machines that were booted by manually entering machine code via switches. Interestingly AWS virtual machines still boot in 16-bit mode, then bootstrap up through 32 and then 64 bit modes, last I heard.
    - freeone3000 7y ago
      All Intel-compatible and emulated Intel-compatible processors do this. You'd have to run ARM to avoid it.
      - yjftsjthsd-h 7y ago
        I thought x86 started in 64-bit mode when using UEFI?
        vertex-four 7y ago
        More like the UEFI puts it into 32-bit/64-bit mode before it gets to the kernel, the hardware didn’t change.
- w56rjrtyu6ru 7y ago
  Yes, if you call "the 7th layer of the stack" the people who started with it.
aidenn0 7y ago
A lot of embedded systems have been doing this for years.
cperciva 7y ago
Just last weekend I was talking to one of FreeBSD's VM gurus about doing exactly this in FreeBSD too. You have to be careful about it though -- some data structures are autosized at boot, and if you're not careful you can end up with systems "running out of memory" because they have too much memory.
- throwaway8941 7y ago
  Reminds me of Windows 95 "running out of memory" on systems with too much RAM. https://devblogs.microsoft.com/oldnewthing/20030814-00/?p=42903 https://devblogs.microsoft.com/oldnewthing/20030814-00/?p=42...
- toast0 7y ago
  > some data structures are autosized at boot It's probably a good idea to audit all the things that are autosized at boot by ram size; some of them are probably much too large on a system with 128 GB or so, given some of the sizing was written when 128 MB was big. I know I've run into the size of IP fragment buffers being way too big, but not sure what else.
- hansendc 7y ago
  On Linux, most (maybe even all) of the data structures that must grow with memory size can be carved out of the memory being added. The main culprit is 'struct page' which typically needs 64b of metadata for each 4k physical page (on x86 at least). Much of the code to do this stuff is in Linux's "sparsemem" infrastructure: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/sparse.c https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
Twirrim 7y ago
There are some drawbacks to that. Memory flagged as hotplug can't be used by the kernel itself (because otherwise unplugging the RAM would risk crashing the system). That includes things like network buffers and a number of things which use the total RAM size as a guideline. There was a bug in a proposed kernel patch a while back, that we were testing. It ended up designating a good chunk of the machine's RAM as hotpluggable. Test suite was passing, but the benchmarks showed a drastic performance drop. Took a while to track down why that was happening.
- madez 7y ago
  Isn't the solution introducing another type of memory abstraction in the kernel next to "available since boot" and "hot pluggable", namely "later added but won't be removed"?
  - hansendc 7y ago
    This solution didn't introduce it. The concepts being used have existed in Linux for quite some time. Here's a patch of mine that added (some of) it from 2005: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3947be1969a9 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- hansendc 7y ago
  I think you may be confusing some of the aspects of hot-add with hot-remove. Memory which you want to later remove should be onlined as "movable" and it has some content limitations. But, not all hot-added memory must be onlined that way. You can very easily just online it as "normal", which is (believe it or not) normal and can have any allocation type placed there.

anon1m0us 7y ago

We've been progressively loading webpages for years. It's like the 7th layer of the stack is working its way down.

satanspastaroll 7y ago
Lazy loading is not unique for the web or in anyway a new idea. The web is in fact one of the last places it has appeared in.
- diroussel 7y ago
  Indeed. In apps lazy loading is something you add when you need it. But for kernels, it’s the first problem you tackle: how to pick yourself up with your bootstraps. Booting is tricky now, and it always has been. Right back to the early machines that were booted by manually entering machine code via switches. Interestingly AWS virtual machines still boot in 16-bit mode, then bootstrap up through 32 and then 64 bit modes, last I heard.
  - freeone3000 7y ago
    All Intel-compatible and emulated Intel-compatible processors do this. You'd have to run ARM to avoid it.
    - yjftsjthsd-h 7y ago
      I thought x86 started in 64-bit mode when using UEFI?
      - vertex-four 7y ago
        More like the UEFI puts it into 32-bit/64-bit mode before it gets to the kernel, the hardware didn’t change.
w56rjrtyu6ru 7y ago
Yes, if you call "the 7th layer of the stack" the people who started with it.

aidenn0 7y ago

A lot of embedded systems have been doing this for years.

cperciva 7y ago

Just last weekend I was talking to one of FreeBSD's VM gurus about doing exactly this in FreeBSD too. You have to be careful about it though -- some data structures are autosized at boot, and if you're not careful you can end up with systems "running out of memory" because they have too much memory.

throwaway8941 7y ago
Reminds me of Windows 95 "running out of memory" on systems with too much RAM. https://devblogs.microsoft.com/oldnewthing/20030814-00/?p=42903 https://devblogs.microsoft.com/oldnewthing/20030814-00/?p=42...
toast0 7y ago
> some data structures are autosized at boot It's probably a good idea to audit all the things that are autosized at boot by ram size; some of them are probably much too large on a system with 128 GB or so, given some of the sizing was written when 128 MB was big. I know I've run into the size of IP fragment buffers being way too big, but not sure what else.
hansendc 7y ago
On Linux, most (maybe even all) of the data structures that must grow with memory size can be carved out of the memory being added. The main culprit is 'struct page' which typically needs 64b of metadata for each 4k physical page (on x86 at least). Much of the code to do this stuff is in Linux's "sparsemem" infrastructure: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/sparse.c https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Twirrim 7y ago

There are some drawbacks to that. Memory flagged as hotplug can't be used by the kernel itself (because otherwise unplugging the RAM would risk crashing the system). That includes things like network buffers and a number of things which use the total RAM size as a guideline. There was a bug in a proposed kernel patch a while back, that we were testing. It ended up designating a good chunk of the machine's RAM as hotpluggable. Test suite was passing, but the benchmarks showed a drastic performance drop. Took a while to track down why that was happening.

madez 7y ago
Isn't the solution introducing another type of memory abstraction in the kernel next to "available since boot" and "hot pluggable", namely "later added but won't be removed"?
- hansendc 7y ago
  This solution didn't introduce it. The concepts being used have existed in Linux for quite some time. Here's a patch of mine that added (some of) it from 2005: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3947be1969a9 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
hansendc 7y ago
I think you may be confusing some of the aspects of hot-add with hot-remove. Memory which you want to later remove should be onlined as "movable" and it has some content limitations. But, not all hot-added memory must be onlined that way. You can very easily just online it as "normal", which is (believe it or not) normal and can have any allocation type placed there.