5 ms·

A typical system has dozens or hundreds of processes running even when nothing is going on. If they all had their own copy of every system library in RAM, it wo

A typical system has dozens or hundreds of processes running even when nothing is going on. If they all had their own copy of every system library in RAM, it wouldn’t look so free. Especially on mobile devices.

zingermc 7y ago
What if the OS could dedupe pages?
- kelnos 7y ago
  Do any OSes actually do this, though? If not, this is academic until one actually does. Linux does have something called "KSM" that de-dupes memory pages, but it appears that it's only so a hypervisor can de-dupe guest OS pages? Odd that it's not more general than that. Though I guess there is a cost: the KSM code needs to scan through memory to find duplicate pages, which I can't imagine is fast.
- loeg 7y ago
  On the fly memory dedupe is incredibly expensive. Especially compared to just not duplicating in the first place. Also, it's quite likely that static linked libraries wouldn't be located at the exact same page offsets every time they're linked in to a binary, making page-level dedupe not useful.
- adrianmonk 7y ago
  You'd need to arrange for the linking process to result in identical (and identically-aligned!) code pages. If you're going to all the trouble to ensure that, why not just write it all to a file and call it a shared library?
  - bayareanative 7y ago
    Exactly. Emperors' new clothes from naive convention abandonment of solved problems.
- sebcat 7y ago
  What if the OS could map a shared object that resides in one section of memory into the virtual address space of multiple processes?
- choeger 7y ago
  Would not work with any kind of LTO.
CamperBob2 7y ago
Meh, not so much. The processes will only page in the code they actually use. Shared libraries are a goofy hack dating back to a time when people mistakenly thought they needed them, and they need to go away.
- Annatar 7y ago
  Shared object libraries are incredibly useful because they save memory and CPU resources. Cloning the process space during a fork(2) call, initializing structures, then context switching and whatnot is incredibly expensive.
  - coldtea 7y ago
    >Cloning the process space during a fork(2) call, initializing structures, then context switching and whatnot is incredibly expensive. Err, we're running Electron apps today, with a full blown embedded browser engine, a vm, and a hellish complex DOM for rendering. Others run apps inside containers, with their own basic OS and standard libs -- compared to that the above is negligible.
    - saagarjha 7y ago
      > we're running Electron apps today Some of us try not to on the belief that they don't perform acceptably.
    - bayareanative 7y ago
      Nope. A strawman wrong doesn't justify anything related to the topic. fork() duplicates everything, which maybe overkill, which is why pthreads and vfork+exec also exist.
      - notacoward 7y ago
        That's not why pthreads exist, and vfork barely does. Pthreads exist to facilitate parallel operation (hence the name) on shared data, not as an efficiency measure. As for vfork, it has been deprecated since every UNIX implementation started marking pages as copy-on-write instead of actually copying. I was doing some of that work myself in 1990-91, so it's hardly new. The real problems with static linking don't have to do with fork, except in the sense of increasing reliance on VM overcommit (which IMO is a bad idea already). It's more to do with masking reuse of the same library across unrelated processes, as an efficiency issue but even more importantly in terms of tracking dependencies and updating software. No matter how good your configuration management (or similar mechanism) is, rebuilding large-N statically linked executables is less efficient and more error-prone than using the same information to update small-N shared libraries.

zingermc 7y ago

What if the OS could dedupe pages?

kelnos 7y ago
Do any OSes actually do this, though? If not, this is academic until one actually does. Linux does have something called "KSM" that de-dupes memory pages, but it appears that it's only so a hypervisor can de-dupe guest OS pages? Odd that it's not more general than that. Though I guess there is a cost: the KSM code needs to scan through memory to find duplicate pages, which I can't imagine is fast.
loeg 7y ago
On the fly memory dedupe is incredibly expensive. Especially compared to just not duplicating in the first place. Also, it's quite likely that static linked libraries wouldn't be located at the exact same page offsets every time they're linked in to a binary, making page-level dedupe not useful.
adrianmonk 7y ago
You'd need to arrange for the linking process to result in identical (and identically-aligned!) code pages. If you're going to all the trouble to ensure that, why not just write it all to a file and call it a shared library?
- bayareanative 7y ago
  Exactly. Emperors' new clothes from naive convention abandonment of solved problems.
sebcat 7y ago
What if the OS could map a shared object that resides in one section of memory into the virtual address space of multiple processes?
choeger 7y ago
Would not work with any kind of LTO.

CamperBob2 7y ago

Meh, not so much. The processes will only page in the code they actually use. Shared libraries are a goofy hack dating back to a time when people mistakenly thought they needed them, and they need to go away.

Annatar 7y ago
Shared object libraries are incredibly useful because they save memory and CPU resources. Cloning the process space during a fork(2) call, initializing structures, then context switching and whatnot is incredibly expensive.
- coldtea 7y ago
  >Cloning the process space during a fork(2) call, initializing structures, then context switching and whatnot is incredibly expensive. Err, we're running Electron apps today, with a full blown embedded browser engine, a vm, and a hellish complex DOM for rendering. Others run apps inside containers, with their own basic OS and standard libs -- compared to that the above is negligible.
  - saagarjha 7y ago
    > we're running Electron apps today Some of us try not to on the belief that they don't perform acceptably.
  - bayareanative 7y ago
    Nope. A strawman wrong doesn't justify anything related to the topic. fork() duplicates everything, which maybe overkill, which is why pthreads and vfork+exec also exist.
    - notacoward 7y ago
      That's not why pthreads exist, and vfork barely does. Pthreads exist to facilitate parallel operation (hence the name) on shared data, not as an efficiency measure. As for vfork, it has been deprecated since every UNIX implementation started marking pages as copy-on-write instead of actually copying. I was doing some of that work myself in 1990-91, so it's hardly new. The real problems with static linking don't have to do with fork, except in the sense of increasing reliance on VM overcommit (which IMO is a bad idea already). It's more to do with masking reuse of the same library across unrelated processes, as an efficiency issue but even more importantly in terms of tracking dependencies and updating software. No matter how good your configuration management (or similar mechanism) is, rebuilding large-N statically linked executables is less efficient and more error-prone than using the same information to update small-N shared libraries.