4 ms·

Evolution of the Unix System Architecture: An Exploratory Case Study

103 points by rodrigo975 5y ago

pram 5y ago
"We could not study Unix versions that derive from the Research editions via AT&T System V, such as Solaris, AIX, and HPUX, because most of the corresponding code remains proprietary and inaccessible." Understandable but disappointing. HPUX and AIX in particular are probably the closest thing to "living" UNIX time capsules.
- userbinator 5y ago
  ...and even more disappointing because a quick search of the Internets found both AIX and HPUX source. But then again, given IEEE member's pro-DRM pro-copyright stance, maybe they didn't want to.
  - hulitu 5y ago
    If you know too much we will have to ... you. It's sad that in the western civilization, copyright and DRM are used to limit access to information.
  - marbu 5y ago
    Even if you found a single snapshot of allegedly AIX or HPUX source code of unknown origin, you would have a problem with referencing to it so that your claim can be both trusted and verified. Moreover it would still be nowhere near the level of detail we have for systems with public source code, where you have snapshots of most versions, later on we have particular commits ...
- jjav 5y ago
  Solaris was open sourced as OpenSolaris, so they do have access to that. Oracle stopped merging to it, to noone's surprise, but the source of what's essentially Solaris 10 is available and formed the basis of illumos and other related projects. https://en.wikipedia.org/wiki/OpenSolaris https://en.wikipedia.org/wiki/OpenSolaris
  - throwawaylinux 5y ago
    There was a very interesting time around 2000 where SMP scalability was a very hot topic among the serious open source kernels, with heated debates in Linux and BSD communities (it precipitated the forking of FreeBSD into DragonflyBSD) about how they should be designed, or whether it was even feasible to have the same kernel design that would work well on a desktop as a high end server. Proprietary UNIXes had already gone through this, and scaled up to pretty big machines like the Sun E10K. Although none were aimed at small desktop machines, and they tended to have a reputation (deserved or not) for being weighted down by, among other things, locking ("Slowaris"). Complex layers of locks, which open source developers feared would result in a complexity explosion that would sink their kernels. Linux famously was granted the use of the patented RCU algorithm by IBM, and that's the way they went. Linux does have a lot of locks and locking complexity, but RCU massively helped on that front. The free BSDs did trail in scalability but they ended up developing other similar techniques and lock management and debugging facilities that has seemed to hold them in pretty good stead despite the huge developer-hours advantage Linux has. All this stuff was pretty interesting to me so when OpenSolaris was released later, I had to take a look and see what they did. And it was a big clunky looking thing. They had big hash tables of locks, layers of them. Hashes of locks is a simple trick for scalability which can work very well, but it also constrains your data structures significantly. A nice salable and cache-friendly tree can't (so easily) be used with hashed locks, hash tables are the more natural structure you're steered to. Which can be great and simple for some cases, but are not very cache friendly in most accesses with locality (even cache friendly hashes are a bit hacky or at least come with tradeoffs, not that I remember seeing any of those). I went back and looked at it again -- like a page cache page lookup in Solaris was layers locks, recursive locks etc. inode->i_contents rwlock(R) (may be recursed) page pse mutex (protects page selock) page selock (shared/exclusive, custom lock) page hash lock (hash of locks protecting page cache hash table) 4 locks, they all nest, recurse, etc, and they tend to be these complex boutique special lock functions -- the page se mutex and hash locks are taken and released by special locking functions (usr/src/uts/common/vm/page_lock.c) hundreds of lines of special lock beahvior. It looks like the hash lock can be avoided in the fast path, but I list it because the nature of it means the page cache structure is a global hash table (which doesn't scale in terms of memory usage, and has poor cache locality, and problematic collision QoS issues, possibly DoS type attacks between privilege domains, etc. Just a poor, clunky data structure for the job. The equivalent path in Linux has no locks: lock-free lookup of a tree attached to the inode to find the page and increment its refcount. It really is quite amazing the leaps and bounds this type of scalable programming techniques have come since the turn of the century contrasting Linux and Solaris (today now Linux is much more scalable than Solaris ever was). I can absolutely see why people were so worried these things would end in a complexity death spiral.
- pjmlp 5y ago
  Although Aix has quite a few capabilities that aren't that UNIX and more like VMS/Windows. The way shared libraries work, for example.
  - trasz 5y ago
    Can you say a bit more about the libraries?
    - pjmlp 5y ago
      Easy, just like Windows nowadays, because Aix uses COFF not ELF, and you also get private by default and description files listing which symbols are public. https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=library-compiling-shared https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=libr... CreateExportList is somehow similar to creating an import library in Windows, https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=library-exporting-symbols-createexportlist-utility https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=libr... Here are all details, https://download.boulder.ibm.com/ibmdl/pub/software/dw/aix/es-aix_ll.pdf https://download.boulder.ibm.com/ibmdl/pub/software/dw/aix/e...
      - rjsw 5y ago
        AT&T SysV UNIX used COFF and had a similar process for creating shared libraries, don't think you can claim that it is not UNIX-style.
        pjmlp 5y ago
        Might be, but when I came to Aix 5, all other UNIXes used ELF so I never used that classical version. In fact, I never created shared libraries in either Xenix or DG/UX, and my first Linux was with kernel 1.0.9, where ELF was introduced. So thanks for the correction.
    - inkyoto 5y ago
      AIX introduced an extended COFF object and executable file format, XCOFF, that has eliminated the need to have both, static and shared, library files – the .a files were «both» at once and the linker switch had to be used to select between a statically or dynamically linked executable type. Which caused a lot of confusion for some at the time leading them to believe that AIX had no shared libraries.
dmolony 5y ago
[PDF]
throwawaylinux 5y ago
Extremely interesting trend in cyclomatic complexity increasing then dropping in what looks like all systems and programs analyzed. Interesting that Linux kernel is unusually low, about 4.5 while all the others (except gnu coreutils) are around 6.
cat199 5y ago
"In this paper we study the evolution of Unix along the FreeBSD lineage" For those that don't realize BSD is Unix.
gandalfgeek 5y ago
Short explainer video for those who don’t want to read the whole thing: https://youtu.be/eCYkzciF28E https://youtu.be/eCYkzciF28E

103 points by rodrigo975 5y ago

pram 5y ago

"We could not study Unix versions that derive from the Research editions via AT&T System V, such as Solaris, AIX, and HPUX, because most of the corresponding code remains proprietary and inaccessible." Understandable but disappointing. HPUX and AIX in particular are probably the closest thing to "living" UNIX time capsules.

userbinator 5y ago
...and even more disappointing because a quick search of the Internets found both AIX and HPUX source. But then again, given IEEE member's pro-DRM pro-copyright stance, maybe they didn't want to.
- hulitu 5y ago
  If you know too much we will have to ... you. It's sad that in the western civilization, copyright and DRM are used to limit access to information.
- marbu 5y ago
  Even if you found a single snapshot of allegedly AIX or HPUX source code of unknown origin, you would have a problem with referencing to it so that your claim can be both trusted and verified. Moreover it would still be nowhere near the level of detail we have for systems with public source code, where you have snapshots of most versions, later on we have particular commits ...
jjav 5y ago
Solaris was open sourced as OpenSolaris, so they do have access to that. Oracle stopped merging to it, to noone's surprise, but the source of what's essentially Solaris 10 is available and formed the basis of illumos and other related projects. https://en.wikipedia.org/wiki/OpenSolaris https://en.wikipedia.org/wiki/OpenSolaris
- throwawaylinux 5y ago
  There was a very interesting time around 2000 where SMP scalability was a very hot topic among the serious open source kernels, with heated debates in Linux and BSD communities (it precipitated the forking of FreeBSD into DragonflyBSD) about how they should be designed, or whether it was even feasible to have the same kernel design that would work well on a desktop as a high end server. Proprietary UNIXes had already gone through this, and scaled up to pretty big machines like the Sun E10K. Although none were aimed at small desktop machines, and they tended to have a reputation (deserved or not) for being weighted down by, among other things, locking ("Slowaris"). Complex layers of locks, which open source developers feared would result in a complexity explosion that would sink their kernels. Linux famously was granted the use of the patented RCU algorithm by IBM, and that's the way they went. Linux does have a lot of locks and locking complexity, but RCU massively helped on that front. The free BSDs did trail in scalability but they ended up developing other similar techniques and lock management and debugging facilities that has seemed to hold them in pretty good stead despite the huge developer-hours advantage Linux has. All this stuff was pretty interesting to me so when OpenSolaris was released later, I had to take a look and see what they did. And it was a big clunky looking thing. They had big hash tables of locks, layers of them. Hashes of locks is a simple trick for scalability which can work very well, but it also constrains your data structures significantly. A nice salable and cache-friendly tree can't (so easily) be used with hashed locks, hash tables are the more natural structure you're steered to. Which can be great and simple for some cases, but are not very cache friendly in most accesses with locality (even cache friendly hashes are a bit hacky or at least come with tradeoffs, not that I remember seeing any of those). I went back and looked at it again -- like a page cache page lookup in Solaris was layers locks, recursive locks etc. inode->i_contents rwlock(R) (may be recursed) page pse mutex (protects page selock) page selock (shared/exclusive, custom lock) page hash lock (hash of locks protecting page cache hash table) 4 locks, they all nest, recurse, etc, and they tend to be these complex boutique special lock functions -- the page se mutex and hash locks are taken and released by special locking functions (usr/src/uts/common/vm/page_lock.c) hundreds of lines of special lock beahvior. It looks like the hash lock can be avoided in the fast path, but I list it because the nature of it means the page cache structure is a global hash table (which doesn't scale in terms of memory usage, and has poor cache locality, and problematic collision QoS issues, possibly DoS type attacks between privilege domains, etc. Just a poor, clunky data structure for the job. The equivalent path in Linux has no locks: lock-free lookup of a tree attached to the inode to find the page and increment its refcount. It really is quite amazing the leaps and bounds this type of scalable programming techniques have come since the turn of the century contrasting Linux and Solaris (today now Linux is much more scalable than Solaris ever was). I can absolutely see why people were so worried these things would end in a complexity death spiral.
pjmlp 5y ago
Although Aix has quite a few capabilities that aren't that UNIX and more like VMS/Windows. The way shared libraries work, for example.
- trasz 5y ago
  Can you say a bit more about the libraries?
  - pjmlp 5y ago
    Easy, just like Windows nowadays, because Aix uses COFF not ELF, and you also get private by default and description files listing which symbols are public. https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=library-compiling-shared https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=libr... CreateExportList is somehow similar to creating an import library in Windows, https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=library-exporting-symbols-createexportlist-utility https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=libr... Here are all details, https://download.boulder.ibm.com/ibmdl/pub/software/dw/aix/es-aix_ll.pdf https://download.boulder.ibm.com/ibmdl/pub/software/dw/aix/e...
    - rjsw 5y ago
      AT&T SysV UNIX used COFF and had a similar process for creating shared libraries, don't think you can claim that it is not UNIX-style.
      - pjmlp 5y ago
        Might be, but when I came to Aix 5, all other UNIXes used ELF so I never used that classical version. In fact, I never created shared libraries in either Xenix or DG/UX, and my first Linux was with kernel 1.0.9, where ELF was introduced. So thanks for the correction.
  - inkyoto 5y ago
    AIX introduced an extended COFF object and executable file format, XCOFF, that has eliminated the need to have both, static and shared, library files – the .a files were «both» at once and the linker switch had to be used to select between a statically or dynamically linked executable type. Which caused a lot of confusion for some at the time leading them to believe that AIX had no shared libraries.

dmolony 5y ago

[PDF]

throwawaylinux 5y ago

Extremely interesting trend in cyclomatic complexity increasing then dropping in what looks like all systems and programs analyzed. Interesting that Linux kernel is unusually low, about 4.5 while all the others (except gnu coreutils) are around 6.

cat199 5y ago

"In this paper we study the evolution of Unix along the FreeBSD lineage" For those that don't realize BSD is Unix.

gandalfgeek 5y ago

Short explainer video for those who don’t want to read the whole thing: https://youtu.be/eCYkzciF28E https://youtu.be/eCYkzciF28E