4 ms·

The Undocumented Microsoft “Rich” Header (2018)

123 points by Lammy 5y ago

jchw 5y ago
This seems like it might be useful for reverse engineering projects aiming to do source code reconstruction; if a Rich header is present, presumably that would guide you to being able to figure out if you have the exact builds of the tools that the original binary was made with. Interesting.
jes 5y ago
I was hoping to read this article on my iPhone. But the darn thing won’t narrow and stay narrowed so that it can be read without scrolling left and right to read each line. Is it specific to me or is this some behavior specific to this site?
- Item_Boring 5y ago
  I had the same issue but switching to reader view “fixed it“ :)
- Stratoscope 5y ago
  Same on Android, but Chrome pops up a "Show simplified view" on the bottom that fixes it right up (at the expense of wrapping the hex dumps).
- hbn 5y ago
  Click the aA button in Safari on the left of the URL bar -> "Show reader view" It won't show the highlighting on the hex dump, but you get the article text in a more readable format for the device
- lelandfe 5y ago
  It's because all the text is inside of a big table and there are several long rows. Combined, that means that the text can't really be resized below 1250px. Don't use tables for design, folks.
  - martin_a 5y ago
    > Don't use tables for design, folks. Wow, feels like 15 years ago, when everybody moved from table layouts to divs.
ndiddy 5y ago
One use that the article doesn’t mention: If a large company ships a binary that was compiled with Visual Studio Community, Microsoft knows that they should do a license audit :)
Ericson2314 5y ago
It might just be nostalgia to the first computers I used, but the shear complexity, mystery, and stability that allows this stuff to accrete in Windows fills me with a sense of Wonder, even if it is also terrible. Unix is very less bloated, which is better, but also means there is basically no sense of Wonder. So Linux, my daily driver, I gripe about, but Windows, which I thankfully never have to use, makes my eyebrows raise and lips smile from a safe distance.
- xeromal 5y ago
  Linux is like using a constructed language. It's precise, constantly pared down, and optimized. It's effective and boring. Windows is like English. A mishmash of different languages with no perfect rules. It's a pain in the ass to get right. Lots of fun nuggets to find amusing though.
  - skissane 5y ago
    > A mishmash of different languages with no perfect rules There is some of that in Unix/Linux as well, even if to a lesser degree than Windows exhibits it. One good example is the dd command–both the command name and its rather unique syntax are inspired by the IBM mainframe operating system MVS (nowadays rebranded as z/OS) and the omnipresent DD statement of its (in)famous JCL job control language, although the inspiration is only at a very high level and the actual details are quite different. Another is the fact that one of the fields in the /etc/passwd file is named pw_gecos, after the mainframe operating system General Electric Comprehensive Operating System (GECOS), which was used at Bell Labs in the early days of Unix. Some of those early Unix systems used the Bell Labs GECOS mainframe for print spooling, and pw_gecos was used to store the GECOS account name for each user. pw_gecos is still around, and has even spread from Unix into much newer systems such as LDAP and SCIM, although nowadays it stores the user's full name instead, and sometimes other contact details such as email or phone number as well. GECOS is still around too (if a shadow of its former self), having been passed from General Electric to Honeywell and since then to the French firm Groupe Bull (now part of Atos), and having been renamed from GECOS to GCOS along the way. The Unix terminal subsystem (line drivers, termios, terminfo/termcap, curses, etc) is full of all kinds of crazy legacy crud which made sense back when people used to use real physical terminals running over RS-232, but now in the 21st century it is basically just legacy complexity we are stuck with due to inertia, backward compatibility, and the inability of any redesign to offer enough value to overcome all that inertia. Current versions of Solaris [0] and its derivatives (such as Illumos) still contain the header file "rje.h", which is for an old Bell Labs system for submitting batch jobs to IBM mainframes. I doubt the actual code has been present for a long-time – the header references "/dev/dn2", which I believe is a PDP-11 modem device (see [1], [2]) – but I take it the Solaris developers never removed this file just in case doing so broke backward compatibility, maybe some developer somewhere included it in their C code and removing the file might stop their code from compiling. Linux, lacking the same heritage as Solaris, doesn't contain anything quite as ancient as this. The use of "VTOC" as a term for partition table by a number of Unix systems (including Solaris) is inherited from IBM mainframe systems, where VTOC is one of the most fundamental on-disk structures – although in normal use more like a root directory with embedded allocation information than an actual partition table, but the partition table usage is inspired by the fact that when Unix was ported to IBM mainframes, the mainframe VTOC filesystem was used as a partition table, with each Unix filesystem being stored as a separate file ("dataset" to use the proper mainframe terminology) in the mainframe filesystem. Linux can run on IBM mainframes (z/Linux), and on them it actually literally supports the IBM VTOC filesystem as a partition table format [3], along with a bunch of other weird mainframe features–use of 3270 terminals as a system console [4] and even integration with virtual card punches and card readers [5]. Filesystems in particular are an area in which the Linux kernel supports a huge amount of legacy cruft, with all manner of rarely used filesystems supported for backward compatibility, for example the ADFS filesystem of Acorn RISC OS [6], the classic Mac OS HFS filesystem [7], the HPFS filesystem of OS/2 [8], the "System V" filesystem used by old x86 Unix systems such as Xenix and Coherent [9], the "Boot File System" used by the boot process of SCO Unixware [10] [0] see for example https://github.com/illumos/illumos-gate/blob/master/usr/src/head/rje.h https://github.com/illumos/illumos-gate/blob/master/usr/src/... [1] http://squoze.net/UNIX/v5man/pdf/man4/dn.pdf http://squoze.net/UNIX/v5man/pdf/man4/dn.pdf [2] https://github.com/illumos/illumos-gate/blob/master/usr/src/head/rje.h https://github.com/illumos/illumos-gate/blob/master/usr/src/... [3] https://github.com/torvalds/linux/blob/master/arch/s390/include/uapi/asm/vtoc.h https://github.com/torvalds/linux/blob/master/arch/s390/incl... [4] https://github.com/torvalds/linux/blob/master/Documentation/s390/3270.rst https://github.com/torvalds/linux/blob/master/Documentation/... [5] https://github.com/torvalds/linux/blob/master/drivers/s390/char/vmur.c https://github.com/torvalds/linux/blob/master/drivers/s390/c... [6] https://github.com/torvalds/linux/blob/master/Documentation/filesystems/adfs.rst https://github.com/torvalds/linux/blob/master/Documentation/... [7] https://github.com/torvalds/linux/blob/master/Documentation/filesystems/hfs.rst https://github.com/torvalds/linux/blob/master/Documentation/... [8] https://github.com/torvalds/linux/blob/master/Documentation/filesystems/hpfs.rst https://github.com/torvalds/linux/blob/master/Documentation/... [9] https://github.com/torvalds/linux/blob/master/Documentation/filesystems/sysv-fs.rst https://github.com/torvalds/linux/blob/master/Documentation/... [10] https://github.com/torvalds/linux/blob/master/Documentation/filesystems/bfs.rst https://github.com/torvalds/linux/blob/master/Documentation/...
Blammar 5y ago
Rich Shupak, huh? I used to work with his brother Paul, who had many stories about Rich. The only ones I remember, sadly, were: 1. Rich filed some large percentage of the bug reports on Excel (and possibly other Microsoft executables), along with the recommended code fixes and patches in assembly -- without ever having access to the source code. 2. He was fairly quickly hired by Microsoft, and is responsible for the design and implementation of Visual Basic.
simonh 5y ago
>The problem was that Microsoft intentionally hid and encrypted this information. Since the structure doesn't officially exist, there isn't going to be an official way to disable it. Anyone who develops with Microsoft tools gets this structure crammed in their executable whether or they like it or not. Failing to document this fact can be considered a questionable practice. Yes questionable, but if they had documented it or even left it unencrypted that would have made it trivial to spoof the contents, which would defeat the apparent purpose of the fingerprint. Most of the benefits of this thing would have been rendered useless. Benefits to Microsoft, of course.
colejohnson66 5y ago
Could someone explain how this “header” aids in debugging? There’s this: > To Microsoft's credit, the "Rich" header offers invaluable debugging statistics about how a given executable was built. But how does knowing the compiler version help with debugging? Are we assuming compiler bugs?
- jchw 5y ago
  “Debugging statistics” might mean something like, when analyzing WER data they could see if a new compiler build was having an elevated error rate, for example. That might not imply compiler bugs; it could also be the result of fixes for things that now crash instead of doing something unsafe.

123 points by Lammy 5y ago

jchw 5y ago

This seems like it might be useful for reverse engineering projects aiming to do source code reconstruction; if a Rich header is present, presumably that would guide you to being able to figure out if you have the exact builds of the tools that the original binary was made with. Interesting.

jes 5y ago

I was hoping to read this article on my iPhone. But the darn thing won’t narrow and stay narrowed so that it can be read without scrolling left and right to read each line. Is it specific to me or is this some behavior specific to this site?

Item_Boring 5y ago
I had the same issue but switching to reader view “fixed it“ :)
Stratoscope 5y ago
Same on Android, but Chrome pops up a "Show simplified view" on the bottom that fixes it right up (at the expense of wrapping the hex dumps).
hbn 5y ago
Click the aA button in Safari on the left of the URL bar -> "Show reader view" It won't show the highlighting on the hex dump, but you get the article text in a more readable format for the device
lelandfe 5y ago
It's because all the text is inside of a big table and there are several long rows. Combined, that means that the text can't really be resized below 1250px. Don't use tables for design, folks.
- martin_a 5y ago
  > Don't use tables for design, folks. Wow, feels like 15 years ago, when everybody moved from table layouts to divs.

ndiddy 5y ago

One use that the article doesn’t mention: If a large company ships a binary that was compiled with Visual Studio Community, Microsoft knows that they should do a license audit :)

Ericson2314 5y ago

It might just be nostalgia to the first computers I used, but the shear complexity, mystery, and stability that allows this stuff to accrete in Windows fills me with a sense of Wonder, even if it is also terrible. Unix is very less bloated, which is better, but also means there is basically no sense of Wonder. So Linux, my daily driver, I gripe about, but Windows, which I thankfully never have to use, makes my eyebrows raise and lips smile from a safe distance.

xeromal 5y ago
Linux is like using a constructed language. It's precise, constantly pared down, and optimized. It's effective and boring. Windows is like English. A mishmash of different languages with no perfect rules. It's a pain in the ass to get right. Lots of fun nuggets to find amusing though.
- skissane 5y ago
  > A mishmash of different languages with no perfect rules There is some of that in Unix/Linux as well, even if to a lesser degree than Windows exhibits it. One good example is the dd command–both the command name and its rather unique syntax are inspired by the IBM mainframe operating system MVS (nowadays rebranded as z/OS) and the omnipresent DD statement of its (in)famous JCL job control language, although the inspiration is only at a very high level and the actual details are quite different. Another is the fact that one of the fields in the /etc/passwd file is named pw_gecos, after the mainframe operating system General Electric Comprehensive Operating System (GECOS), which was used at Bell Labs in the early days of Unix. Some of those early Unix systems used the Bell Labs GECOS mainframe for print spooling, and pw_gecos was used to store the GECOS account name for each user. pw_gecos is still around, and has even spread from Unix into much newer systems such as LDAP and SCIM, although nowadays it stores the user's full name instead, and sometimes other contact details such as email or phone number as well. GECOS is still around too (if a shadow of its former self), having been passed from General Electric to Honeywell and since then to the French firm Groupe Bull (now part of Atos), and having been renamed from GECOS to GCOS along the way. The Unix terminal subsystem (line drivers, termios, terminfo/termcap, curses, etc) is full of all kinds of crazy legacy crud which made sense back when people used to use real physical terminals running over RS-232, but now in the 21st century it is basically just legacy complexity we are stuck with due to inertia, backward compatibility, and the inability of any redesign to offer enough value to overcome all that inertia. Current versions of Solaris [0] and its derivatives (such as Illumos) still contain the header file "rje.h", which is for an old Bell Labs system for submitting batch jobs to IBM mainframes. I doubt the actual code has been present for a long-time – the header references "/dev/dn2", which I believe is a PDP-11 modem device (see [1], [2]) – but I take it the Solaris developers never removed this file just in case doing so broke backward compatibility, maybe some developer somewhere included it in their C code and removing the file might stop their code from compiling. Linux, lacking the same heritage as Solaris, doesn't contain anything quite as ancient as this. The use of "VTOC" as a term for partition table by a number of Unix systems (including Solaris) is inherited from IBM mainframe systems, where VTOC is one of the most fundamental on-disk structures – although in normal use more like a root directory with embedded allocation information than an actual partition table, but the partition table usage is inspired by the fact that when Unix was ported to IBM mainframes, the mainframe VTOC filesystem was used as a partition table, with each Unix filesystem being stored as a separate file ("dataset" to use the proper mainframe terminology) in the mainframe filesystem. Linux can run on IBM mainframes (z/Linux), and on them it actually literally supports the IBM VTOC filesystem as a partition table format [3], along with a bunch of other weird mainframe features–use of 3270 terminals as a system console [4] and even integration with virtual card punches and card readers [5]. Filesystems in particular are an area in which the Linux kernel supports a huge amount of legacy cruft, with all manner of rarely used filesystems supported for backward compatibility, for example the ADFS filesystem of Acorn RISC OS [6], the classic Mac OS HFS filesystem [7], the HPFS filesystem of OS/2 [8], the "System V" filesystem used by old x86 Unix systems such as Xenix and Coherent [9], the "Boot File System" used by the boot process of SCO Unixware [10] [0] see for example https://github.com/illumos/illumos-gate/blob/master/usr/src/head/rje.h https://github.com/illumos/illumos-gate/blob/master/usr/src/... [1] http://squoze.net/UNIX/v5man/pdf/man4/dn.pdf http://squoze.net/UNIX/v5man/pdf/man4/dn.pdf [2] https://github.com/illumos/illumos-gate/blob/master/usr/src/head/rje.h https://github.com/illumos/illumos-gate/blob/master/usr/src/... [3] https://github.com/torvalds/linux/blob/master/arch/s390/include/uapi/asm/vtoc.h https://github.com/torvalds/linux/blob/master/arch/s390/incl... [4] https://github.com/torvalds/linux/blob/master/Documentation/s390/3270.rst https://github.com/torvalds/linux/blob/master/Documentation/... [5] https://github.com/torvalds/linux/blob/master/drivers/s390/char/vmur.c https://github.com/torvalds/linux/blob/master/drivers/s390/c... [6] https://github.com/torvalds/linux/blob/master/Documentation/filesystems/adfs.rst https://github.com/torvalds/linux/blob/master/Documentation/... [7] https://github.com/torvalds/linux/blob/master/Documentation/filesystems/hfs.rst https://github.com/torvalds/linux/blob/master/Documentation/... [8] https://github.com/torvalds/linux/blob/master/Documentation/filesystems/hpfs.rst https://github.com/torvalds/linux/blob/master/Documentation/... [9] https://github.com/torvalds/linux/blob/master/Documentation/filesystems/sysv-fs.rst https://github.com/torvalds/linux/blob/master/Documentation/... [10] https://github.com/torvalds/linux/blob/master/Documentation/filesystems/bfs.rst https://github.com/torvalds/linux/blob/master/Documentation/...

Blammar 5y ago

Rich Shupak, huh? I used to work with his brother Paul, who had many stories about Rich. The only ones I remember, sadly, were: 1. Rich filed some large percentage of the bug reports on Excel (and possibly other Microsoft executables), along with the recommended code fixes and patches in assembly -- without ever having access to the source code. 2. He was fairly quickly hired by Microsoft, and is responsible for the design and implementation of Visual Basic.

simonh 5y ago

>The problem was that Microsoft intentionally hid and encrypted this information. Since the structure doesn't officially exist, there isn't going to be an official way to disable it. Anyone who develops with Microsoft tools gets this structure crammed in their executable whether or they like it or not. Failing to document this fact can be considered a questionable practice. Yes questionable, but if they had documented it or even left it unencrypted that would have made it trivial to spoof the contents, which would defeat the apparent purpose of the fingerprint. Most of the benefits of this thing would have been rendered useless. Benefits to Microsoft, of course.

colejohnson66 5y ago

Could someone explain how this “header” aids in debugging? There’s this: > To Microsoft's credit, the "Rich" header offers invaluable debugging statistics about how a given executable was built. But how does knowing the compiler version help with debugging? Are we assuming compiler bugs?

jchw 5y ago
“Debugging statistics” might mean something like, when analyzing WER data they could see if a new compiler build was having an elevated error rate, for example. That might not imply compiler bugs; it could also be the result of fixes for things that now crash instead of doing something unsafe.