7 ms·

An Explainer on Unix’s Most Notorious Code Comment

173 points by inopinatus 10y ago

oftenwrong 10y ago
https://en.wikipedia.org/wiki/Lions'_Commentary_on_UNIX_6th_Edition,_with_Source_Code#.22You_are_not_expected_to_understand_this.22 https://en.wikipedia.org/wiki/Lions'_Commentary_on_UNIX_6th_...
- pimlottc 10y ago
  Thanks for this, I was surprised the article didn't include the actual comment itself!
statictype 10y ago
I thought this was going to be about "Unwarranted chumminess with the compiler" - in the original C implementation of regular expressions and subsequently copied into every language port thereafter.
- tomjakubowski 10y ago
  For those who don't know, that comment accompanied the use of an "unsized" char[] at the end of a heap-allocated struct. See http://c-faq.com/struct/structhack.html http://c-faq.com/struct/structhack.html
TheRealPomax 10y ago
Is the actual talk available somewhere? Much as I like reading a blog post, I'd much rather first watch the actual presentation by Arun Thomas, which was _clearly_ worth watching by far more than just the audience of Systems We Love.
- joshbaptiste 10y ago
  http://systemswe.love/videos/you-are-not-expected-to-understand-this http://systemswe.love/videos/you-are-not-expected-to-underst...
- inopinatus 10y ago
  It's an embedded video in the article; but here's the direct links. This talk specifically is https://www.youtube.com/watch?v=TPe6UXMDMGM&start=26703 https://www.youtube.com/watch?v=TPe6UXMDMGM&start=26703 or http://systemswe.love/videos/you-are-not-expected-to-understand-this http://systemswe.love/videos/you-are-not-expected-to-underst... For further edification, the entire 9h Systems We Love live stream is up on youtube at https://www.youtube.com/watch?v=TPe6UXMDMGM https://www.youtube.com/watch?v=TPe6UXMDMGM or split into videos and via Vimeo at http://systemswe.love/videos http://systemswe.love/videos
  - TheRealPomax 10y ago
    ah, probably noScript and uBlock completely hiding that embed... thanks!
deleted 10y ago
[deleted]
gwu78 10y ago
The best part of this was the link to a story on the bottom of the page about a guy who spent 10 hours scrolling to the bottom of an Excel spreadsheet. It will probably be gone by now. Source was motherboard.vice.com The intellectual contrast between the UNIX way of thinking and the Windows way of "thinking" continues to amaze.
- mos_basik 10y ago
  It was still there when I got to the footer, too. So, for posterity: http://motherboard.vice.com/read/the-excel-challenge http://motherboard.vice.com/read/the-excel-challenge
- tomjakubowski 10y ago
  > The intellectual contrast between the UNIX way of thinking and the Windows way of "thinking" continues to amaze. Um, what? I mean, you're contrasting a dry comment left by one the programmers responsible for creating an OS with the silly antics of the user of another OS; of course there's a difference, it's apples and oranges. I'm certain people have done stupid shit like this on UNIX too. (like, say "time yourself walking to the edge of a flat Minecraft world from the center")
- throwanem 10y ago
  Surely no more so than the idea that one's choice of OS says something meaningful about one's qualities as a person. I hear there are still astrologers, too.
  - yjftsjthsd-h 10y ago
    Actually from a Bayesian perspective the average GNU/Linux user probably is more technically competent than the average Windows/NT user, if only because the former usually requires manually installing the OS.
    - throwanem 10y ago
      Sure, installing Linux a bunch of times probably makes you good at installing Linux. Does that seem to you like a skill with much in the way of wider application? It does not seem so to me, and I've installed Linux a whole bunch of times.
CalChris 10y ago
If I recall correctly, Lion's Commentary covered this pretty well. Just search for swtch but roughly sections 6.9, 8.9, 8.15 and 15.9. Lions refers to understanding it as joining the 2238 club. http://www.c-c-g.de/attachments/article/211/lions_book_commentary.pdf http://www.c-c-g.de/attachments/article/211/lions_book_comme...
justifier 10y ago
> Unix v6 was released in 1975. “It was roughly 9,000 lines of code,” has anyone perused it? 9k is tiny.. is there any value, either direct or extrapolated, in lending it the mindshare?
- deleted 10y ago
  [deleted]
- catmanjan 10y ago
  I guess that would count as virtual antiquing?
- abecedarius 10y ago
  That was just the kernel. There's a classic book by Lions adding commentary to the listings.
- pm215 10y ago
  As abecedarius says, there's a good commentary on it in the Lions book (https://en.wikipedia.org/wiki/Lions'_Commentary_on_UNIX_6th_Edition,_with_Source_Code https://en.wikipedia.org/wiki/Lions'_Commentary_on_UNIX_6th_...). That was reprinted in the late 90s when I was at university and I read through it then. I think it is worthwhile still, with caveats. The C dialect is ancient (pre-K&R), some of the algorithms are rather naive, and the whole thing is "oriented towards a machine that is little more than a memory", to quote the reprint's introduction. That said, the commentary is excellent (giving enough hints to enlighten but often requiring you to work through the detail yourself as a teaching mechanism), and the whole thing is real production code that gives you a feel for how a complete Unix-ish kernel is structured (which still applies to modern kernels, much as a housecat and an elephant have clear skeletal similarities despite massive differences in scale and features). xv6 (https://en.wikipedia.org/wiki/Xv6 https://en.wikipedia.org/wiki/Xv6) is a modern (x86/ANSI C) but similarly sized kernel designed for teaching purposes and to avoid the awkward obsolete-architecture-and-compiler aspects of the original 6th Ed code. If you can live with the PDP-11 assembly and ancient C (and not being able to run the code) I'd recommend the original, though.
  - 10y ago
userbinator 10y ago
It's worth noting that I find "unconventional" control flow like this tends to only puzzle those who started programming at the "structured" level where function calls always nest neatly, and so they have never entertained the possibility of it being any different from that. However, the machine itself doesn't have any real notion of "call" and "return", and although dedicated instructions may be provided for convenience, all a call amounts to is "remember where in the code you were, and go somewhere else" and return is "use the saved information to go back". It then follows that, if this saved information be not precisely that of the immediately preceding call, the processor can be caused to "return" somewhere else; and that is the basis of coroutines and process switching. Call and return are simply a convenient abstraction atop the sequential instruction stream the processor executes.
- crooked-v 10y ago
  This kind of thing is why every programmer should at least once try to build something fancy using only variables and GOTO statements. The classic make-your-own-game engines ZZT and MegaZeux had a lot of this; the scripts for in-game objects had no function calls or well-managed events, just GOTOs and per-object vars, and could get interrupted at any time by GOTOs triggered by other in-game objects.
  - pmiller2 10y ago
    > This kind of thing is why every programmer should at least once try to build something fancy using only variables and GOTO statements. So, basically, assembler without a lot of fancy macros? :)
    - throwanem 10y ago
      Or a line-numbered BASIC without GOSUB - sit them down in front of a IIe emulator and a pile of PDF manuals, and tell them to go to it.
      - crooked-v 10y ago
aap_ 10y ago
I don't see an explanation of this comment at all. I don't think it's the context switching in general that we're not expected to understand, since it's pretty much the same in v7 but the comment is gone. Dmr actually explained the problem on his website (https://www.bell-labs.com/usr/dmr/www/odd.html https://www.bell-labs.com/usr/dmr/www/odd.html). savu is used to save the current call stack, retu is used to switch to a saved call stack. The problem is that the function which did the savu was not necessarily the same as the function that does the retu, so after retu the function could have the call stack of a different function. As dmr explained, this worked with the PDP-11 compiler but not with the interdata compiler. In V7 the stack switching was moved into separate functions, save and resume. save retured 0 and resume returned 1 so that an if statement could be used to check if the return from save was actually that of resume after the stack switch (the same trick as that of fork). This way the code that was to be executed after a stack switch was in the same function and stack frame as the one that did the save (as opposed to swtch). (EDIT: I was wrong about v7 save/resume at first) Note that Lions doesn't explain this either, he assumed that the difficulty was with with u_rsav and u_ssav, but those are still in v7 with the comment gone (he probably wasn't that wrong though, it really is confusing, but it's just not what the comment refers to)
zwp 10y ago
Link to the comment in V6 at Diomidis Spinellis' unix-history-repo: https://github.com/dspinellis/unix-history-repo/blob/Research-V6-Snapshot-Development/usr/sys/ken/slp.c#L325 https://github.com/dspinellis/unix-history-repo/blob/Researc...
Taniwha 10y ago
I ported V6 back in the day, I continually would come back to this comment from different code path, always mystified, one day I came to it and it was obvious .... Essentially this was the days before save/restore were in the C library - savu/aretu are an equivalent that works differently (you savu() somewhere and aretu() somewhere else, it was sad because aretu() played stack games so unlike save() savu() didn't just return a different value on return) Essentially v6 was a swapping system, fork() (or newproc() in the kernel) worked by allocating memory and making a copy of an app in memory, if there wasn't enough memory it would pretend to swap out the existing process, then mark it as the new process and set a flag (SSWAP), this is the case when it's returning - a process is being swapped in, part way through it realises it's really this special case and this code returns into newproc() as the new process
revx 10y ago
Related, a talk on the famous mv error, "Values of β may give rise to dom!" from !!Con 2016: https://www.youtube.com/watch?v=FyLnF9U18rY https://www.youtube.com/watch?v=FyLnF9U18rY
- userbinator 10y ago
  A very close match for this phrase can be found in https://books.google.com/books?id=n3vVBwAAQBAJ&pg=PA429&lpg=PA429&dq=%22values+of%22+%22give+rise+to+domain%22&source=bl&ots=vToXJ1U9RE&sig=4Jyi9e2OgBt0TN61d88EDWGhnas&hl=en&sa=X&redir_esc=y https://books.google.com/books?id=n3vVBwAAQBAJ&pg=PA429&lpg=... which gives credence to the original story that it was part of an interrupted prinout.

173 points by inopinatus 10y ago

https://en.wikipedia.org/wiki/Lions'_Commentary_on_UNIX_6th_Edition,_with_Source_Code#.22You_are_not_expected_to_understand_this.22 https://en.wikipedia.org/wiki/Lions'_Commentary_on_UNIX_6th_...

pimlottc 10y ago
Thanks for this, I was surprised the article didn't include the actual comment itself!

statictype 10y ago

I thought this was going to be about "Unwarranted chumminess with the compiler" - in the original C implementation of regular expressions and subsequently copied into every language port thereafter.

tomjakubowski 10y ago
For those who don't know, that comment accompanied the use of an "unsized" char[] at the end of a heap-allocated struct. See http://c-faq.com/struct/structhack.html http://c-faq.com/struct/structhack.html

TheRealPomax 10y ago

Is the actual talk available somewhere? Much as I like reading a blog post, I'd much rather first watch the actual presentation by Arun Thomas, which was _clearly_ worth watching by far more than just the audience of Systems We Love.

joshbaptiste 10y ago
http://systemswe.love/videos/you-are-not-expected-to-understand-this http://systemswe.love/videos/you-are-not-expected-to-underst...
inopinatus 10y ago
It's an embedded video in the article; but here's the direct links. This talk specifically is https://www.youtube.com/watch?v=TPe6UXMDMGM&start=26703 https://www.youtube.com/watch?v=TPe6UXMDMGM&start=26703 or http://systemswe.love/videos/you-are-not-expected-to-understand-this http://systemswe.love/videos/you-are-not-expected-to-underst... For further edification, the entire 9h Systems We Love live stream is up on youtube at https://www.youtube.com/watch?v=TPe6UXMDMGM https://www.youtube.com/watch?v=TPe6UXMDMGM or split into videos and via Vimeo at http://systemswe.love/videos http://systemswe.love/videos
- TheRealPomax 10y ago
  ah, probably noScript and uBlock completely hiding that embed... thanks!

deleted 10y ago

[deleted]

gwu78 10y ago

The best part of this was the link to a story on the bottom of the page about a guy who spent 10 hours scrolling to the bottom of an Excel spreadsheet. It will probably be gone by now. Source was motherboard.vice.com The intellectual contrast between the UNIX way of thinking and the Windows way of "thinking" continues to amaze.

mos_basik 10y ago
It was still there when I got to the footer, too. So, for posterity: http://motherboard.vice.com/read/the-excel-challenge http://motherboard.vice.com/read/the-excel-challenge
tomjakubowski 10y ago
> The intellectual contrast between the UNIX way of thinking and the Windows way of "thinking" continues to amaze. Um, what? I mean, you're contrasting a dry comment left by one the programmers responsible for creating an OS with the silly antics of the user of another OS; of course there's a difference, it's apples and oranges. I'm certain people have done stupid shit like this on UNIX too. (like, say "time yourself walking to the edge of a flat Minecraft world from the center")
throwanem 10y ago
Surely no more so than the idea that one's choice of OS says something meaningful about one's qualities as a person. I hear there are still astrologers, too.
- yjftsjthsd-h 10y ago
  Actually from a Bayesian perspective the average GNU/Linux user probably is more technically competent than the average Windows/NT user, if only because the former usually requires manually installing the OS.
  - throwanem 10y ago
    Sure, installing Linux a bunch of times probably makes you good at installing Linux. Does that seem to you like a skill with much in the way of wider application? It does not seem so to me, and I've installed Linux a whole bunch of times.

CalChris 10y ago

If I recall correctly, Lion's Commentary covered this pretty well. Just search for swtch but roughly sections 6.9, 8.9, 8.15 and 15.9. Lions refers to understanding it as joining the 2238 club. http://www.c-c-g.de/attachments/article/211/lions_book_commentary.pdf http://www.c-c-g.de/attachments/article/211/lions_book_comme...

justifier 10y ago

> Unix v6 was released in 1975. “It was roughly 9,000 lines of code,” has anyone perused it? 9k is tiny.. is there any value, either direct or extrapolated, in lending it the mindshare?

deleted 10y ago
[deleted]
catmanjan 10y ago
I guess that would count as virtual antiquing?
abecedarius 10y ago
That was just the kernel. There's a classic book by Lions adding commentary to the listings.
pm215 10y ago
As abecedarius says, there's a good commentary on it in the Lions book (https://en.wikipedia.org/wiki/Lions'_Commentary_on_UNIX_6th_Edition,_with_Source_Code https://en.wikipedia.org/wiki/Lions'_Commentary_on_UNIX_6th_...). That was reprinted in the late 90s when I was at university and I read through it then. I think it is worthwhile still, with caveats. The C dialect is ancient (pre-K&R), some of the algorithms are rather naive, and the whole thing is "oriented towards a machine that is little more than a memory", to quote the reprint's introduction. That said, the commentary is excellent (giving enough hints to enlighten but often requiring you to work through the detail yourself as a teaching mechanism), and the whole thing is real production code that gives you a feel for how a complete Unix-ish kernel is structured (which still applies to modern kernels, much as a housecat and an elephant have clear skeletal similarities despite massive differences in scale and features). xv6 (https://en.wikipedia.org/wiki/Xv6 https://en.wikipedia.org/wiki/Xv6) is a modern (x86/ANSI C) but similarly sized kernel designed for teaching purposes and to avoid the awkward obsolete-architecture-and-compiler aspects of the original 6th Ed code. If you can live with the PDP-11 assembly and ancient C (and not being able to run the code) I'd recommend the original, though.
- 10y ago

userbinator 10y ago

It's worth noting that I find "unconventional" control flow like this tends to only puzzle those who started programming at the "structured" level where function calls always nest neatly, and so they have never entertained the possibility of it being any different from that. However, the machine itself doesn't have any real notion of "call" and "return", and although dedicated instructions may be provided for convenience, all a call amounts to is "remember where in the code you were, and go somewhere else" and return is "use the saved information to go back". It then follows that, if this saved information be not precisely that of the immediately preceding call, the processor can be caused to "return" somewhere else; and that is the basis of coroutines and process switching. Call and return are simply a convenient abstraction atop the sequential instruction stream the processor executes.

crooked-v 10y ago
This kind of thing is why every programmer should at least once try to build something fancy using only variables and GOTO statements. The classic make-your-own-game engines ZZT and MegaZeux had a lot of this; the scripts for in-game objects had no function calls or well-managed events, just GOTOs and per-object vars, and could get interrupted at any time by GOTOs triggered by other in-game objects.
- pmiller2 10y ago
  > This kind of thing is why every programmer should at least once try to build something fancy using only variables and GOTO statements. So, basically, assembler without a lot of fancy macros? :)
  - throwanem 10y ago
    Or a line-numbered BASIC without GOSUB - sit them down in front of a IIe emulator and a pile of PDF manuals, and tell them to go to it.
    - crooked-v 10y ago

aap_ 10y ago

I don't see an explanation of this comment at all. I don't think it's the context switching in general that we're not expected to understand, since it's pretty much the same in v7 but the comment is gone. Dmr actually explained the problem on his website (https://www.bell-labs.com/usr/dmr/www/odd.html https://www.bell-labs.com/usr/dmr/www/odd.html). savu is used to save the current call stack, retu is used to switch to a saved call stack. The problem is that the function which did the savu was not necessarily the same as the function that does the retu, so after retu the function could have the call stack of a different function. As dmr explained, this worked with the PDP-11 compiler but not with the interdata compiler. In V7 the stack switching was moved into separate functions, save and resume. save retured 0 and resume returned 1 so that an if statement could be used to check if the return from save was actually that of resume after the stack switch (the same trick as that of fork). This way the code that was to be executed after a stack switch was in the same function and stack frame as the one that did the save (as opposed to swtch). (EDIT: I was wrong about v7 save/resume at first) Note that Lions doesn't explain this either, he assumed that the difficulty was with with u_rsav and u_ssav, but those are still in v7 with the comment gone (he probably wasn't that wrong though, it really is confusing, but it's just not what the comment refers to)

zwp 10y ago

Link to the comment in V6 at Diomidis Spinellis' unix-history-repo: https://github.com/dspinellis/unix-history-repo/blob/Research-V6-Snapshot-Development/usr/sys/ken/slp.c#L325 https://github.com/dspinellis/unix-history-repo/blob/Researc...

Taniwha 10y ago

I ported V6 back in the day, I continually would come back to this comment from different code path, always mystified, one day I came to it and it was obvious .... Essentially this was the days before save/restore were in the C library - savu/aretu are an equivalent that works differently (you savu() somewhere and aretu() somewhere else, it was sad because aretu() played stack games so unlike save() savu() didn't just return a different value on return) Essentially v6 was a swapping system, fork() (or newproc() in the kernel) worked by allocating memory and making a copy of an app in memory, if there wasn't enough memory it would pretend to swap out the existing process, then mark it as the new process and set a flag (SSWAP), this is the case when it's returning - a process is being swapped in, part way through it realises it's really this special case and this code returns into newproc() as the new process

revx 10y ago

Related, a talk on the famous mv error, "Values of β may give rise to dom!" from !!Con 2016: https://www.youtube.com/watch?v=FyLnF9U18rY https://www.youtube.com/watch?v=FyLnF9U18rY

userbinator 10y ago
A very close match for this phrase can be found in https://books.google.com/books?id=n3vVBwAAQBAJ&pg=PA429&lpg=PA429&dq=%22values+of%22+%22give+rise+to+domain%22&source=bl&ots=vToXJ1U9RE&sig=4Jyi9e2OgBt0TN61d88EDWGhnas&hl=en&sa=X&redir_esc=y https://books.google.com/books?id=n3vVBwAAQBAJ&pg=PA429&lpg=... which gives credence to the original story that it was part of an interrupted prinout.