3 ms·

...the data flow diagram in the OP..."what happens when you execute a command"...is wrong. It implies that the command (ls -l) is read by the terminal emulator,

...the data flow diagram in the OP..."what happens when you execute a command"...is wrong. It implies that the command (ls -l) is read by the terminal emulator, which gives it to bash, which gives it to the kernel. It's accurate, the author just left off where the command is coming from (implying it's the user sitting at a keyboard). You type into the terminal-emulator, which passes that input to the program it's running, the shell, which figures out what it means. A terminal (or something emulating a terminal) is always between the keyboard and the input of the program running. Even when you login on the console, the terminal driver is accepting your input, transforming it in some fashion, and giving it to the nxt level. A terminal doesn't exist between this echo command and the sh program: echo ls -l | sh And is, in fact, why things like expect(1) exist and that ssh can be configured to allocate a pseudo-terminal or not depending on how you want/need to interact with the remote command. Incidentally, there seems to be a bug in yum whereby it detects if its output is a terminal or not tries to format it, but it gets the check reversed and pretty-print word-wraps output that goes to a non-terminal. Which is why yum search something and yum search something | sort produce different output and the latter has blank lines scattered throughout. (yum 3.2.28 on Fedora 14). And of course, the kernel is not really of importance here. The selected command does whatever it does, which might not even access the kernel ("Hello, world"). Every Hello, World program I've seen displays text to the user (or sends it somewhere), which means a system call, which is calling "kernel functions". The kernel is the mediator between programs/processes, there are very few cases where it isn't directly involved. And bash can and sometimes does directly call kernel functions without an intermediate binary (e.g., unlink(2) to overwrite a file for output redirection). Not really a good example, but the idea is sound. Bash has builtin functionality that emulate the functionality of system calls or other binaries. For this case, bash opens files for redirection when it's going to overwrite with the O_TRUNC flag set. This can be changed with various shell options (clobber) to have it not implicitly overwrite. You can see this with strace. And technically, "overwriting" a file means truncation or seek-and-write, not unlink and recreate. The difference has implications for hardlinked files.

mturmon 15y ago
OK, I see your point regarding the terminal emulator. There are some keys that are intercepted by the terminal emulator and processed by it alone (say, whatever does "print screen" or "close window"). The shell never sees these key presses. The thing I didn't (and still don't) like about the diagram is it implies the whole "ls -l" is delivered by the terminal emulator to the shell. Which isn't how it works. The emulator is transparent to all normal keystrokes and the shell is doing all the work, character by character, even for stuff like ctrl-H which, years ago, could have been (and sometimes was) intercepted and handled by the terminal driver. And the presence of the kernel in the diagram is also confused. (You are right, system calls are all over, even printf() will bottom out in the kernel; I was trying to say that there is no requirement for the user program to call in to the kernel.) Here's a sentence in the OP that has the same confusion: "When you execute a program from Python you can choose to execute it directly with the kernel or via a shell." The kernel is not "executing the program directly". This is nonsense. What they want to say is that you can use a system call (execve) to directly load and run the program, or you can use an interpreter like sh. This same point of confusion is exhibited in their diagram.
- thwarted 15y ago
  The thing I didn't (and still don't) like about the diagram is it implies the whole "ls -l" is delivered by the terminal emulator to the shell. Which isn't how it works. The emulator is transparent to all normal keystrokes and the shell is doing all the work, character by character, even for stuff like ctrl-H which, years ago, could have been (and sometimes was) intercepted and handled by the terminal driver. I don't think it's as simple as this. When the shell spawns a program, it gets out of the way and the program spawned receives all the input directly, the terminal intermediates, but the shell doesn't. For example, if you want to turn off echo (the printing of the characters you type), you need to change the terminal settings. This is done with stty -echo (go ahead and type that, you won't be able to see what you're typing), which communicates with the terminal driver to tell it to not print characters that are input. This is how password prompts are done. Witness: $ od -c hello<backspace><backspace><backspace><backspace><backspace><control-d> 0000000 od sees no input. The input was deleted by the terminal driver. Usually, the terminal is in line-input mode, so this can happen. Now, in my case, my backspace key maps to ^?, so to actually get a control-h from my backspace key, I had to do stty erase ^H. Then, the above produces: 0000000 h e l l o 177 177 177 177 177 because the backspace/erasure didn't occur at the terminal level (in fact, nothing got erased at all, nothing interpreted the input that way, and the input was passed to od plainly). Note the output of stty -a. There are a number of line editing control characters that modify the input at the terminal level before the program/shell ever even sees it. You can see this with strace -o /tmp/s.out -f bash then running od -c in that shell, typing the above, erasing it, and seeing that neither bash wakes up to handle the input and od sees no input (od's only read shows up as returning zero bytes, indicating end-of-file).

mturmon 15y ago

OK, I see your point regarding the terminal emulator. There are some keys that are intercepted by the terminal emulator and processed by it alone (say, whatever does "print screen" or "close window"). The shell never sees these key presses. The thing I didn't (and still don't) like about the diagram is it implies the whole "ls -l" is delivered by the terminal emulator to the shell. Which isn't how it works. The emulator is transparent to all normal keystrokes and the shell is doing all the work, character by character, even for stuff like ctrl-H which, years ago, could have been (and sometimes was) intercepted and handled by the terminal driver. And the presence of the kernel in the diagram is also confused. (You are right, system calls are all over, even printf() will bottom out in the kernel; I was trying to say that there is no requirement for the user program to call in to the kernel.) Here's a sentence in the OP that has the same confusion: "When you execute a program from Python you can choose to execute it directly with the kernel or via a shell." The kernel is not "executing the program directly". This is nonsense. What they want to say is that you can use a system call (execve) to directly load and run the program, or you can use an interpreter like sh. This same point of confusion is exhibited in their diagram.

thwarted 15y ago
The thing I didn't (and still don't) like about the diagram is it implies the whole "ls -l" is delivered by the terminal emulator to the shell. Which isn't how it works. The emulator is transparent to all normal keystrokes and the shell is doing all the work, character by character, even for stuff like ctrl-H which, years ago, could have been (and sometimes was) intercepted and handled by the terminal driver. I don't think it's as simple as this. When the shell spawns a program, it gets out of the way and the program spawned receives all the input directly, the terminal intermediates, but the shell doesn't. For example, if you want to turn off echo (the printing of the characters you type), you need to change the terminal settings. This is done with stty -echo (go ahead and type that, you won't be able to see what you're typing), which communicates with the terminal driver to tell it to not print characters that are input. This is how password prompts are done. Witness: $ od -c hello<backspace><backspace><backspace><backspace><backspace><control-d> 0000000 od sees no input. The input was deleted by the terminal driver. Usually, the terminal is in line-input mode, so this can happen. Now, in my case, my backspace key maps to ^?, so to actually get a control-h from my backspace key, I had to do stty erase ^H. Then, the above produces: 0000000 h e l l o 177 177 177 177 177 because the backspace/erasure didn't occur at the terminal level (in fact, nothing got erased at all, nothing interpreted the input that way, and the input was passed to od plainly). Note the output of stty -a. There are a number of line editing control characters that modify the input at the terminal level before the program/shell ever even sees it. You can see this with strace -o /tmp/s.out -f bash then running od -c in that shell, typing the above, erasing it, and seeing that neither bash wakes up to handle the input and od sees no input (od's only read shows up as returning zero bytes, indicating end-of-file).