4 ms·

Reminds me of the fun fact that on Linux env vars are not really key-value, but just a list of strings; it's just a convention by libc to parse them as key-valu

by zokier 4y ago

Reminds me of the fun fact that on Linux env vars are not really key-value, but just a list of strings; it's just a convention by libc to parse them as key-value by splitting on =. If you don't care about compatibility you could hypothetically use any format you want for env

Hello71 4y ago
afaik the linux kernel doesn't read environment variables, but it does write some: processes spawned by the kernel, such as init, modprobe, and request_key, start with some pre-defined environment variables, like HOME=/ TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin.
maxbond 4y ago
Well, it's a key-value data structure, serialized in the way you describe. You could say that a JSON object wasn't a key-value structure by the same logic. It's the same way for structures serialized in binary rather than text, like BSON objects. You can think of a native dictionary as a declarative structure where each of the pairs "happen" simultaneously. A serialized structure is necessarily imperative because it must be an ordered series of bytes that are evaluated in sequence; it must be in order to be written to disk or sent across the network (with /proc being a sort of hybrid, IPC implemented as a virtual filesystem). So we generally need to convert our dictionaries into lists to serialize them. You might think of something like sending an entire SQLite database as an exception to this rule, but there's an equivalent perspective that it is an example of the rule instead. Lists of key-value pairs and dictionaries of key-value pairs are duals of each other so you're free to take the interpretation you prefer.
- deleted 4y ago
  [deleted]
nikeee 4y ago
This seems similar to the way command like arguments work on windows. On Linux, the system call für spawning a process uses a string array for them. On windows, it's a single string. The string needs to be parsed by the application, making escape sequences in command line arguments implementation defined by the compiler/runtime/etc. https://nullprogram.com/blog/2022/02/18/ https://nullprogram.com/blog/2022/02/18/ Some examples are here: https://daviddeley.com/autohotkey/parameters/parameters.htm#WINNOSTANDARD https://daviddeley.com/autohotkey/parameters/parameters.htm#...
- chasil 4y ago
  In theory, a POSIX layer in Windows will implement the exec() family of system calls. The Windows (NT) kernel was built to conform to POSIX, although this functionality was removed, then reimplemented and expanded. 'Broad software compatibility was initially achieved with support for several API "personalities", including Windows API, POSIX, and OS/2 APIs – the latter two were phased out starting with Windows XP.' https://en.m.wikipedia.org/wiki/Windows_NT#Major_features https://en.m.wikipedia.org/wiki/Windows_NT#Major_features
  - yellowapple 4y ago
    IIRC, WSL1 was another such "personality". Disappointing that WSL2 dropped that in favor of what's basically a glorified VM; maybe that's just easier to do than maintaining a (presumably clean-room, given the need to comply with the GPLv2 if any actual Linux code was ever used) reimplementation of the Linux userspace ABI?
- wahern 4y ago
  > This seems similar to the way command like arguments work on windows. The previous poster overstates things. Environment variables passed to the process by the kernel[1] are NUL-delimited, including a double-NUL (or zero-length pair) to terminate the list of name=value pairs. Neither is there any syntactical ambiguity, as within each NUL-terminated pair the first '=' character delimits the variable name from the value, regardless of whether there are subsequent '=' characters. So there's no real parsing of the name=value content, only locating each NUL character in turn as a C array is built up by the entry code. The only limitation is that neither names nor values can contain the NUL character, and names cannot contain a '=' character. The process startup code uses the exact same logic for constructing argv as it does for constructing environ/envp as they're both passed by the kernel in the same manner, including the same NUL-delimited structure. In fact, they're written as one long, serial sequence to the stack, above or beneath (depending on stack direction) the initial stack frame setup for the entry point. And usually argv and environ/envp arrays simply hold pointers directly to this memory region, at least initially. On Linux /proc/$$/environ reads out this region, so it won't necessarily reflect a process' actual environment variables once a process starts changing them. [1] For both Linux and, at least historically, the BSDs.
chenxiaolong 4y ago
Some programs, like sshd and chromium, make use of this fact to change the process name as shown in procfs and ps. On Linux, every string that argv[] and envp[] point to are stored in one contiguous block of memory. Programs can change the name of the process by writing a new string starting at *argv[0] and overflowing into the *envp[] memory as needed. (Though the program needs to strdup() the environ[] entries beforehand since they initially point to envp[] strings.) For example, chromium's implementation: https://source.chromium.org/chromium/chromium/src/+/refs/tags/110.0.5481.44:content/common/set_process_title_linux.cc https://source.chromium.org/chromium/chromium/src/+/refs/tag...
- 10000truths 4y ago
  Interesting that procfs reads directly from userspace memory to answer queries from /proc/{pid}/cmdline. I wonder what happens if someone unmaps the memory containing those argv/envp strings?
  - chenxiaolong 4y ago
    I just gave it a try in gdb on a C program with an empty main(). The argv/envp data was in this range: 7ffffffdd000-7ffffffff000 rw-p 00000000 00:00 0 [stack] Running `call (int) munmap(0x7ffffffdd000, 0x22000)` in gdb did indeed unmap the stack... and the program segfaulted immediately afterwards. But while the process was still in the stopped state and before it finished crashing/exiting, /proc/<pid>/cmdline was a 0-byte file.
chasil 4y ago
The author also has not realized that xargs -0 can transform null-delimited fields into space-delimited fields. I believe the invocation could be: export $(xargs -0 echo < /proc/1/environ) It could be dangerous to use echo in this way in some circumstances.
- theamk 4y ago
  eh, that sounds like a bad idea, as vars can contain spaces as well. You can solve this with quoting, but it is tricky, it is much nicer (and safer in case of errors!) to ensure you don't need to do that.
  - chasil 4y ago
    Somehow, "$@" would have to be involved for safe processing. I will ponder it.
- teddyh 4y ago
  “echo” is the default action for xargs, so you don’t need to specify it.

by zokier 4y ago

Hello71 4y ago

afaik the linux kernel doesn't read environment variables, but it does write some: processes spawned by the kernel, such as init, modprobe, and request_key, start with some pre-defined environment variables, like HOME=/ TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin.

maxbond 4y ago

Well, it's a key-value data structure, serialized in the way you describe. You could say that a JSON object wasn't a key-value structure by the same logic. It's the same way for structures serialized in binary rather than text, like BSON objects. You can think of a native dictionary as a declarative structure where each of the pairs "happen" simultaneously. A serialized structure is necessarily imperative because it must be an ordered series of bytes that are evaluated in sequence; it must be in order to be written to disk or sent across the network (with /proc being a sort of hybrid, IPC implemented as a virtual filesystem). So we generally need to convert our dictionaries into lists to serialize them. You might think of something like sending an entire SQLite database as an exception to this rule, but there's an equivalent perspective that it is an example of the rule instead. Lists of key-value pairs and dictionaries of key-value pairs are duals of each other so you're free to take the interpretation you prefer.

deleted 4y ago
[deleted]

nikeee 4y ago

This seems similar to the way command like arguments work on windows. On Linux, the system call für spawning a process uses a string array for them. On windows, it's a single string. The string needs to be parsed by the application, making escape sequences in command line arguments implementation defined by the compiler/runtime/etc. https://nullprogram.com/blog/2022/02/18/ https://nullprogram.com/blog/2022/02/18/ Some examples are here: https://daviddeley.com/autohotkey/parameters/parameters.htm#WINNOSTANDARD https://daviddeley.com/autohotkey/parameters/parameters.htm#...

chasil 4y ago
In theory, a POSIX layer in Windows will implement the exec() family of system calls. The Windows (NT) kernel was built to conform to POSIX, although this functionality was removed, then reimplemented and expanded. 'Broad software compatibility was initially achieved with support for several API "personalities", including Windows API, POSIX, and OS/2 APIs – the latter two were phased out starting with Windows XP.' https://en.m.wikipedia.org/wiki/Windows_NT#Major_features https://en.m.wikipedia.org/wiki/Windows_NT#Major_features
- yellowapple 4y ago
  IIRC, WSL1 was another such "personality". Disappointing that WSL2 dropped that in favor of what's basically a glorified VM; maybe that's just easier to do than maintaining a (presumably clean-room, given the need to comply with the GPLv2 if any actual Linux code was ever used) reimplementation of the Linux userspace ABI?
wahern 4y ago
> This seems similar to the way command like arguments work on windows. The previous poster overstates things. Environment variables passed to the process by the kernel[1] are NUL-delimited, including a double-NUL (or zero-length pair) to terminate the list of name=value pairs. Neither is there any syntactical ambiguity, as within each NUL-terminated pair the first '=' character delimits the variable name from the value, regardless of whether there are subsequent '=' characters. So there's no real parsing of the name=value content, only locating each NUL character in turn as a C array is built up by the entry code. The only limitation is that neither names nor values can contain the NUL character, and names cannot contain a '=' character. The process startup code uses the exact same logic for constructing argv as it does for constructing environ/envp as they're both passed by the kernel in the same manner, including the same NUL-delimited structure. In fact, they're written as one long, serial sequence to the stack, above or beneath (depending on stack direction) the initial stack frame setup for the entry point. And usually argv and environ/envp arrays simply hold pointers directly to this memory region, at least initially. On Linux /proc/$$/environ reads out this region, so it won't necessarily reflect a process' actual environment variables once a process starts changing them. [1] For both Linux and, at least historically, the BSDs.

chenxiaolong 4y ago

Some programs, like sshd and chromium, make use of this fact to change the process name as shown in procfs and ps. On Linux, every string that argv[] and envp[] point to are stored in one contiguous block of memory. Programs can change the name of the process by writing a new string starting at *argv[0] and overflowing into the *envp[] memory as needed. (Though the program needs to strdup() the environ[] entries beforehand since they initially point to envp[] strings.) For example, chromium's implementation: https://source.chromium.org/chromium/chromium/src/+/refs/tags/110.0.5481.44:content/common/set_process_title_linux.cc https://source.chromium.org/chromium/chromium/src/+/refs/tag...

10000truths 4y ago
Interesting that procfs reads directly from userspace memory to answer queries from /proc/{pid}/cmdline. I wonder what happens if someone unmaps the memory containing those argv/envp strings?
- chenxiaolong 4y ago
  I just gave it a try in gdb on a C program with an empty main(). The argv/envp data was in this range: 7ffffffdd000-7ffffffff000 rw-p 00000000 00:00 0 [stack] Running `call (int) munmap(0x7ffffffdd000, 0x22000)` in gdb did indeed unmap the stack... and the program segfaulted immediately afterwards. But while the process was still in the stopped state and before it finished crashing/exiting, /proc/<pid>/cmdline was a 0-byte file.

chasil 4y ago

The author also has not realized that xargs -0 can transform null-delimited fields into space-delimited fields. I believe the invocation could be: export $(xargs -0 echo < /proc/1/environ) It could be dangerous to use echo in this way in some circumstances.

theamk 4y ago
eh, that sounds like a bad idea, as vars can contain spaces as well. You can solve this with quoting, but it is tricky, it is much nicer (and safer in case of errors!) to ensure you don't need to do that.
- chasil 4y ago
  Somehow, "$@" would have to be involved for safe processing. I will ponder it.
teddyh 4y ago
“echo” is the default action for xargs, so you don’t need to specify it.