3 ms·

Have you ever exported anything in a shell script? Sure you can keep the necessary changes in local state and pass those to execve(2)/execvpe(3)/posix_spawn(3),

by oefrha 3y ago

Have you ever exported anything in a shell script? Sure you can keep the necessary changes in local state and pass those to execve(2)/execvpe(3)/posix_spawn(3), and that would be safe AFAIK, but setenv(3) is there and more convenient if you're unaware of the hidden dangers. Also that doesn't work for PATH in execvp/execvpe, which is read from the current process; how do you change search paths for execvp without setenv (short of doing the search yourself)? Edit: I just realized macOS/FreeBSD has execvP() that allows passing a custom search path, so PATH is now safe, but without a -e variant, everything else is again unsafe.

quickthrower2 3y ago
Shell scripts are different as you are likely exporting environment variables and then starting new processes.
- oefrha 3y ago
  Shell scripts aren’t different from “real” programs using exec or posix_spawn in this regard, it’s just that fewer people have done the latter than the former, so the former is a more relatable example. “Real” programs spawn other processes too you know, sometimes with modified environ.
  - quickthrower2 3y ago
    So I understand this right, I thought the issue is about multiple threads but in shell you wouldn’t have this just new processes. In a program you could have either.
rcxdude 3y ago
Shell scripts are not really prone to this problem because AFAIK no shells are multithreaded: subshells and the like are implemented with fork()
- oefrha 3y ago
  Yes, I’m not saying shell scripts are affected, merely using them as an example to answer the question “Why do any of these [env vars] need to change at runtime?”
  - xxs 3y ago
    The discussion is only relevant for a shared unguarded resource (the env) modified and read by multiple threads. Single threaded operations are just fine.
    - Someone 3y ago
      > Single threaded operations are just fine. Sort of. https://pubs.opengroup.org/onlinepubs/9699919799/functions/getenv.html https://pubs.opengroup.org/onlinepubs/9699919799/functions/g...: “The returned string pointer might be invalidated or the string content might be overwritten by a subsequent call to getenv()” There’s little you can do with a broken API, so Linux has that ‘feature’, too. https://man7.org/linux/man-pages/man3/getenv.3.html https://man7.org/linux/man-pages/man3/getenv.3.html: “The string pointed to by the return value of getenv() may be statically allocated, and can be modified by a subsequent call to getenv(), putenv(3), setenv(3), or unsetenv(3).” FreeBSD chooses to leak memory, instead. https://man.freebsd.org/cgi/man.cgi?getenv(3) https://man.freebsd.org/cgi/man.cgi?getenv(3): “Successive calls to setenv() that assign a larger-sized value than any previous value to the same name will result in a memory leak. The FreeBSD semantics for this function (namely, that the contents of value are copied and that old values remain accessible indefinitely) make this bug unavoidable”
xxs 3y ago
>Have you ever exported anything in a shell script So, shells use a single thread that can safely modify the environment - then start new child processes by the same thread. The child processes get a =copy= of the said environment. That's a textbook example how to use env. Starting multiple threads on your own, then modifying env should be considered a textbook example how not to do things - env is not intended for interprocess communication.
anttihaapala 3y ago
In the case of execvp, you would pretty much be required to fork before it and then you could change PATH.
- oefrha 3y ago
  Yeah, fork()+immediately exec() should be safe, but those use cases are almost always better with posix_spawn(), due to issues with fork(), like memory copying. And if you want to use the p-variant of posix_spawn you’re back to setting PATH beforehand. These APIs designed back in Stone Age just aren’t very well thought-out wrt concurrency and high performance.
  - jstimpfle 3y ago
    Why would you change the path just to call posix_spawnp()? If you want that control, that is an indication that you want to specify the path to the executable, not use PATH.
eqvinox 3y ago
Shells don't generally use the libc environment; this would be too limited to implement even standard POSIX shell functions with local variables, or non-exported variables. It's much easier to set up purpose-built data structures to track variables, and construct an argument for execve(). (Edit: removed unneeded pointing out execve) Also shells generally have their own program search anyway since they need to support built-in commands. It's not particularly hard to implement PATH search.
- oefrha 3y ago
  Once again, the OP asked why setenv is even needed, which implies they likely don’t have much experience with spawning processes in low level languages, so I used the more familiar shell script setting as an illustrative example, as setenv is analogous to export in POSIX sh. I never said export is implemented with setenv, or shell script exports aren’t thread safe. Unfortunately, replies hung up on shell scripts. As for I’m not aware of execve etc… You need to re-read my comment which clearly mentions execve, execvep, posix_spawn, as well as implementing PATH search on your own.
  - eqvinox 3y ago
    > Once again, the OP asked why setenv is even needed, which implies they likely don’t have much experience with spawning processes in low level languages I am the OP and your assumption is incorrect. You may consider why the post ends with: (NB: starting a new process is not "at runtime")
    - Izkata 3y ago
      "export" in shells has to change the environ before they start the new process. It may not be "at runtime" for the new process, but it would be for the shell.
      - account42 3y ago
        Wrong, export does not have to change the shell's environment at all. There are plenty of exec variants that accept a different environment pointer, same for posix_spawn.

by oefrha 3y ago

quickthrower2 3y ago

Shell scripts are different as you are likely exporting environment variables and then starting new processes.

oefrha 3y ago
Shell scripts aren’t different from “real” programs using exec or posix_spawn in this regard, it’s just that fewer people have done the latter than the former, so the former is a more relatable example. “Real” programs spawn other processes too you know, sometimes with modified environ.
- quickthrower2 3y ago
  So I understand this right, I thought the issue is about multiple threads but in shell you wouldn’t have this just new processes. In a program you could have either.

rcxdude 3y ago

Shell scripts are not really prone to this problem because AFAIK no shells are multithreaded: subshells and the like are implemented with fork()

oefrha 3y ago
Yes, I’m not saying shell scripts are affected, merely using them as an example to answer the question “Why do any of these [env vars] need to change at runtime?”
- xxs 3y ago
  The discussion is only relevant for a shared unguarded resource (the env) modified and read by multiple threads. Single threaded operations are just fine.
  - Someone 3y ago
    > Single threaded operations are just fine. Sort of. https://pubs.opengroup.org/onlinepubs/9699919799/functions/getenv.html https://pubs.opengroup.org/onlinepubs/9699919799/functions/g...: “The returned string pointer might be invalidated or the string content might be overwritten by a subsequent call to getenv()” There’s little you can do with a broken API, so Linux has that ‘feature’, too. https://man7.org/linux/man-pages/man3/getenv.3.html https://man7.org/linux/man-pages/man3/getenv.3.html: “The string pointed to by the return value of getenv() may be statically allocated, and can be modified by a subsequent call to getenv(), putenv(3), setenv(3), or unsetenv(3).” FreeBSD chooses to leak memory, instead. https://man.freebsd.org/cgi/man.cgi?getenv(3) https://man.freebsd.org/cgi/man.cgi?getenv(3): “Successive calls to setenv() that assign a larger-sized value than any previous value to the same name will result in a memory leak. The FreeBSD semantics for this function (namely, that the contents of value are copied and that old values remain accessible indefinitely) make this bug unavoidable”

xxs 3y ago

>Have you ever exported anything in a shell script So, shells use a single thread that can safely modify the environment - then start new child processes by the same thread. The child processes get a =copy= of the said environment. That's a textbook example how to use env. Starting multiple threads on your own, then modifying env should be considered a textbook example how not to do things - env is not intended for interprocess communication.

anttihaapala 3y ago

In the case of execvp, you would pretty much be required to fork before it and then you could change PATH.

oefrha 3y ago
Yeah, fork()+immediately exec() should be safe, but those use cases are almost always better with posix_spawn(), due to issues with fork(), like memory copying. And if you want to use the p-variant of posix_spawn you’re back to setting PATH beforehand. These APIs designed back in Stone Age just aren’t very well thought-out wrt concurrency and high performance.
- jstimpfle 3y ago
  Why would you change the path just to call posix_spawnp()? If you want that control, that is an indication that you want to specify the path to the executable, not use PATH.

eqvinox 3y ago

Shells don't generally use the libc environment; this would be too limited to implement even standard POSIX shell functions with local variables, or non-exported variables. It's much easier to set up purpose-built data structures to track variables, and construct an argument for execve(). (Edit: removed unneeded pointing out execve) Also shells generally have their own program search anyway since they need to support built-in commands. It's not particularly hard to implement PATH search.

oefrha 3y ago
Once again, the OP asked why setenv is even needed, which implies they likely don’t have much experience with spawning processes in low level languages, so I used the more familiar shell script setting as an illustrative example, as setenv is analogous to export in POSIX sh. I never said export is implemented with setenv, or shell script exports aren’t thread safe. Unfortunately, replies hung up on shell scripts. As for I’m not aware of execve etc… You need to re-read my comment which clearly mentions execve, execvep, posix_spawn, as well as implementing PATH search on your own.
- eqvinox 3y ago
  > Once again, the OP asked why setenv is even needed, which implies they likely don’t have much experience with spawning processes in low level languages I am the OP and your assumption is incorrect. You may consider why the post ends with: (NB: starting a new process is not "at runtime")
  - Izkata 3y ago
    "export" in shells has to change the environ before they start the new process. It may not be "at runtime" for the new process, but it would be for the shell.
    - account42 3y ago
      Wrong, export does not have to change the shell's environment at all. There are plenty of exec variants that accept a different environment pointer, same for posix_spawn.