4 ms·

Path globbing, pipes, redirection, job control (fg/bg), and all shell variables -- not just $PATH -- are all handled by the shell. The kernel has no idea what

Path globbing, pipes, redirection, job control (fg/bg), and all shell variables -- not just $PATH -- are all handled by the shell. The kernel has no idea what the current process' environment $PATH is, and doesn't even parse any process environment variables at all.

thayne 1y ago
PATH isn't just handled by the shell though. Many (but not all!) of the exec* family of functions in libc respect PATH.
- opello 1y ago
  It seems too far to go to say that because a system library holds some implementation details that the responsibility doesn't lie with the program using them. There's all sorts of complex interdependent details that make those kind of boundary distinctions difficult in many operating systems.
  - matheusmoreira 1y ago
    On Linux the main boundary between user space and kernel is quite clear: the system call layer. It is stable and well documented. https://github.com/torvalds/linux/blob/master/Documentation/ABI/stable/syscalls https://github.com/torvalds/linux/blob/master/Documentation/... System libraries like glibc are not part of the kernel, they are just components that can be replaced. I wrote an article about it: https://www.matheusmoreira.com/articles/linux-system-calls https://www.matheusmoreira.com/articles/linux-system-calls I even asked Greg Kroah-Hartman about it: https://old.reddit.com/r/linux/comments/fx5e4v/im_greg_kroahhartman_linux_kernel_developer_ama/fn3rpht/ https://old.reddit.com/r/linux/comments/fx5e4v/im_greg_kroah... > So we rely on different libc projects to provide this, and work with them when needed. > This ends up being more flexible as there are different needs from a libc, and for us to "pick one" wouldn't always be fair. > And yes, you can just use a "nolibc" type implementation of you like. > I know I do that for new syscalls when working on them, there's nothing stopping anyone else from doing that as well. You can trash the entire GNU system and rewrite it all in Rust or Lisp if you wanted. It doesn't have to be some POSIX-like thing either, it could be whatever you wanted it to be. It doesn't need to have things like PATH. You could write a static freestanding application and boot Linux directly into it. Nobody does stuff like this it's a lifetime of work. But it could be done.
    - opello 1y ago
      That is indeed one of the more well defined boundaries in the system. Also worth understanding is that programs aren't generally invoking system calls directly, for example calling interrupt 0x80, glibc provides wrapper functions that invoke system calls, blurring the boundary a bit. Further blurring the boundary is the vDSO layer that intercepts some system call wrappers for more efficient access. At issue in this article and comment thread is the boundary between the shell, environment, and Linux. This is a blurrier boundary still because the shell sets up the environment, which is passed through the kernel, and interpreted for downstream processes, generally (but not necessarily) by that shared system library.
      - matheusmoreira 1y ago
        > Also worth understanding is that programs aren't generally invoking system calls directly They don't generally do that but they absolutely can. I wrote a Lisp interpreter that does just that. It's completely static, has zero dependencies and talks to the kernel directly. The idea is to implement every primitive on top of Linux, and everything else on top of the primitives. From the kernel's perspective, every program is talking to it directly. They just typically use glibc routines to do it for them. There's no actual need for glibc to be there though. At some point I even tried adding Linux system call builtins to GCC so that the compiler itself would generate the code in the correct calling convention. Lost that work due to a hard disk crash but on the mailing list I didn't get the impression the maintainers favored merging it anyway. > for example calling interrupt 0x80, glibc provides wrapper functions that invoke system calls, blurring the boundary a bit Not all of them. It still doesn't support all of the clone system calls. https://www.man7.org/linux/man-pages/man2/clone.2.html https://www.man7.org/linux/man-pages/man2/clone.2.html Note: glibc provides no wrapper for clone3(), necessitating the use of syscall(2). It's not just niche system calls either. It took years for glibc to provide getrandom. https://www.man7.org/linux/man-pages/man2/getrandom.2.html https://www.man7.org/linux/man-pages/man2/getrandom.2.html https://lwn.net/Articles/711013/ https://lwn.net/Articles/711013/ It's really annoying how these glibc wrappers get confused with the actual Linux system calls which work very differently. The most notable difference is there's no global thread local errno nonsense with the real system calls, the kernel just gives you a perfectly normal return value in a register. There's also a ton of glibc machinery related to system call cancellation that gets linked in if you use it. Documentation out there conflates the two. I expected the man page above to describe only the Linux system call but it also describes the glibc specific stuff. That way people get the impression they are one and the same. > Further blurring the boundary is the vDSO layer that intercepts some system call wrappers for more efficient access. The vDSO is a documented stable Linux kernel interface: https://github.com/torvalds/linux/blob/master/Documentation/ABI/stable/vdso https://github.com/torvalds/linux/blob/master/Documentation/... It's just a perfectly normal ELF shared object that the kernel maps into the address space of every process on certain architectures. Its address is passed via the auxiliary vector which is located immediately after the environment vector. Glibc merely finds it and uses it. I can make my interpreter use it too. It's completely optional. Its purpose is making certain system calls faster by eliminating the switch to kernel mode. This is useful for time/date system calls which are invoked frequently. The original system calls are still available though. > This is a blurrier boundary still because the shell sets up the environment, which is passed through the kernel, and interpreted for downstream processes, generally (but not necessarily) by that shared system library. The shell passes the environment to the execve system call but the kernel does not interpret it in any way. It doesn't even enforce the "key=value" format since this is just a convention. It's essentially an opaque array of strings and it's up to user space to make sense of whatever it contains. Glibc chooses to parse those strings into program state in the form of environment variables whose values programmers can query.
- kccqzy 1y ago
  There is also paths.h usually located at /usr/include/paths.h. It contains the default PATH macro _PATH_DEFPATH.
- matheusmoreira 1y ago
  Those functions aren't the real system calls provided by Linux, they're just glibc wrappers with added functionality. Linux kernel execve has absolutely no concept of PATH, it just opens the file at the provided pathname. That's a good thing too, user space might want to customize that stuff.
  - thayne 1y ago
    Sure, but it is also not the same thing as the shell.
    - matheusmoreira 1y ago
      Yes. Shells typically do their own path resolution as well. I know GNU bash does, at least. I customized that logic in order to make a little library system for shell scripts.
  - deleted 1y ago
    [deleted]

thayne 1y ago

PATH isn't just handled by the shell though. Many (but not all!) of the exec* family of functions in libc respect PATH.

opello 1y ago
It seems too far to go to say that because a system library holds some implementation details that the responsibility doesn't lie with the program using them. There's all sorts of complex interdependent details that make those kind of boundary distinctions difficult in many operating systems.
- matheusmoreira 1y ago
  On Linux the main boundary between user space and kernel is quite clear: the system call layer. It is stable and well documented. https://github.com/torvalds/linux/blob/master/Documentation/ABI/stable/syscalls https://github.com/torvalds/linux/blob/master/Documentation/... System libraries like glibc are not part of the kernel, they are just components that can be replaced. I wrote an article about it: https://www.matheusmoreira.com/articles/linux-system-calls https://www.matheusmoreira.com/articles/linux-system-calls I even asked Greg Kroah-Hartman about it: https://old.reddit.com/r/linux/comments/fx5e4v/im_greg_kroahhartman_linux_kernel_developer_ama/fn3rpht/ https://old.reddit.com/r/linux/comments/fx5e4v/im_greg_kroah... > So we rely on different libc projects to provide this, and work with them when needed. > This ends up being more flexible as there are different needs from a libc, and for us to "pick one" wouldn't always be fair. > And yes, you can just use a "nolibc" type implementation of you like. > I know I do that for new syscalls when working on them, there's nothing stopping anyone else from doing that as well. You can trash the entire GNU system and rewrite it all in Rust or Lisp if you wanted. It doesn't have to be some POSIX-like thing either, it could be whatever you wanted it to be. It doesn't need to have things like PATH. You could write a static freestanding application and boot Linux directly into it. Nobody does stuff like this it's a lifetime of work. But it could be done.
  - opello 1y ago
    That is indeed one of the more well defined boundaries in the system. Also worth understanding is that programs aren't generally invoking system calls directly, for example calling interrupt 0x80, glibc provides wrapper functions that invoke system calls, blurring the boundary a bit. Further blurring the boundary is the vDSO layer that intercepts some system call wrappers for more efficient access. At issue in this article and comment thread is the boundary between the shell, environment, and Linux. This is a blurrier boundary still because the shell sets up the environment, which is passed through the kernel, and interpreted for downstream processes, generally (but not necessarily) by that shared system library.
    - matheusmoreira 1y ago
      > Also worth understanding is that programs aren't generally invoking system calls directly They don't generally do that but they absolutely can. I wrote a Lisp interpreter that does just that. It's completely static, has zero dependencies and talks to the kernel directly. The idea is to implement every primitive on top of Linux, and everything else on top of the primitives. From the kernel's perspective, every program is talking to it directly. They just typically use glibc routines to do it for them. There's no actual need for glibc to be there though. At some point I even tried adding Linux system call builtins to GCC so that the compiler itself would generate the code in the correct calling convention. Lost that work due to a hard disk crash but on the mailing list I didn't get the impression the maintainers favored merging it anyway. > for example calling interrupt 0x80, glibc provides wrapper functions that invoke system calls, blurring the boundary a bit Not all of them. It still doesn't support all of the clone system calls. https://www.man7.org/linux/man-pages/man2/clone.2.html https://www.man7.org/linux/man-pages/man2/clone.2.html Note: glibc provides no wrapper for clone3(), necessitating the use of syscall(2). It's not just niche system calls either. It took years for glibc to provide getrandom. https://www.man7.org/linux/man-pages/man2/getrandom.2.html https://www.man7.org/linux/man-pages/man2/getrandom.2.html https://lwn.net/Articles/711013/ https://lwn.net/Articles/711013/ It's really annoying how these glibc wrappers get confused with the actual Linux system calls which work very differently. The most notable difference is there's no global thread local errno nonsense with the real system calls, the kernel just gives you a perfectly normal return value in a register. There's also a ton of glibc machinery related to system call cancellation that gets linked in if you use it. Documentation out there conflates the two. I expected the man page above to describe only the Linux system call but it also describes the glibc specific stuff. That way people get the impression they are one and the same. > Further blurring the boundary is the vDSO layer that intercepts some system call wrappers for more efficient access. The vDSO is a documented stable Linux kernel interface: https://github.com/torvalds/linux/blob/master/Documentation/ABI/stable/vdso https://github.com/torvalds/linux/blob/master/Documentation/... It's just a perfectly normal ELF shared object that the kernel maps into the address space of every process on certain architectures. Its address is passed via the auxiliary vector which is located immediately after the environment vector. Glibc merely finds it and uses it. I can make my interpreter use it too. It's completely optional. Its purpose is making certain system calls faster by eliminating the switch to kernel mode. This is useful for time/date system calls which are invoked frequently. The original system calls are still available though. > This is a blurrier boundary still because the shell sets up the environment, which is passed through the kernel, and interpreted for downstream processes, generally (but not necessarily) by that shared system library. The shell passes the environment to the execve system call but the kernel does not interpret it in any way. It doesn't even enforce the "key=value" format since this is just a convention. It's essentially an opaque array of strings and it's up to user space to make sense of whatever it contains. Glibc chooses to parse those strings into program state in the form of environment variables whose values programmers can query.
kccqzy 1y ago
There is also paths.h usually located at /usr/include/paths.h. It contains the default PATH macro _PATH_DEFPATH.
matheusmoreira 1y ago
Those functions aren't the real system calls provided by Linux, they're just glibc wrappers with added functionality. Linux kernel execve has absolutely no concept of PATH, it just opens the file at the provided pathname. That's a good thing too, user space might want to customize that stuff.
- thayne 1y ago
  Sure, but it is also not the same thing as the shell.
  - matheusmoreira 1y ago
    Yes. Shells typically do their own path resolution as well. I know GNU bash does, at least. I customized that logic in order to make a little library system for shell scripts.
- deleted 1y ago
  [deleted]