4 ms·

It seems like there is potential for making bash faster. 1. JIT compilation, caching and invalidating on updated source files 3. Rearchitecting similar a-la b

It seems like there is potential for making bash faster. 1. JIT compilation, caching and invalidating on updated source files 3. Rearchitecting similar a-la busybox all of coreutils, sharutils, grep, awk and sed as internal components to avoid forking as much as possible. Tool symlinks back to itself for compatibility. 4. Rearchitecting to simulate sub-shells with threads and local contexts, rather than globals/singular locals. With formally-proven safety methodologies, there only needs to be one "shell" process running, a server. All other invocations become thread children if called directly or interact as a client if exec()ed. Subshells become just lightweight threads with Copy-On-Write contexts. No more shell fork-bombs, scripts get crazy fast and reduce first PID.

dfox 7y ago
I think that there is really good reason why you don't want to do any of that: bash shell and not general purpose language/VM. And I say that as the author of what essentially is point of sale system implemented in bash with ui based on dialog/whiptail...
saagarjha 7y ago
If you really care about speed, you probably shouldn't be using Bash.
- yesenadam 7y ago
  Also true of Python, but I almost never hear people say that. Strange.
  - saagarjha 7y ago
    Python's a bit faster than Bash.
    - vaingloriole 7y ago
      For what? Is it faster than (g)awk? How about sed? How about dd? How about any of the thousands of tools in userspace that you wrap in bash and are faster than python at the same task? Use the right tool is being lost in the 'you are too stupid to make a good decision, do it this way' generation.
      - saagarjha 7y ago
        It depends, of course! But usually the issue with shell scripts is that they create many new processes, which has high overhead. In the instances you're spending more time in dd than bash, in Python you're probably inside an in-process memcpy.
lhoursquentin 7y ago
Regarding 4. it's an interesting point, and this actually exists, few people know that ksh93 (which was a big inspiration for bash and zsh) does not fork when it doesn't really need to, for instance if you strace the following: echo $(echo $(echo echo)) Due to those command substitutions, on zsh/bash/mksh/dash/yash/busybox/osh you'll see 2 forks/clones, but on ksh93 there will be none, and this allows ksh93 to be insanely fast on "pure" shell scripts that have little use of external binaries and pipes, even faster than dash which is usually known to outperform most shells.
marcthe12 7y ago
Shell has DSL like nature to it so sh actually is better any other language. But it is not easy to perform introspection or ffi. So basically issues develop when you call alot programs with pipe and environmental variables and you need to also have unshare or poll. Like one program I had. Basically calling bunch programs in a pipeline but need to poll output of some shell async. C or python makes the pipe code complicated while emulating async Io with tmp is also complicated. In a few cases, is to have second program in python or c or perl and send messages over a pipe. 3. and 4. are helpful. JIT not so much since bottleneck seems to to be parsing rather than executing.
hnlmorg 7y ago
Hi, murex author here. Murex is not a POSIX compliant shell but there are some lessons I've learned developing murex which relate to your points: 1. I do this in murex but actually the performance improvement barely registers in benchmarks because that's not where the real bottleneck is. It's forking and it's passing data via standard streams (stdout et al) rather than sharing memory. There's lots of expensive syscalls to do stuff that you wouldn't need to do in a monolithic service. It's also worth noting that shell scripting is designed to be a highly dynamic language - even command names can be variables(!) - so there is a limit to just how much up-front compilation and caching you do (at least if you want to retain POSIX compatibility). 2. No 2? 3. I also do this to some extent in murex too, except it's not coreutils but rather a set of additional utils specific for more complex data wrangling (JSON, YAML, TOML, CSV, etc). Unfortunately it comes with a set of additional complexities, any of which are deal breakers: i) If you don't fork then processes don't register PIDs so you can't easily kill a long running process or command run in error. To get around this in murex I had to create a new, separate, function list. This list only includes processes launched from murex but it has both external files (awk et al) as well as builtins. This works for murex because murex isn't going for POSIX compatibility but this would never fly with Bash because you then need to train users to use different `kill`, `ps`, etc commands. ii) Many users take advantage of job control, which is where you can stop, resume and background already running processes. This works via signals (like SIGINT which you'd be familiar with) and is controlled by UNIX/Linux and not the shell. To get this working with non-forked builtins takes a hell of a lot of hacking and, once again, breaks POSIX compatibility. In fact I ended up removing support for SIGSTSP from murex builtins because it wasn't reliable (though I do plan on adding it again at some point). I also have a bug where external commands don't get sent a SIGTTIN (signal to tell background processes to pause if they're reading from stdin) and that's largely due to some of the hacks I've made regarding your first point. iii) re-implementing coreutils is non-trivial. In fact calling it "non-trivial" is a massively understatement. And that's without touching the other utils you've highlighted. This effort alone is enough to be a deal breaker without taking into account i and ii. The reason busybox works is because it's a subset of coreutils. In fact busybox can be a little jarring at times because it is missing so many common flags. However it does just enough to be a useful emergency / embedded shell. 4. This is how murex approaches sub-shells but you're still limited by the caching problems (raised to point 1) plus the forking requirements (addressed in point 3) in terms of just how much additional performance it will squeeze. However out of all the points you have discussed, this is the easiest to implement. It's also worth noting that fork-bombs (or equivalent exploits) will always be possible as long as your code has functions. Removing sub-shells wouldn't prevent someone from writing a self-referencing function.

dfox 7y ago

I think that there is really good reason why you don't want to do any of that: bash shell and not general purpose language/VM. And I say that as the author of what essentially is point of sale system implemented in bash with ui based on dialog/whiptail...

saagarjha 7y ago

If you really care about speed, you probably shouldn't be using Bash.

yesenadam 7y ago
Also true of Python, but I almost never hear people say that. Strange.
- saagarjha 7y ago
  Python's a bit faster than Bash.
  - vaingloriole 7y ago
    For what? Is it faster than (g)awk? How about sed? How about dd? How about any of the thousands of tools in userspace that you wrap in bash and are faster than python at the same task? Use the right tool is being lost in the 'you are too stupid to make a good decision, do it this way' generation.
    - saagarjha 7y ago
      It depends, of course! But usually the issue with shell scripts is that they create many new processes, which has high overhead. In the instances you're spending more time in dd than bash, in Python you're probably inside an in-process memcpy.

lhoursquentin 7y ago

Regarding 4. it's an interesting point, and this actually exists, few people know that ksh93 (which was a big inspiration for bash and zsh) does not fork when it doesn't really need to, for instance if you strace the following: echo $(echo $(echo echo)) Due to those command substitutions, on zsh/bash/mksh/dash/yash/busybox/osh you'll see 2 forks/clones, but on ksh93 there will be none, and this allows ksh93 to be insanely fast on "pure" shell scripts that have little use of external binaries and pipes, even faster than dash which is usually known to outperform most shells.

marcthe12 7y ago

Shell has DSL like nature to it so sh actually is better any other language. But it is not easy to perform introspection or ffi. So basically issues develop when you call alot programs with pipe and environmental variables and you need to also have unshare or poll. Like one program I had. Basically calling bunch programs in a pipeline but need to poll output of some shell async. C or python makes the pipe code complicated while emulating async Io with tmp is also complicated. In a few cases, is to have second program in python or c or perl and send messages over a pipe. 3. and 4. are helpful. JIT not so much since bottleneck seems to to be parsing rather than executing.

hnlmorg 7y ago

Hi, murex author here. Murex is not a POSIX compliant shell but there are some lessons I've learned developing murex which relate to your points: 1. I do this in murex but actually the performance improvement barely registers in benchmarks because that's not where the real bottleneck is. It's forking and it's passing data via standard streams (stdout et al) rather than sharing memory. There's lots of expensive syscalls to do stuff that you wouldn't need to do in a monolithic service. It's also worth noting that shell scripting is designed to be a highly dynamic language - even command names can be variables(!) - so there is a limit to just how much up-front compilation and caching you do (at least if you want to retain POSIX compatibility). 2. No 2? 3. I also do this to some extent in murex too, except it's not coreutils but rather a set of additional utils specific for more complex data wrangling (JSON, YAML, TOML, CSV, etc). Unfortunately it comes with a set of additional complexities, any of which are deal breakers: i) If you don't fork then processes don't register PIDs so you can't easily kill a long running process or command run in error. To get around this in murex I had to create a new, separate, function list. This list only includes processes launched from murex but it has both external files (awk et al) as well as builtins. This works for murex because murex isn't going for POSIX compatibility but this would never fly with Bash because you then need to train users to use different `kill`, `ps`, etc commands. ii) Many users take advantage of job control, which is where you can stop, resume and background already running processes. This works via signals (like SIGINT which you'd be familiar with) and is controlled by UNIX/Linux and not the shell. To get this working with non-forked builtins takes a hell of a lot of hacking and, once again, breaks POSIX compatibility. In fact I ended up removing support for SIGSTSP from murex builtins because it wasn't reliable (though I do plan on adding it again at some point). I also have a bug where external commands don't get sent a SIGTTIN (signal to tell background processes to pause if they're reading from stdin) and that's largely due to some of the hacks I've made regarding your first point. iii) re-implementing coreutils is non-trivial. In fact calling it "non-trivial" is a massively understatement. And that's without touching the other utils you've highlighted. This effort alone is enough to be a deal breaker without taking into account i and ii. The reason busybox works is because it's a subset of coreutils. In fact busybox can be a little jarring at times because it is missing so many common flags. However it does just enough to be a useful emergency / embedded shell. 4. This is how murex approaches sub-shells but you're still limited by the caching problems (raised to point 1) plus the forking requirements (addressed in point 3) in terms of just how much additional performance it will squeeze. However out of all the points you have discussed, this is the easiest to implement. It's also worth noting that fork-bombs (or equivalent exploits) will always be possible as long as your code has functions. Removing sub-shells wouldn't prevent someone from writing a self-referencing function.