3 ms·

I’ve done my share of shell tricks in the past and I love them, but I got very disappointed when I noticed that the performance sucks for anything more data int

by haolez 7y ago

I’ve done my share of shell tricks in the past and I love them, but I got very disappointed when I noticed that the performance sucks for anything more data intensive in the shell. At least in bash, which was the interpreter that I’ve used. I ended up porting my code to Python and gained at least 10x in performance. But I prefer the shell code for tasks interfacing with system tools. I wonder if there is a shell interpreter around that is really fast.

chubot 7y ago
If you have some realistic shell benchmarks, please file a bug and link the files: https://github.com/oilshell/oil https://github.com/oilshell/oil Oil isn't fast at the moment, but I'm currently optimizing it (auto-translating some code to C++, and eventually hand-optimizing other parts.) I run benchmarks on every release, so speed is definitely a goal. I haven't seen that many shell scripts that are "CPU bound" (as opposed to waiting for processes or I/O). (Although now I recall a shell-script-library bundler project that was.) Of course, one big problem with shell is that you often have to spawn an external process like sed or awk to do string manipulation that's just a library call in Python. That could easily be a three order-of-magnitude difference -- e.g. milliseconds vs. microseconds.
- stephenr 7y ago
  I have a feeling the “bundler” lib you mentioned is mine - we discussed it a bit on lobste.rs a few years ago? Good to see you’re still plugging away on oil!
  - chubot 7y ago
    Yes! I remember it was a good workload because it was doing a lot of string manipulation in pure shell. I'm still interested in more feedback on Oil (e.g. does it still work on the latest version of shell-script-library? :) ) It's changed a lot, and the Oil language now exists! Latest post: You Can Now Try the Oil Language -- http://www.oilshell.org/blog/2019/10/04.html http://www.oilshell.org/blog/2019/10/04.html
    - stephenr 7y ago
      The testing setup for the lib is much improved, with actual unit tests and a built in capability for building/testing against a specific shell. Next time I’m working on it I’ll grab a release of oil and see how it fares!
haolez 7y ago
Great! I hope you manage to pull this off! It's a much needed tool in our IT world. I remember seeing something like what you are doing a while back, but it was such a small project that I can't recall its name or find it via Google. But I think it was a shell favoring functional programming style meant for plan9.

by haolez 7y ago

chubot 7y ago

If you have some realistic shell benchmarks, please file a bug and link the files: https://github.com/oilshell/oil https://github.com/oilshell/oil Oil isn't fast at the moment, but I'm currently optimizing it (auto-translating some code to C++, and eventually hand-optimizing other parts.) I run benchmarks on every release, so speed is definitely a goal. I haven't seen that many shell scripts that are "CPU bound" (as opposed to waiting for processes or I/O). (Although now I recall a shell-script-library bundler project that was.) Of course, one big problem with shell is that you often have to spawn an external process like sed or awk to do string manipulation that's just a library call in Python. That could easily be a three order-of-magnitude difference -- e.g. milliseconds vs. microseconds.

stephenr 7y ago
I have a feeling the “bundler” lib you mentioned is mine - we discussed it a bit on lobste.rs a few years ago? Good to see you’re still plugging away on oil!
- chubot 7y ago
  Yes! I remember it was a good workload because it was doing a lot of string manipulation in pure shell. I'm still interested in more feedback on Oil (e.g. does it still work on the latest version of shell-script-library? :) ) It's changed a lot, and the Oil language now exists! Latest post: You Can Now Try the Oil Language -- http://www.oilshell.org/blog/2019/10/04.html http://www.oilshell.org/blog/2019/10/04.html
  - stephenr 7y ago
    The testing setup for the lib is much improved, with actual unit tests and a built in capability for building/testing against a specific shell. Next time I’m working on it I’ll grab a release of oil and see how it fares!

haolez 7y ago

Great! I hope you manage to pull this off! It's a much needed tool in our IT world. I remember seeing something like what you are doing a while back, but it was such a small project that I can't recall its name or find it via Google. But I think it was a shell favoring functional programming style meant for plan9.