9 ms·
The temptation of writing shell scripts, illustrated
- bigfuz 5y agoShow the shell script!
- readingnews 5y agoAgreed, I found the problem interesting in that I have run into this issue but never thought to look in /sys/fs/cgroup...
- klodolph 5y agoPerl seems to be fading away, but for a long time it was the obvious tool to use when your program got too big for a shell script. > Python, Go and so on need more structure and often don't have quite as simple and convenient methods of doing various shell like things. This is Perl's niche, or at least it was Perl's first niche before it started getting used for CGI. The things you'd do in a shell script are generally convenient in Perl too, the main inconvenient part is selecting from the many different ways to invoke a program in Perl, like system(), backticks, qx//, open(), etc. Perl is also more commonly installed and has faster startup time than Python, in my experience.
- contingencies 5y agoperl 5 was great but it broke down as soon as you needed a depth or complexity of data structure. The number of times you had to man perldsc ... every single one had a different syntax for declaration and dereferencing ... AoAoHoHoAARGH!
- zengargoyle 5y ago$ perl -E 'my $f=[[{X=>{Y=>["FOO"]}}]]; say $f->[0][0]{X}{Y}[0];my $g=$f->[0][0]{X}{Y}; say $g->[0]' FOO FOO Seems pretty trivial.
- contingencies 5y agoYeah now try iterating. What used to get me is for some types $ is used only when de-referencing. It was terrible trying to track those errors down. Basically I loved perl but the language has a terrible design, inexcusable really. There was essentially no point in having types, it took all the worst parts of C and then reinvented them slightly differently with new syntax, twice per datatype. Glad to be rid of it.
- ryl00 5y agoI have the opposite problem... I find perl's data structures so ridiculously easy to write that I can quickly build up arrays of hashes of arrays of (ad infinitum) while in the "flow". Throw in the ability to actually get references to things, and control de-referencing of things, and I've never felt perl has stopped me (for better or worse!) from doing anything with data structures.
- lightbulbjim 5y agoRuby makes for a good “super shell” script too.
- boesboes 5y agoDefinitly. I've been replacing some of my more janky & complex bash scripts with ruby and it becomes a lot cleaner imo. Also, I recently 'discovered' Pathname, it takes away a lot of the pain of dealing with files and directories in ruby and allows for pretty clean code imo.
- viraptor 5y agoAnd if you're conflicted between thinking both this and "but go is nice and static and free of runtime", you can look at https://crystal-lang.org/ https://crystal-lang.org/ Ruby syntax with static compilation for your tools.
- VariableStar 5y agoYes, Perl is still used and it is fast and excellent for anything that becomes too big or complicated for a shell or awk script. As you write it is installed in the vast majority of *NIXes as default.
- ilyash 5y agoIt is now the niche of Next Generation Shell. Totally not humble and biased opinion... maybe. I'm the author. You are very welcome to make your own judgement - https://ngs-lang.org/ https://ngs-lang.org/
- mg 5y agoPro shell scripts: - I often encounter machines without Python or Go. I never encounter machines without Bash. - Python and Go will probably change faster than Bash. So the maintenance cost of scripts in Python or Go will be higher. - When you figured out something tricky on the command line, you can paste it into a Bash script verbatim. - Bash is the perfect glue language. If the task at hand is to glue together some commands, nothing beats Bash.
- HKH2 5y agoWhy would anyone change Bash now?
- danmur 5y agoIt would make a superb April fool's if done correctly :)
- inferiorhuman 5y agoApple shipped an "old" version of bash due to licensing issues which means if you're relying on any bash 5 extensions you're SOL. Now that zsh is the default is Apple even shipping bash anymore? Things get even more fun if you've multiple versions of bash installed at once.
- andrewl-hn 5y agoYes, they still ship bash because 99.9% of shell scripts have bash in their shebang. Backward compatibility is king.
- matsemann 5y agoPython is such a mess, most likely the script you wrote won't work on a coworker's machine. Either it accidentally needs some global package installed, and then if course the needed version interfers with something else. Or the script only works for certain versions of python etc. For most of our big scripts at work, I've actually resorted to just running them inside a docker container, mounting my current folder. So tired of trying to get my env to match someone else's.
- xg15 5y agoBit of questionable advice: I've found using jq for all the argument mangling/postprocessing/glue tasks instead of sed, awk etc to be pretty workable lately. The -R switch reads in arbitrary text instead of json and presents it as a sequence of json strings internally. The capture() filter applies regexes to the string and extracts subgroups (which will be represented as json objects). The gsub() filter performs search-and-replace on a string. Finally, the -r switch writes the result back to stdout as a raw value, removing any json quoting and escaping. By default, processing is linewise, but there are options to process the whole input at once instead. There is also supposed to be a way how you can have per-line processing with state tracking across lines (like sed's "holding space"), but I haven't tried that out yet. I found this approach often more flexible than sed and easier to reason about. Also, I believe it may be faster, as more things are happening in a single process. The one exception: grep on files is still orders of magnitude faster than anything else. I believe there also was an article about this on HN a while ago. So if you have to find some data in a larger collection of files and then have to turn the data into something structured, a good approach is to make a "coarse" query with grep or grep -R first, to narrow down your set of candidates, then postprocess your remaining candidates with jq. Also, apropos single process: Another questionable advice is that for/while loops in bash are subshells and you can use them as part of a pipe chain. This often lets you reduce the number of processes you need. E.g. instead of for f in $files; do foo "$f" | sed ...; done You can often write: for f in $files; do foo "$f"; done | sed ...; Which will only launch a single sed process for all files instead of one per file. But yeah, shell scripts do have many shortcomings and in particular the error handling situation seems extremely hard to fix. So actually sensible advice is still: Use python.
- dmurray 5y ago> The one exception: grep on files is still orders of magnitude faster than anything else. I believe there also was an article about this on HN a while ago. I'd like to see that article! Faster than jq for sure, but my experience is that ack/ag/rg are all orders of magnitude faster than grep for typical uses.
- xg15 5y ago
- xg15 5y agoI think one of the things that still make shell scripts so alluring is that it's the same language you use for your shell. Many of my scripts for personal use were basically just cleaned up and slightly extended versions of the command sequences I typed in anyway. This is different from writing a script in a proper language where you usually start from scratch. I believe, this was Perl's approach originally: Let you use your shell commands as a starting point and then gradually extend the code in a safer language. However, this still leaves you with many of bash's syntax issues. On the opposite side, many modern languages do have REPLs - but I have not seen anyone so far use them as actual day-to-day shells. So I think to really get rid of shell scripting, you'd have to develop some syntax that would work well both for scripts and for shell use.
- mjevans 5y agoI think that's the void PowerShell TIRED to fill, but I loath even trying to read such a script because everything is so obtuse. Unix style shells only blindly pass around octet streams, implicitly delimited by \n characters, but really a blind pipe of bytes is the true interface. As it _can_ be anything, to get anything other than human text each program needs an extra flag that alters it's output and thus documents the new format. There's also nothing that stops a program from accepting arguments that arrange for a nest of files and/or fifos (which can also be seen as a type of named pipe) for their interface instead. Unix style shells start simple and get as complex as desired, rather than a set of typed objects which must all be kept in memory at the same time. I think what PowerShell showed me is the next step above something the complexity of a Unix shell is a "real programming language", no matter if it's interpreted live (scripting) or compiled. There might be reason to revisit the exact syntax, but I'll wait to hold my breath until we've replaced keyboard layouts designed to slow down typing to a point which doesn't break a mechanical typewriter.
- csydas 5y agoI'm always very mixed on PowerShell as it can be quite useful and it's mature enough in windows that it really does simplify a lot of windows administration tasks to be trivial and can save a ton of time there by having a nice and safe way to avoid working with the windows ui, a goal that everyone should strive for. The verbosity is not lost on me a d k do find it tiring to read scripts, and it's exacerbated when you read a poweshell script written by someone who started with a "real" language and tries to make PowerShell work like those languages. Windows apps that have PowerShell cmdlets are a boon also to administration also and I'm grateful when the option is there. My biggest gripe with PowerShell though is that as soon as you get past the "I don't want to click around windows/.net apps", Powershell shows how shallow it is and its quirks affect performance, design, etc. The PowerShell community encourages a lot of practices that end up trapping you in bad design which makes maintenancen and readability of scripts a challenge. (the handling of variables and their scope can be confusing as PowerShell allows for a lot of "cheating" on which data is accessible and the script will work, but it's very hard to tell sometimes the state of the data being worked on and where it came from. Use of += for adding to arrays is fastest to write at first blush, but you end up with awful performance because it copies the array, but your average PowerShell script writer wouldn't know it and ends up wondering why your script struggles outside of simple lab scale tests) The answer to this is getting into reflection and using .net calls, but to get the right performance and behavior from PowerShell you end up with a good portion of the script being just straight up .net code and at that point I wonder why bother with powershell at all and not just use .net and some faster API to get data from the app or going through wmi to work with windows instead. You can get complex PowerShell programs for sure. But they're PowerShell only in name, the force powering the program is usually something else that PowerShell only hinders as you need to now deal with PowerShell-isms to make it execute. So in a sense it's very successful in that I think this is exactly what PowerShell was supposed to do. But I find as I got past short scripts that PowerShell just can't really do anything it does better than another language would. It can't work anywhere near as fast as bash or any other *nix shell can, you can get a lot of depth into windows but mostly through .net calls, and for applications, a good rest API returns the same data, but faster and in a format more useful for other applications. (both program wise and just in terms of concept) It feels like PowerShelk was (is?) supposed to be a gateway drug for .net. But it's just too easy to get hooked on it and suffer the side effects and never really take that next step to a more mature language, so you get less of high from using it but way more nasty side effects. (Also the less said about PowerShell ISE the better. I wish MS would just remove it entirely as its difference in behavior from the actual shell wastes so much time in debugging scripts)
- pm90 5y agoThe “best practice” of preferring higher level languages is questionable when it comes to System level tasks such as this one. If it’s something that needs to be part of a complex system, sure higher level langs can be powerful. If it’s a standalone thing, shell scripts are generally ok. In fact higher level languages add complexity: their build process, library updates, breaking changes of dependencies all make it a pain to maintain. Shell scripts? “ls” is never gonna change. IMO the biggest reason shell scripts are avoided is simply because most software developers aren’t very comfortable with the syntax (tbh it does feel archaic in certain places) or know how unixy stuff like signals, pipes etc really work. If you have to do system level tasks though it’s probably worth the effort to understand these things.
- jbverschoor 5y agoThe problem with shell scripts is the same as automation a gui. It’s flakey and hard to maintain. Grepping through output No uniform handling of spaces, tabs. No data structures. Multiple dialects. Shell scripts mainly work with files and process management. So many scripts don’t handle spaces correctly. This is not a problem of the creator, but because of the language.
- zelphirkalt 5y agoAlthough not following best practices when writing shell scripts is the fault of the author. For example not thinking about spaces and using $varname directly instead of adhering to best practice and enclosing it in "${...}". It is uninformed and one should inform oneself, if one is to write a shell script. Plus there is no shame in looking up, how to do it correctly. There is little excuse for getting that one wrong. I mean, if this is like one of the first shell scripsts one writes, OK, but a seasoned engineer ... nah.
- pm90 5y agoThere’s also automations like shellcheck that figure this out for you.
- jbverschoor 5y ago
- ogogmad 5y agoSome people on this website keep insisting that you should use the 20 or so Unix DSLs to solve small problems, and then graduate to a general-purpose language like Python when you have a big one. I have pointed out one problem with this, which is that it means you have to learn a ton of arcania for each Unix DSL, and you end up with a Rube Goldberg machine of DSLs invoking DSLs. This post points out another set of problems. I now just use Xonsh (a Python dialect with some syntax sugar for shell things). Some people say that this might stop working after some updates, but I don't do this for a living, and those of you who likewise don't should consider following me.
- rubatuga 5y agoTerrible idea to mix shell and Python, what were they thinking. Have random bash lines suddenly be treated as Python?
- ogogmad 5y agoThere's an escape sequence for shell commands, and an escape sequence within that for ordinary Python code. It works fine. The reality is that most Unix DSLs are similar to subsets of Python. Yes, Python is not the global optimum of programming languages forever more amen, but most Unix DSLs are effectively just ad-hoc procedural languages, and therefore special cases of Python. And if you're going to use Python aggressively for other things, then there's a benefit to using it as a shell as well, and thereby get some consistency.
- rubatuga 5y agoIf the project is just for "glue", then any reasons it shouldn't be used? I'm currently creating a "glue" project with shell scripts, and the biggest issue is that I'm creating a lot of environment variables.
- nathan_phoenix 5y agoNot sure if someone mentioned it already, but for such use cases Xonsh[1] is amazing. It allows you to use shell stuff in a python script. Basically it allows you to use the best of both worlds, i. e. you can use python for the logic flow while retaining all your quick shell oneliners when neede. And it makes this interaction of python and shell super convenient, i. e. run your oneliner but process its output with python. Much faster than coding everything in python or in shell. [1]: https://xon.sh/ https://xon.sh/
- themodelplumber 5y agoThat's cool. I'm curious how it differs from e.g. ABS... https://www.abs-lang.org/ https://www.abs-lang.org/ It was enjoyable to use for a project recently.
- nathan_phoenix 5y agoFrom a quick glance seems that the biggest difference is that ABS is a whole new language while Xonsh is just a python superset. I. e. Xonsh is python which can also parse and execute shell commands. What I see as a benefit is that it allows you to reuse your existing python and bash knowledge without having to learn (almost[1]) anything new. [1]: Xonsh does add one or two new things to python. But they are pretty straightforward and aren't required. But are hella nice for more advanced interplay between shell and python.
- ducktective 5y agoI think people are underestimating the power of `bash+curl+jq`. In pretty much every scenario where I want to consume an API, I just fire up my vim+shellcheck+shfmt setup, split my logic into functions (using `local` for private variable, mind you), then just spam `get_endpoint | jq filter`s in a `getopts` loop, and bam, suddenly you have a clean and concise CLI. Granted, Bash is quirky. So I should say I program in bash+shellcheck not just Bash. And Jq. Jq is a functional programming language in itself! I do regex or string mangling inside jq filters. Why should I use Python over this? The moment you want a clean and cool feature that Python has over Bash+jq, you should incorporate third party libraries. And adding just one non-std lib, means you have to resort to `virtual env`. And why not adding also `click` or `requests` or .. and now you have ~10 moving dependencies. Let's not talk about Python's subtle differences between versions. Compare it to Bash-as-glue-of-unix-tools approach. You want image processing? Just use `imagemagick`. You want web scraping? `pup` is your friend. ProTip: do `my_command || exit 1` if my_command is bound to fail. Or `if ! my_command; then printf >&2 "[ERROR] reasons"; exit 4 fi` for more elaborate exception-handling.
- mistrial9 5y agothat exact sequence is possible in lots of desktop environments, with their own list and exception handling. Many people here show that the result of the fast (and fun?) steps result in something that other people can't change much, and have a lot of non-obvious footguns. Last, if you like it, no judgement from me about that!
- brainzap 5y agothis is a fair response, the curse of knowledge is real
- andrewl-hn 5y agoShell scripting is not bad these days. There’s a linter - Shellcheck - that will catch most gotchas in scripts, there are a lot more resources to learn shell online, and there are tools like Bats to make your life easier if your shell scripts grow too big. Writing logic - functions, for loops, or if statements- is still very annoying in Bash. Likewise writing a piece of code that glues calls to external commands is extremely clunky in almost all other languages (except Perl an Ruby), notably Python which is often treated as a Bash replacement. Finding the right balance can be tricky, but choosing only Shell or only Python is not the right way to go.
- GekkePrutser 5y agoYeah I think PowerShell has a perfect balance there. Of course they were able to do so by taking lessons learned from 30 years and didn't have any legacy baggage. Piping structured objects between apps without having to parse output is amazing. I hope Linux will come up with something similar. Of course PowerShell is open source but I don't trust MS enough to adopt it privately. I do use it at work though which makes sense as we're a MS shop.
- eternityforest 5y agoThe only time I use shell is if I expect a pure list of commands with no loops or significant logic. Actually, I'm not sure I've ever done a loop in bash at all. The few times I've tried to write even trivial programs in bash resulted in me eventually encountering something ugly and unpleasant, then quitting and starting over in Python, and having a working solution extremely fast.
- crucialfelix 5y agoI wrote a few scripts in Deno this week. In a single file I can import packages like markdown for parsing. It's typescript. I can handle arg parsing. Shebang works. This will be my go to for small local scripts
- donatj 5y ago“Master Foo and the Ten Thousand Lines” is all that needs be said http://catb.org/~esr/writings/unix-koans/ten-thousand.html http://catb.org/~esr/writings/unix-koans/ten-thousand.html
- tomohawk 5y agoI had this problem, so I wrote a script. Then, I had 2 problems... - old saying Seriously, though, treating shell script as a prototyping tool can be really powerful. Don't be afraid to rewrite it again as a 'proper' program if you have concerns about maintainability, etc.
- kkfx 5y agoShell are old classic human-computer UIs, so they are battle tested, well known, daily used by almost anyone who can write a script. That means: scripting is natural for a vast cohort of users, it's natural because while weird in language terms it's a glue of a set of known and frequently used tools. There are much more shell users than Go, Python, ... programmers. Also those who happen to be programmers normally use certain languages, typically more than one, typically far more diverse than just most used shells, for a certain kind of activity, like programming in a team a not-that-small project. As a result their mindset is not "quickly do something/quickly automate something for me, myself and I only, on my own desktop" but to be part of a larger game with need to cooperate, produce code easy to read by others, create a set of entry points for integration etc. Those programmers are not all, but mostly also shell users and they tend to use the shell not just for specific large projects but also for quick, simple&dirty personal stuff. So as non-programmers who use the shell they know and use it daily. In the past we have had perhaps better tools, like Smalltalk on Xerox desktops or Emacs/zmacs lisp on LispM BUT their actual public is next to 0 and modern Emacs-ers are on modern desktops not LispM and they are typically born on modern desktop so they know the shell and in Emacs they'll wrap anyway countless shell-centric tools so end the end a script is often quicker and simpler...
- t3we334 5y agoShell scripts require more direct systems knowledge than languages with an abstracted approach to _any_ system. As a rule of thumb if I am using python to drive (popen) system binaries I don't write in python and if I have significant data structures I avoid the shell. TBH the popular movement and younger crowd have kind of dismissed aptitude with the traditional unix shell/awk/sed as not worthy of significant effort due to any number of (mostly shallow) complaints. (gnu) Awk is a wonderful language in itself but sadly most users don't know anything about it and use it for trivial operations. These days if I am working with younger people in a team environment I default to python and avoid the bickering. That's the way wars are won - make it unpleasant enough that surrender is a better option.
- mc4ndr3 5y agoshell: I don't give a care about anything. posix shells with `set -euf`: I give one care. perl, erlang: I care about arity. python: I give one care about data type consistency. java: I care more about types than command line interoperability. ruby: I care about some things but not speed or memory or scalar/vector predictability. lua: I care about everything but indexing. c/c++: I care too much. go: I care about concurrency. rust: I care about making fun of Stroustrup. haskell: I care about... whatever. I'll find out some day when I feel more eager. javascript: Burn the world! Burn it to the ground! bf: I care about obfuscation. altJVM/altJS: I care about obfuscation--with data types. .NET: I care about taking a paycheck. cobol: I care about indentation even more than Python does. random DSL's: I care about doing it all over again. d: I care so much but nobody ever listens to me. PowerShell: I care about COMSPEC environments. non-POSIX shell: I care about the good old days. assembler: No you don't. HDL's: Yes you do. php: I care about pre-Rails projects. lisp: I cdar care less. regex: You'll care when I break prod in a week.
- pvinis 5y agoHas anyone else used https://github.com/google/zx https://github.com/google/zx here? I've started using that instead of bash script and js/node/deno scripts. it's kind of a nice combination of things. the only pain with this is writing to files. > doesn't just work.
- patrck 5y agoHere's an attempt at the script. Let's golf it! N=10 find /sys/fs/cgroup/ -name memory.current | while read f; do test -e $f || continue echo `cat $f` $f done | sort -n | tail -n $N | awk ' match($2, /system.slice.([a-z-]*).service/, arr) { $2 = "/system/" arr[1]; } match($2, /user.slice.user-([0-9]*).slice/, arr) { cmd = "id -un " arr[1]; cmd | getline uid; close(cmd); $2 = "/u/" uid; } match($2, /cgroup.([a-z]*).slice/, arr) { $2 = "/system/" arr[1]; } { print }'
- patrck 5y agoThat said, not sure why `ps` isn't a better choice? ps -e -o size,user,comm | sort -n | tail
- nikosg 5y ago> you shouldn't write anything much as a shell script and should instead use a proper programming language why?