28 ms·

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real langu

• marklgr 4y ago
  > "If you have to write more than 10 lines, then use a real language" I swear, there should be a HN rule against those. It pollutes every single Shell discussions, bringing nothing to them and making it hard for others do discuss the real topic.
  • dotancohen 4y ago
    There are three numbers in this industry: 0, 1 and infinity. Any other number - especially when stated as a rule, limitation, or law - is highly suspect.
    • ndsipa_pomu 4y ago
      0, 1, 3 and infinity
      • NateEag 4y ago
        Which works not just to preserve the previous statement from internal inconsistency, but also in regards to the incredibly useful Rule of Three (https://en.m.wikipedia.org/wiki/Rule_of_three_(computer_programming) https://en.m.wikipedia.org/wiki/Rule_of_three_(computer_prog...).
        • palunon 4y ago
          > Which works not just to preserve the previous statement from internal inconsistency It doesn't. You now have 4 numbers.
          • ndsipa_pomu 4y ago
            0, 1, 3, 4 and infinity - there's four numbers in this industry. Five There's five numbers in this industry 0, 1, 3, 4, 5 and infinity Wait, I'll come in again
            • tuatoru 4y ago
              0, 1, 7, and indeterminate, IME. The 7 being for design. If there are more than 7 boxes on the whiteboard, try again.
              • sargstuff 4y ago
                ah, log base 2 of 7 is 127 bits (aka 8, y 1). Unicode character can have more than 7 font boxes associated with one character box and still be a valid determinate character form.
            • sargstuff 4y ago
              thought the industry was broken down in 8 bit increments (0, 8, 16, 32, 64, 128, etc) log base 2 of 4 is only 16bits
          • NateEag 4y ago
            Good point. I'm not sure why I thought what I'd written above worked... shrug
    • pferde 4y ago
      Are you one of those people who take everything literally, so any and all jokes fly far over their heads? This rule of ten lines or less is clearly meant as an illustrative guideline. Obviously if you have a shell script that has 11 lines, but does what it has to do reliably, nobody will be bothered. The idea that the rule is trying to convey is "don't write long, complex programs in shell". Arguing about exact numbers or wording here is detracting from the topic at hand.
  • throw10920 4y ago
    The majority of those comments have significantly more thought put into them (and adhere more closely to the HN guidelines) than this comment does.
    • sargstuff 4y ago
      Is there a link to HN line discipline criteria? (beyond asci ranges 0 through 31)
  • sargstuff 4y ago
    Think use a real line discipline like n 8 1 would make more semantic sense than 'use a real lanaguage'. Unless, the language is APL, in which case, 10 lines is an operating system.
• psychstudio 4y ago
  Kindred spirit. I particularly love variable variables and exploit them often. Some would call it abuse I guess.
• ducktective 4y ago
  >However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against. Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data. dmenu + fzf + jq + curl is my bread and butter in shell scripts. However, I still haven't managed to find a way to do a bunch of tasks concurrently. No, xargs and parallel don't cut it. Just give me an opinionated way to do this that is easily inspectable, loggable and debuggable. Currenly I hack together functions in a `((job_i++ < max_jobs)) || wait -n` spaghetti.
  • kqr 4y ago
    I think this comment points to an even deeper insight: shell is a crappy programming language but with amazing extensibility. I would argue that once you pull in jq, you're no longer writing in "shell", you're writing in jq, which is a separate and different language. But that's precisely the point! Look at how effortless it is to (literally) shell out to a slew of other languages from shell. The power of shell isn't in the scripting language itself, it's in how fluidly it lets you embed snippets of tr, sed, awk, jq, and whatever else you need. And, critically, these languages callable from shell were not all there when shell was designed. The extension interface of spawning processes and communicating with arguments and pipes is just that powerful. That's where shell shines.
    • chasil 4y ago
      The shell is an ambiguous language that cannot be directly implemented with an LR parser. Perhaps some of the power emerges from that ambiguity, but it is quite difficult to implement. This presentation sums up the woes of an implementor: https://archive.fosdem.org/2018/schedule/event/code_parsing_posix_s_hell/ https://archive.fosdem.org/2018/schedule/event/code_parsing_...
  • synergy20 4y ago
    new to 'rq', it's not in active development, any other alternatives? it seems doing a lot other than convert structured data to json.
• bheadmaster 4y ago
  I also like Bash - it's a powerful language, especially when combined with a rich ecosystem of external commands that can make your life easier, e.g. GNU Parallel. Handling binary data can also work in Bash, provided that you just use it as a glue for pipelines between other programs (e.g. feeding video data into ffmpeg). One time, while working on some computer vision project, I had a need to hack up a video-capture-and-upload program for gathering training data during a certain time of day. It took me about 20 minutes and 50 lines of Bash to setup the whole thing, test it, and be sure it works.
• bitofhope 4y ago
  There are workloads where shell scripts are the so-called right tool for a job. All too often I see people writing scripts in "proper" languages and calling os.system() on every other line. Shell scripts are good for gluing programs together. It's fine to use them for that.
  • Spivak 4y ago
    For me it's once you make switch to a "proper" language you realize how much lifting pipelines do when it comes to chaining external binaries together.
    • sargstuff 4y ago
      Heaping things together is better than letting things stack up/down.
  • eschneider 4y ago
    1000% THIS. The trick, of course, is knowing when it's time to abandon shell for something more powerful, but that usually comes with experience.
  • GoblinSlayer 4y ago
    I wrote such a program, that runs other programs for heavy lifting but also parses text which you can't possibly do in bash.
    • sargstuff 4y ago
      bootloader, systemd, or init ? Parsing text isn't anything fancy. It's just knowing what the marker is for a word/item boundary. For bash, that marker is defined in IFS
      • GoblinSlayer 4y ago
        A build system for single file programs.
• snidane 4y ago
  Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It speeds you up several times. I started using inline 'python -c' more often than the python repl now as it stores the command in shell history and it is then one fzf search away. While neither Shell or SQL are perfect, there have been many ideas to improve them and for sure people can't wait for something new like oil shell to get production ready, getting the shell quoting hell right, or somebody fixing up SQL, bringing old ideas from Datalog and QUEL into it, fixing the goddamn NULL joins, etc. But honestly, nothing else even comes close to this 10x productivity increase over the next best alternative. No, Thank you, I will not rewrite my 10 lines of sh into python to explode it into 50 lines of shuffling clunky objects around. I'll instead go and reread that man page how to write an if expression in bash again.
  • j0hnyl 4y ago
    I noticed that I became so much more quick after taking 1 hour to properly learn awk. Yes, it literally takes about 1 hour.
    • ketanmaheshwari 4y ago
      Awk is awesome but saying it literally takes 1 hour to properly learn it is a bit overselling.
      • j0hnyl 4y ago
        I really don't think so! If you have experience with any scripting, you can fully grok the fundamentals of awk in 1 hour. You might not memorize all the nuances, but you can establish the fundamentals to a degree that most things you would try to achieve would take just a few minutes of brushing up. For those that haven't taken the time yet, I think this is a good place to start: https://learnxinyminutes.com/docs/awk/ https://learnxinyminutes.com/docs/awk/ Of course, some people do very advanced things in awk and I absolutely agree that 1 hour of study isn't going to make you a ninja, but it's absolutely enough to learn the awk programming paradigm so that when the need arises you can quickly mobilize the solution you need. For example: If you're quick to the draw, it can take less time to write an awk one liner to calculate the average of a column in a csv than it does to copy the csv into excel and highlight the column. It's a massive productivity booster.
• ilyt 4y ago
  > Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. It is "opinion" based on debugging scripts made by people (which might be "you but few years ago") that don't know the full extent of death-traps that are put in the language. Or really writing anything more complex. About only strong side of shell as a language is a pipe character. Everything else is less convenient at best, actively dangerous at worst. Sure, "how to write something in a limited language" might be fun mental excercise but as someone sitting in ops space for good part of 15 years, it's just a burden. Hell, I'd rather debug Perl script than Bash one... Yeah, if it is few pipes and some minor post processing I'd use it too (pipe is the easiest way to do it out of all languages I've seen) but that's about it. It is nice to write one-liners in cmdline but characteristic that make it nice there make it worse programming language. A bit like Perl in that matter
  • sophacles 4y ago
    You say this as if it wasn't extremely common to find giant python monstrosities that can be replaced by a handful of lines of shell. TBF the shell code often is not just cleaner and easier to follow, but also faster. It's possible to use the wrong tool for the job in any language - including language choice itself. Dismissing a programming language because it's not shell and dismissing shell because it's not a proramming language are the same thing - a bad idea if that's your only decision criteria.
    • GoblinSlayer 4y ago
      Bash is a good tool if the script is short enough, but if you have to write more than 10 lines, then use a real language.
      • sophacles 4y ago
        Nonsense. That's a terrible metric. If i need to run 11 commands in a row, suddenly i need to make sure new tooling is installed in my instance and/or ship a binary? What if that 11 lines is setting up some networking? Now i need to go write a 400 line go program to use netlink to accomplish the same task? Or should I condense that to 80 lines of go to shell out the commands that replicate the 11 lines of simple bash? There are plenty of reasons to do this, I have done it more than once. None of those reasons are "crossed an arbitrary magic number of 'lines of shell'".
• strunz 4y ago
  Eh, as soon as you have to deal with arrays and hash tables/dicts or something like JSON, bash becomes very painful and hard to read.
  • Spivak 4y ago
    I mean they're not that bad. declare -A mydict( [lookma]=initalization ) mydict[foo]=bar echo "${mydict[foo]}" list=() list+=(foo bar baz) echo "${list[0]}"
    • kotlin2 4y ago
      Now do an associative array containing another associative array.
      • Spivak 4y ago
        Sometimes a you just have to accept a language's limitations. Try in Python to make a nested defaultdict you can access like the following. d = <something> d["a"]["b"]["c"] # --> 42 Can't be done because it's impossible for user code to detect what the last __getitem__ call is and return the default. Edit: Dang it, I mean arbitrary depth.
        • kotlin2 4y ago
          c = defaultdict(lambda: 42) b = defaultdict(lambda: c) a = defaultdict(lambda: b) a["a"]["b"]["c"] # --> 42
          • Spivak 4y ago
            Okay fair, I deserve that. I assumed it was obvious I meant arbitrary depth. Also d["a"] and d["a"]["b"] aren't 42.
            • kotlin2 4y ago
• throwawaaarrgh 4y ago
  I would agree, with the caveat that Bourne Shell isn't really a programming language, and has to be seen as such to be loved. Bourne Shell Scripting is literally a bunch of weird backwards compatible hacks around the first command line prompt from 1970. The intent was to preserve the experience of a human at a command prompt, and add extra functionality for automation. It's basically a high-powered user interface. It emphasizes what the operator wants for productivity, instead of the designer in her CS ivory tower of perfection. You can be insanely productive on a single line, or paste that line into a file for repeatability. So many programmers fail to grasp that programming adds considerations that the power user doesn't care about. The Shell abstracts away all that unnecessary stuff and just lets you get simple things done quickly.
• Spivak 4y ago
  Also the fact that $() invokes a supparser which lets use double quotes in an already double quoted expression is something I miss when using Python-f strings.
• dimitar 4y ago
  Line-oriented pipelines are great and have their place but I'm still sticking to a high-level general purpose programming language (lets abbreviate this as HGPPL) for scripts longer than 10 lines, because the following reasons: * I like to the HGPPL data structures and convenient library for manipulating them (in my case this is Clojure which has a great core library). Bash has indexed and associative arrays. * Libraries for common data formats are also used in a consistent way in the HGPPL. I don't have to remember a DSL for every data format - i.e. how to use jq when dealing with JSON. Similarly for YAML, XML, CSVs, I can also do templating for configuration files for nginx and so on. I've seen way too many naive attempts to piece together valid YAML from strings in bash to know its just not worth doing. * I don't want to switch programming language from the main application and I find helps "break down silos" when everyone can read and contribute to some code. If a team is just sysadmins - sure, make bash the official language and stick to it. * I can write scripts without repeating myself using namespaces and higher-order functions, which my choice of paradigm for abstractions, others write cleanly with classes. You can follow best practices, avoid the use of ENV vars, but that requires extra discipline and it is hard to enforce on other for the type of places where bash is used.
• mrlemke 4y ago
  To add to this, it's designed to work in conjunction with small programs. You don't write everything using bash (or whatever shell) built-ins. It will feel like a crappier Perl. If there is some part of your script where you're struggling to use an existing tool (f.g. built-ins, system utils), write your own small program to handle that part of the stream and add it in to your pipe. Since shell is a REPL, you get instant feedback and you'll know if it's working properly. It's also important to learn your system's environment too. This is your "standard library", and it's why POSIX compatibility is important. You will feel shell is limited if you don't learn how to use the system utilities with shell (or if your target system has common utilities missing). As an example of flexibility, you can use shell and system utilities in combination with CGI and a basic web server to send and receive text messages on an Android phone with termux. Similar to a KDE Connect or Apple's iMessage.
• usrbinbash 4y ago
  > What is the Shell paradigm? I would argue that it's line-oriented pipelines. Which python can do realitively well, by using the `subprocess` module. Here is an example including a https://porkmail.org/era/unix/award https://porkmail.org/era/unix/award (useless use of cat) finding all title lines in README.md and uppercasing them with `tr` import subprocess as sp cat = sp.Popen( ["cat", "README.md"], stdout=sp.PIPE, ) grep = sp.Popen( ["grep", "#"], stdin=cat.stdout, stdout=sp.PIPE, ) tr = sp.Popen( ["tr", "[:lower:]", "[:upper:]"], stdin=grep.stdout, stderr=sp.PIPE, stdout=sp.PIPE, ) out, err = tr.communicate() print(out.decode("utf-8"), err.decode("utf-8")) Is this more complicated than doing it in bash? Certainly. But on the other side of that coin its alot easier in python to do a complex regular expression (maybe depending on a command line argument) on one of those, using the result in an HTTP request via the `requests` module, packing the results into a digram rendered in PNG and sending it via email. Yes, that is a convoluted example, but it illustrates the point I am trying to make. Everything outlined could probably done in a bash script, but I am pretty certain it would be much harder, and much more difficult to maintain, than doing this in python. Bash is absolutely fine up to a point. And with enough effort, bash can do extremely complex things. But as soon as things get more complex than standard unix tools, I rather give up on the comfort of having specialiced syntax for pipes and filehandles, and write a few more lines handling those, if that means that I can do the more complex stuff easily using the rich module ecosystem of Python.
  • lbhdc 4y ago
    > But on the other side of that coin its alot easier in python to do a complex regular expression I am not sure I would agree. Sed fills this role quite nicely. cat README.md | grep # | tr '[:lower:] [:upper:]' | sed 's/something/something_else/'
    • asicsp 4y ago
      grep+tr can be done within sed too (or go with perl for more features and easier portability)
    • usrbinbash 4y ago
• jayd16 4y ago
  Eh, this is true but I dont think its because of the programming model of bash. I feel like this is conflating the *nix ecosystem with bash. If every programming language was configured by default and had access to standard unix tools with idiomatic bindings, Shell's advantages would be greatly reduced. You still get a scripting language with some neat tricks but I don't think I would reach for it nearly as often if other things were an option. And sure sure you can call any process from a language but the assumptions are different. No one wants to call a Java jar that has a dependency on the jq CLI app being available.
  • sophacles 4y ago
    This has been tried repeatedly - language idiomatic bindings tend to be clunky compared to (e.g.) a simple | pipeline or a couple of <() io redirections. Shell is a tool that turns out to be pretty good for some things, particularly composing functionality out of other programs and also doing system configuration/tuning stuff to tailor an environment for other programs. It's also really handy for automating tasks you find yourself repeating. Programming languages are a tool that are pretty good for other things - making new programs, tricky logic, making the most (or at least more than a shell script launching 1000s of new processes) efficient use of a computer. Trying to replace one with the other is not really useful - they have different jobs. Learning to use them in conjunction on the other hand... there's a lot of power in that. By comparison - javascript and html. They don't replace each other - yet they are both computer languages used in the same domain, and both have strengths and weaknesses. They have different jobs. And when you use them in conjunction you get something pretty darn powerful.
• floitsch 4y ago
  Shell scripting also inspired some choices (especially syntax) of the Toit language (toitlang.org). Clearly, it's for a different purpose, and there are some things that wouldn't work in a general-purpose language that isn't as focused on line-based string processing, but we are really happy with the things we took from bash.
• cryptonector 4y ago
  I've written a lot of shell scripts. I have my own best practices that work for me. I don't like it one bit. I mean, it's enjoyable to write shell scripts, it's just not enjoyable to deal with them long-term.
• throw10920 4y ago
  > My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different programming paradigm than languages like Perl, Python, whatever. This argument is essentially the same as "dynamic typing is just a different programming paradigm than static typing, and not intrinsically better or worse" - but to an even greater extent, because bash isn't really typed at all. To those who think that static (and optional/gradual) typing brings strong benefits with little downsides over dynamic typing and becomes increasingly important as the size of a program increases, bash is simply unacceptable for any non-trivial program. Other people (like yourself) that think that static typing isn't that important and "it's just a matter of preference" will be fine with an untyped language like bash. Unfortunately, it's really hard to find concrete, clear evidence that one typing paradigm is better than the other, so we can't really make a good argument for one or the other using science. However, I can say that you're conflating different traits of shell languages here. You say "dynamic scope, focus on line-oriented text, and pipelines" - but each of those are very different, and you're missing the most contested one (typing). Shell's untypedness is probably the biggest complaint about it, and the line-oriented text paradigm is really contentious, but most people don't care very much about the scoping, and lots of people like the pipelines feature. A shell language that was statically-typed, with clear scoping rules, non-cryptic syntax, structured data, and pipelines would likely be popular and relatively non-controversial.
• gtowey 4y ago
  As someone who has used a lot of shell over my career, I do love it as a utility and a programming paradigm. However the biggest issues I've had is that the code is really hard to test, error handling in shell isn't robust, and reusability with library type methods is not easy to organize or debug. Those are deal breakers for me when it comes to building any kind of non trivial system.
• krylon 4y ago
  I sometimes regret I never learned to "really" write shell scripts. I stumbled across Perl early on, and for anything more complex than canned command invocation(s) or a simple loop, I usually go for Perl. There is something to be said in favor of the shell being always available, but Perl is almost always available. FreeBSD does not have it base of the base system, but OpenBSD does, and most Linux distros do, too. But it is fun to connect a couple of simple commands via pipes and create something surprisingly complex. I don't do it all the time, but it happens.
• deleted 4y ago
  [deleted]
• AtlasBarfed 4y ago
  Hard Disagree. Bash programming: - no standard unit testing - how do you debug except with printlns? Fail. - each line usually takes a minimum of 10 minutes to debug unless you've done bash scripting for... ten years - basic constructs like the arg array are broken once you have special chars and spaces and want to pass those args to other commands. and UNICODE? Ha. - standard library is nil, you're dependent on a hodgepodge of possibly installed programs - there is no dependency resolution or auto-install of those programs or libraries or shell scripts. since it is so dependent on binary programs, that's a good thing, but also sucks for bash programmers - horrid rules on type conversions, horrid syntax, space-significant rules - as TFA shows, basic error checking and other conventions is horrid, yeah I want a crap 20 line header for everything - effective bash is a bag of tricks. Bag of tricks programming is shit. You need to do ANYTHING in it for parsing, etc? Copy paste in functions is basically the solution. - I'm not going to say interpreter errors are worse than C++ errors, but it's certainly not anything good. Honestly since even effing JAVA added a hashbang ability, I no longer need bash. Go ahead, write some bash autocompletion scripts in bash. Lord is that awful. Try writing something with a complex options / argument interface and detect/parse errors in the command line. Awful. Bash is basically software engineering from the 1970s, oh yeah, except take away the word "engineering". Because the language is actively opposed to anything that "engineering" would entail.
  • orwin 4y ago
    > - how do you debug except with printlns? Fail. With Trace. Which is talked about in TFA. By the way nobody use exclusively bash. When i worked for a cloud provider, it was basically 30% python(ansible), 30% perl, 5 to 10% bash, and a bit of other languages depending on the client needs (mostly java, but also Julia and R).
  • ronyclau 4y ago
    > - basic constructs like the arg array are broken once you have special chars and spaces and want to pass those args to other commands. and UNICODE? Ha. Any example with this? The following works reasonably well for me. args=(-a --b 'arg with space' "一 二 三") someprog "${args[@]}"
• wutbrodo 4y ago
  > I feel like those assertions are more socially-founded opinions than technically-backed arguments You think the complaints about rickety, unintuitive syntax are "socially founded"? I can't think of another language that has so many pointless syntax issues every time I revisit it. I haven't seen a line of Scheme in over a decade, and I'm still fairly sure I could write a simple if condition with less likelihood of getting it wrong than Bash. I came at it from the other end, writing complex shell scripts for years because of the intuition that python would be overkill. But there was a moment when I realized how irrational this was: shell languages are enough of a garbage fire that Python was trivially the better choice for my scripts the minute flow control enters the picture.
• kazinator 4y ago
  > with it's dynamic scope Bash has dynamic scope with its local variables. The standard POSIX language has only global variables: one pervasive scope.
• TristanBall 4y ago
  Aye.. I've been saying for years that shell scripting is how I meditate, and I'm only mostly joking Shell quoting though, Aieeee... I find I have to shift gears quite substantially moving from shell or powershell to anything else... "I'll just pipe the output of this function into.. oh, right"
• arendtio 4y ago
  The biggest Issue is that error handling is completely broken in POSIX shell scripting (including Bash). Even errexit doesn't work as any normal language would implement it (One could say it is broken by design). So if you don't care about error cases everything is fine, but if you do, it gets ugly really fast. And that is the reason why other languages are probably be better suited if you want to write something bigger that 10 lines. However, I have to admit, I don't follow that advice myself...
  • stephan-cr 4y ago
    > The biggest Issue is that error handling is completely broken in POSIX shell scripting (including Bash). Even errexit doesn't work as any normal language would implement it (One could say it is broken by design). I guess you're referring to http://mywiki.wooledge.org/BashFAQ/105 http://mywiki.wooledge.org/BashFAQ/105. Got recently hit by these as well.
    • arendtio 4y ago
      Yes and my personal favorite: Functions can behave differently depending on, if they are being called from a conditional expression vs. from a normal context. Errexit has no effect if the function is called from a conditional expression.