15 ms·
Shells are two things
- cb321 4y agoThe proposed solution of an API with a thinly wrapped auto-generated CLI is not terrible. I have heard it is common within Google, for example. In the Python world, there are various solutions starting from https://github.com/ialbert/plac https://github.com/ialbert/plac or https://argh.readthedocs.io/en/latest/ https://argh.readthedocs.io/en/latest/ and moving on to https://github.com/pallets/click https://github.com/pallets/click or https://github.com/Lucretiel/autocommand https://github.com/Lucretiel/autocommand and probably N others. Personally, I prefer Nim to Python which has https://github.com/c-blake/cligen https://github.com/c-blake/cligen. As mentioned in https://github.com/c-blake/cligen/blob/master/MOTIVATION.md https://github.com/c-blake/cligen/blob/master/MOTIVATION.md, but not in the borretti article, the overhead of dispatch to a program in shell REPLs can also be thousands to millions of times higher than an API call.
- GauntletWizard 4y agoI have had great success creating DSLs using ipython, where you simply define a set of classes and functions on them, then use ipython.embed() (Before it was terminal.embed, so you can see where the experience is dated).
- tzhenghao 4y agoI've used click [1] a lot to build Python tooling scripts the past few years. Click usage is "sort of" similar to the author's proposed solution. There's also a small section here [2] that describes some of the issues covered in the article (in context of argparse). [1] - https://github.com/pallets/click https://github.com/pallets/click [2] - https://click.palletsprojects.com/en/8.1.x/why/#why-not-argparse https://click.palletsprojects.com/en/8.1.x/why/#why-not-argp...
- js8 4y agoI don't see a reason why Forth or Factor can't have both.
- nmz 4y agoCertainly factor/retroforth, Forth is too low level to be useful, but with support for quotations and streams, factor/(retro?) might be good, dealing with series a la rebol is also a good idea. Unfortunately all of those languages are kind of dead. The interesting thing is that pipes behave in an rpn like fashion, but standard behavior is prefix based, So a rebol like language should be the normal operation, and then it changes to rpn when you pipe, the only languages I see implementing this is rye, which is highly experimental. Powershell is also a really good option, at least for what OP wants, I like terseness. I can read bash code fine enough, until someone starts golfing, but that's true of all languages. python also has xonsh.
- Galanwe 4y agoI don't think the author realized that the `configure` script at the center of his rant is auto-generated by `autoconf`, and not written by a human... The source is a straightfoward `configure.ac` of 75 commented lines of code (13 with comments removed)
- kelnos 4y agoI felt like the author did realize that, and the "written by the devil" comment was a direct reference to autoconf (which certainly qualifies as a devil). Who knows, though.
- segfaultbuserr 4y agozetalyrae also correctly pointed out that much of the text is "handwritten but transcluded". Most of the code snippets that appeared in autoconf was originally written by hand and stored in reusable modules, including 100+ tests in autoconf. So overall the volume of code is still huge. I guess it at least demonstrated the power of modularity. Almost nobody can understand it, but it still works...
- zetalyrae 4y agoBash is not an acceptable compilation target. And much of the text is handwritten but transcluded.
- Galanwe 4y ago> Bash is not an acceptable compilation target The whole point of autotools is to make sure that your project can run on the largest possible set of unix-like operating systems. `sh` is actually the very best compilation target for that, looking back in time, whatever other language that could have been chosen (most likely perl) would have been a huge mistake.
- zetalyrae 4y agoThe alternative is a C program querying the filesystem (or, better yet, a database and a daemon to prevent duplicate ./configure work).
- travisd 4y agoOil is a decently interesting “competitor” in this space. They’ve also acknowledged the dual nature of shell as REPL and programming language. https://www.oilshell.org/blog/2021/01/why-a-new-shell.html https://www.oilshell.org/blog/2021/01/why-a-new-shell.html
- SomeoneOnTheWeb 4y agoA very hard thing to do though is the requirement of generics. If everything is typed, you may have functions that take an argument and produce another value of the same type for instance. And generics are really, really hard to implement correctly.
- forgotpwd16 4y agoOil is first thing that came to mind reading article. Interestingly the submitted article and the one you linked open in similar way with two bullet points saying shell is made of two things.
- agumonkey 4y agonever understood the drive for oil, seems like a lot of effort in the wrong direction
- H4ZB7 4y agoexactly this.
- Arcuru 4y agoI've looked into Oil a bit, and I think Andy has the right idea for how to go about updating the legacy scripts. I've written a few local scripts in Oil and it's a definite upgrade from Bash. If anyone wants to understand Oil a bit better, I found that the best intro is actually the Github wiki: https://github.com/oilshell/oil/wiki https://github.com/oilshell/oil/wiki
- chasil 4y agoThis is what you need to understand about the POSIX shell: $ rpm -ql dash /bin/dash /usr/share/doc/dash-0.5.10.2 /usr/share/doc/dash-0.5.10.2/COPYING /usr/share/doc/dash-0.5.10.2/ChangeLog /usr/share/man/man1/dash.1.gz $ ll /bin/dash -rwxr-xr-x. 1 root root 113536 Nov 5 2018 /bin/dash Embedded systems are going to give you that much room, and no more.
- Snarwin 4y agoA number of command-line programs are actually just wrappers around a more strongly-typed library API. For example, curl and libcurl, or ffmpeg and a whole collection of audio/video libraries. [1] Of course, in the POSIX world, "library API" means "C API", so if you want to write nice-looking Python code like in the article, there is still some more work to do. [1] https://trac.ffmpeg.org/wiki/Using%20libav https://trac.ffmpeg.org/wiki/Using%20libav*
- charcircuit 4y agoffmpeg is a bad example as libav is a library specialized towards making ffmpeg. It's not a generic audio/video manipulation library.
- account42 4y agoWhere does the requirement for it being a generic library come from? The point is that a lot of things that are available as command-line utilities for shell scripts already have a C API available as well. And libav* is used by a lot more than /usr/bin/ffmpeg
- thwarted 4y agoOf these, 79 are over one thousand lines of text, the largest being /usr/share/libtool/configure, a 16,394-line shell script the devil wrote² ² The fact that the line count is just a hair off 2¹⁶ surely has some sinister, esoteric significance A hair off‽ That's two base-two orders of magnitude from 2¹⁶, being 10 more than 2¹⁴.
- alxmng 4y agoI disagree. I think nushell does both quite well. But it requires eschewing backwards-compatibility and interoperability with sh/bash
- DangitBobby 4y agoI was just thinking the same. There's not really any reason a shell can't have terse syntax for quick command things and explicit, careful syntax for things that require it. And if python had a pipe operator, you could basically do open('file.txt') |> grep(?,"lasagna") |> set |> sorted Which is really pretty good. It's not at all hard to imagine a Python that was designed to be good in the shell. They just didn't try.
- eviks 4y agoThere is a python-based shell https://xon.sh https://xon.sh with a plugin that allows piping :) https://github.com/anki-code/xontrib-pipeliner https://github.com/anki-code/xontrib-pipeliner
- GoblinSlayer 4y agoShould it really be as terse as posix shell? If you think before you type, it's not a high-frequency REPL anymore.
- TeMPOraL 4y agoThe main high-frequency bits are filesystem navigation and incrementally building pipes. cd foo/<tab> # oh, too many options <ret> ls -la cd bar/baz git status cat quux | grep xckd cat quux | grep -i xckd cat quux | grep -i xkcd cat quux | grep -i xkcd | xargs ... etc. The trial-and-error command refining is arguably what REPLs are for. It's the filesystem navigation that are the odd part. (INB4: don't give me the "useless use of cat", it factors out better like this and everyone does it anyway.)
- GoblinSlayer 4y agoThat's what I mean: it's a high-frequency REPL if you don't think before you type, i.e. trial and error.
- marklgr 4y agoFWIW, if you just like the filename on the left, you still can save forks with: `<quux grep xkcd`
- Joker_vD 4y agoIt's like trying to save money on matches.
- hnlmorg 4y agoI’ve written about this problem previously and it is a problem I’m attempting to solve with my own shell (along with the added problem that most shells still have a UI from the 80s). I do think it is possible to find a sane middle ground however it’s not an easy task negotiating the right compromises to make. Plus any shell that does achieve this wouldn’t be POSIX compliant, which then automatically breaks support with sh/Bash etc. My shell https://murex.rocks https://murex.rocks
- julienfr112 4y agoWhat is your take on Microsoft's PowerShell ?
- deafpolygon 4y ago> And you can’t do both. You can’t be explicit and implicit, you can’t be terse and readable, you can’t be flexible and robust. PowerShell does quite well in this case.
- MrGilbert 4y agoYep, totally agree. Never thought PowerShell would become my go-to utility knife, but here we go.
- deafpolygon 4y agoSame. Story time: I've been a diehard bash/zsh fan for the better part of the last two decades (I was a regular user of it since 1999 so that's 24 years now). I daily-drove some form of Linux since 1999 up until ~2010 when I switched to MacOS (zsh). I only switched to using Windows as my daily driver for ~2 1/2 years ago. Around 2 years ago, I had a script in python that I wrote to process my directory of media files based on the directory name and looked for a subtitle file and pulled that down. For fun, it was a simple enough script that I re-wrote it in bash (WSL2). Six months later, I needed to make a change and was surprised at how much I couldn't read it. I just forgot. That was OK, I had a boring weekend and rewrote the whole thing in PowerShell. A few things I noticed: - It was more LOC owing mostly to brackets, etc, but I'm over the whole "less LOC is better" mentality. - It was faster - but this is probably the result of WSL2, but also, I didn't have to call out to other CLI utilities as much. You can do just about everything in PowerShell and call out to .NET for anything more complex. - It was far more readable. Having to parse while [ ]; do ... done; is harder than ForEach-Object ( ) { } - Passing things around as an object is a very pleasant way of scripting (Python could do the same, but it feels more "Programming Language" than "Scripting" at this point) - Calling an CLI utility is just the same as in Bash (Unless it has spaces, then you just do & "Path to CLI.exe") I had to make a change recently and was pleasantly surprised at how much I could remember the layout, functions, etc. I also tried to adhere to the MS-isms like using a Verb-Noun for my functions where it makes sense, and breaking things up in modules. I don't absolutely love PowerShell, but it is pleasant to use and pulling in .NET stuff is just a line of code away. Now I use it directly as a daily driver, with Windows Terminal - which is also surprisingly pleasant to use.
- dsab 4y ago> Historically, operating systems that have fanciful structured interfaces between programs have been left in the dust by Unix, because Unix maximizes flexibility by favoring the lowest-common-denominator interface, which is the string. I am curious for examples for such a operating systems and their program interfaces
- mannschott 4y agoz/OS, OpenVMS and OS/400 spring to mind.
- eviks 4y agoIn reality there isn't much contradiction between the two goals, you "just" need a bit more design thought instead of getting stuck in the ugliness of POSIX. Author's first example cat beef.txt | grep "lasagna" | sort -n | uniq is already possible literally as written in Xonsh (a Python-based shell) with a pipeliner plugin https://xon.sh/ https://xon.sh/ https://github.com/anki-code/xontrib-pipeliner https://github.com/anki-code/xontrib-pipeliner
- H4ZB7 4y agoit doesnt solve the problem give me functions that output typed data. none of this string parsing shit. the sole fact that they have syntax to call bash: $() makes me not want to use this ever. this is why every single effort like this is wasted. i used to get excited by these projects and after about 2 of them i realized it's always a bunch of crap duct taped together, and also bends over to be compatible with un*x and so everything is ruined. they're also often so poorly implemented, like with those early fancy python shells they'd go beserk if one command you execute goes into a time consuming process (or infinite loop) whereas why the hell can i not just run another command below while waiting for the one above to complete? this is the thought i had in 2008. i could have made something better out of my ass. really, there's no other way to put it. here's another thing: a language console should operate on semantic objects, not text. as in, you move objects around to program in it. editing code by typing one character at a time is too slow. no amount of vim or emacs fixes this. in fact, we can just scrap those un*x garbage (along with the rest of un*x) while we're at it, when implementing an OS that works on actual meaningful data instead of text, and semantic objects instead of a language represented by text.
- ratboy666 4y agoBut, bash has this... (job control) : root@llana fred #; dd if=/dev/sda of=/dev/null ^Z [1]+ Stopped dd if=/dev/sda of=/dev/null : root@llana fred #; bg [1]+ dd if=/dev/sda of=/dev/null & : root@llana fred #; jobs [1]+ Running dd if=/dev/sda of=/dev/null &
- hoelle 4y agoF# is readable and terse. F#'s REPL is really fun to work in, too. The significant whitespace makes it a bit awkward in shell mode, but why do that when editing a script in VS Code, Emacs, or even screen + nano is far more convenient.
- aerzen 4y agoLast year I was burdened with similar idea, which I summarized as "Shells need strict typing". https://aljaz.murerzen.eu/posts/2022-11-11-shells-need-typing/ https://aljaz.murerzen.eu/posts/2022-11-11-shells-need-typin...
- Annatar 4y ago[dead]
- jwilk 4y agoThe Python code is unnecessarily verbose and doesn't actually work: • No need to involve Path objects; open("beef.txt") works. • The first argument for filter() should be the filter, not the iterable. • The stream itself is iterable, so no need to call the readlines() method. • The first argument for re.match() should be the pattern, not the string. • You probably wanted re.search(), not re.match(). Or, since this is a fixed string, use the ‘in’ operator. • ‘is not None’ is not necessary for checking if the match was successful. • It's ‘reversed’, not ‘reverse’. • All that sorting and reversing is unnecessary when you're going to put it in a set anyway.
- amelius 4y agoCould you post a rewritten version?
- alpaca128 4y agoSomething like this with open("beef.txt") as file: lines = [line for line in file if "lasagna" in line] print(set(lines))
- forgotpwd16 4y agoIn author's defense, code is apparently meant to exemplify the differences between shell and general programming languages mentioned in bullet points. So working doesn't matter and isn't even mentioned that is in Python.
- BoppreH 4y agoIt's also not performing the numeric sort ('-n'), which is arguably the hardest part because it's not a Python built-in. A bad example all around. Here's my attempt[1]: import re lines = list(set(line for line in open('beef.txt') if 'lasagna' in line)) lines.sort(key=lambda line: int(re.match('\d*', line)[0] or 0)) for line in lines: print(line, end='') Python is especially bad at this task for a number of reasons: - Iterating over the lines yields strings ending with newlines. If you naively loop and print the sorted lines, `print` will add a second newline, resulting in every other line being empty. Hence the `end=''`. - Python's str has many helper functions, but none for numeric sort, so you need an import, a regex, and a fallback value. - It's very imperative with a big mutation in the middle. Using `sorted(lines, key=...)` and `print(''.join(lines))` would be simpler and safer, but each would double the memory usage for large inputs. - Filters using list comprehensions ('x for x in y if z in x') are idiomatic and readable but very verbose. - The concepts are a lot more complicated. Here we have file handles, magic iterators, first-class-functions, sets, generators, lists, keyword arguments, regexes, overloaded indexes, etc. A beginner would blindly copy and run, and have no chance to fix any issues that appear. - Because Python has almost no static analysis (undefined names are runtime errors!), you'll have to find most errors by running your code. For a quick shell script this usually mean running over the real inputs, giving a horribly slow edit-compile-run loop and be potentially dangerous. [1]: https://gist.github.com/boppreh/3cf5489da92d85f11e152739320d147e https://gist.github.com/boppreh/3cf5489da92d85f11e152739320d...
- t43562 4y agoShells need to be good at composing programs together, languages compose libraries. Pipes and output redirection, running things in parallel and waiting till they end - for some reasons these things suck in some languages compared to what the most basic shell can provide. Shells let you do unexpected, unintended integrations. I think languages could do better without breaking
- dolmen 4y agoPrograms called from the shell for integration are not all consistent and that's another reason why shell programmins is not easy. Exit status are inconsistent. Stderr output: is it a warning (ignorable) or an error?
- t43562 4y agoStderr output does not indicate that a program has failed. It's just a communication channel to the user that's not affected when you pipe or redirect stdout. So useful for programs where to want to make sure that data doesn't get mixed up with messages to the user. exit codes are the only true way to tell if some process failed and programs should set them properly.
- eru 4y agoThough to be honest, eg Bash isn't all that great at composing programs, too. Especially when you want to handle failure. Mostly shells are good at scaling downward, ie having relatively little syntactic (and brain) overhead for very simple things. Especially when you are ok to handle any failures manually.
- throwawaaarrgh 4y agoYes! Shells excel at integrating loosely-coupled interfaces. Languages seem to suck at it.
- chasil 4y agoSpecific shells. "Bash is like the neighbor who seems nice and sweet, but maybe they don’t have a life you’d want to have personally. They have some pretty serious character flaws and obviously make mistakes, but in the end they are a nice person who tries hard to make sure the people that depend on them are doing ok. "Batch [cmd.exe] is the guy who looks like that neighbor, but then turns out to be a serial killer." https://blog.nullspace.io/batch.html https://blog.nullspace.io/batch.html
- benrutter 4y agoThis is well made case - but I'm not sure I buy the central argument. Within some basic limits, I don't think terseness and readability have the contradiction made out here, because in programming we have abstraction, which gives us both. To take the example command that's given: beef.txt | grep "lasagna" | sort -n | uniq Sure, writing the logic out for this in something like python straight out the bat with only the standard library might look messy, but with one basic convenience function it could quickly be: search(for='lasagna', in='beef.txt', clear_duplicates=False).sorted() Obviously you have to write the function in the first place, but I'd say if you're doing something like this often, it's easily worth spending that 2 minutes. And if you're not doing this often, you'll have a faster time writing more code, but keeping less heavy lifting of "how does bash pipe together" in your head. I shared a project here a few weeks ago experimenting with what my dream shell might look like, what surprised me more than anything else, was how easy writing a repl environment actually is. I put a scrappy one together as one person in a few hours, so I don't understand why as developers we've reached general language models before being able to make a powerful, but new-user friendly shell. Also, completely unrelated note, but posix only allows passing back strings - but isn't this true of web apis too which we use all the time? How come no json as a standard passback from programs? Shameless plug for the project I mentioned earlier: https://github.com/benrutter/clamshell https://github.com/benrutter/clamshell
- eru 4y agoYour shell example beef.txt | grep "lasagna" | sort -n | uniq is a lot more compositional than your Python-style example: search(for='lasagna', in='beef.txt', clear_duplicates=False).sorted() The former uses a few general purpose functions, the latter seems to use bigger building blocks that aren't as universal. You could more faithfully mirror the bash example in Haskell, if you wanted to, if you wrote the necessary library functions: cat "beef.txt" | grep "lasagna" | sortBy numeric | uniq This would be actual valid Haskell syntax.
- marcosdumay 4y ago> Your shell example ... is a lot more compositional than your Python-style example Yes, the Python equivalent would be: lines("beef.txt").filter(lambda(x): x.contains("lasagna")).sort(type="natural").remove_duplicates() Of course, all of those things would need to be generators. Python accepts that kind of code, but most "Python developers" wouldn't think on it. Haskell is a much more natural fit for replacing Bash. (By the way, Haskell has at least one Bash replacement library that I remember, but it's hard to find).
- BulgarianIdiot 4y agoThat’s one thing.
- falcor84 4y agoThat Python example is broken in all kinds of ways, and is overly verbose. I would go with something like the following, which does spark some joy in me: matches = {l for l in open("beef.txt") if "lasagna" in l} print(*sorted(matches, reverse=True), sep="") On And on the same note, the shell command is weird too, and should probably be: grep "lasagna" beef.txt | sort -nur So the shell command is still definitely simpler for this case, but for anything more complex, I'd go with Python. EDIT: I put back the `n` flag, in case the lines start with numbers.
- sigg3 4y agoSort has -u too.
- benj111 4y agoI wonder if that's a part of the issue. There's almost no fixed grammar. Especially the basic things, many programs do the thing as an option. Imagine if python functions /classes had their own add implementation. You aren't just looking for foo + bar. You've also got baz(foo, bar, add), biz(add, foo, bar), buz(foo, bar).add(). Some people will use one. Some another, they'll be subtle incompatilities, etc.
- BoppreH 4y agoThat's much better, but like the original implementation it's still missing the -n flag for sort, which extracts numbers at the beginning of lines and sorts them numerically instead of lexicographically.
- falcor84 4y agoOh, I always thought it only works on numbers, and am very pleasantly surprised to learn it works on numeric prefixes, thank you! EDIT: After playing around with the python approach, I arrived at this, which is definitely more verbose, but arguably, more maintainable: import re NUMERIC_PREFIX_RE = re.compile(r"^(\d+.?\d*)") def numeric_descending(line): if m := NUMERIC_PREFIX_RE.match(line): return -float(m.group()) return float("inf") matches = {l for l in open("beef.txt") if "lasagna" in l} print(*sorted(matches, key=numeric_descending), sep="")
- HeavyStorm 4y agoThe author must familiarize himself with PowerShell. Aside from the terrible, terrible idea of using verbs as prefixes, Power Shell is a great combination between a programming language and a shell.
- zwischenzug 4y agoI remember joking that PowerShell is a properly architected shell, which is why it was so uninspiring to use. I still use Bourne-derived shells because they're just so damn human.
- btschaegg 4y agoWell it might work as a joke, but I have to say that I had more than enough experiences with Powershell that drastically disprove that notion in my opinion. That is not to say that bash and consorts are much better, but e.g. a shell that decides to arbitrarily violate its normal error handling logic based on internal voodoo heuristics about if it's running in a console or not [1] is pretty much the total antithesis of sanely architected in my book. As in: Architecture must be the reason someone thought this insane scheme has any merit. And arbitrarily starting to throw exceptions whenever a subprocess dares to write to stderr, especially if the shell doesn't do this in normal operation, doesn't get any other label than "insane" from me. [1]: https://stackoverflow.com/a/57272097 https://stackoverflow.com/a/57272097
- zwischenzug 4y agoFair enough, I only used PS for a day or two.
- AnIdiotOnTheNet 4y agoAlso the terrible terrible idea of using common unix shell command names as aliases for PowerShell commands that are similar but don't quite work the same way and take entirely different parameters. ...and that time period where Invoke-WebRequest used IE for parsing by default. But other than those flaws I agree that it is really great.
- jimwhite42 4y agoMaybe I'm missing something, but the example at the bottom, you can do just as much type checking with the terse command-line syntax as with what he calls the 'typed interface'.
- OJFord 4y agoThe conclusion is eerily close to using a programming language, Python say, having a natural library API for that, and then a small `if __name__ == '__main__'` that parses args (with argoarse or docopt or whatever) and calls it. The problem of course is so now you want your script to 'pipe' to pandoc, say, and pandoc even if implemented like this might not have chosen Python but Ruby, say. So you have to (or ~have to, most reasonable way is to) use the CLI to interface with it anyway.
- Joker_vD 4y ago> You can’t be explicit and implicit, you can’t be terse and readable, you can’t be flexible and robust. Unless it's Perl!.. although some would claim it's unreadable and not robust at all.
- crabbone 4y agoOP's Python is... exaggerated? (there's too much noise in it, was that deliberate, as an illustration, or just bad Python?), but so is their use of nomenclature... Shell is not and doesn't need to be a REPL, but, I think, OP is using that word the same way some use REST to mean any kind of HTTP interface, i.e. just a common misconception about what that word means. Also, OP makes claims about "readable syntax" and "static types", both of which are sorts of common misconceptions. Like... English is readable to you, right? You had little problem reading this after all. All types in Shell are static when you look at the script, and, actually are much more well-behaved, because all of them are string. Both at definition time and at run time. ---- In general, I just don't agree with OP's analysis because, I think, OP interprets status quo as the only possible way things may be. But the reality is that we have a bunch of popular but atrocious shells and a handful of popular but atrocious languages that aren't usually used in shells. The fact that the two groups don't seem to have a lot of overlap is just an accident. Now, take for example SQL. It's fine to use it interactively, in a way more similar to how system shells are used, but also to write programs in it. It doesn't have modules or some other organizational features other languages have, but do you think it's somehow impossible in principle? -- Of course not. Had there been a will, a new SQL standard would add modules to SQL etc. There's nothing impossible about it. Similarly, had there been a will, SQL could've been used in the same capacity Shell is used today. No reason it couldn't. We use a decrepit UNIX Shell because of standard and tradition. Or we use the circus of PowerShell because that's the only reliably available tool on that OS. It doesn't have to be that way. It's just that we have a sample size of two, and both are defective beyond belief.
- samsquire 4y agoI've been experimenting with alternative pipeline syntaxes. One example is using Graphviz graph files as an execution plan for software deployments. You can see a graph here: https://github.com/samsquire/mazzle/blob/master/docs/architecture.tb.png https://github.com/samsquire/mazzle/blob/master/docs/archite... I wrote code with networkx to parse this graph dot file and then follow the plan topologically, spinning up packer, terraform, bash scripts, compilation and so on. It also executes graph nodes in parallel where it can. I talk about it here https://devops-pipeline.com/ https://devops-pipeline.com/ My problem with bash pipelines is passing data along and referring to previous steps' data. You kind of need to enrich the data that goes along the pipeline. In my use case, I need to pass along outputs from previous instantiations.
- shp0ngle 4y agoDidn’t PowerShell on Windows try this and basically failed? Well some people really like PowerShell but I didn’t see the concepts move over to UNIX side
- crabbone 4y agoOne of the reasons why PoverShell approach cannot succeed now is that the system interface produces exit codes and unstructured text output. So, no matter how hard PoverShell tries to pretend that there's structure or that there's anything but strings and exit codes, it's all fake, and it starts breaking up very quickly, especially if you are used to work with programs whose interface is just that: exit codes and unstructured / randomly / uniquely structured output. Had the system offered programs an interface for outputting structured information, had it worked better at standardizing and unifying the way programs process structured input... PoverShell as a concept would have more ground to stand on. Right now, in the UNIX world, it is an even worse option than Python. So, there's no motivation to accept it. At least I haven't met a single system programmer who would have wanted to have PowerShell on UNIX. Only people who do that are Windows users who have some prior experience and who simply don't know much about the system they have for whatever reason ended up using.
- getoffmyyawn 4y agoMaybe I'm missing something, but it has always seemed to be that people who complain about shell scripts do not approach shell scripting the way they approach any other programming language. By this I mean that they seem to actively resist the idea that they need to learn the language to get the most out of it. One of my early jobs involved maintaining a pile of shell scripts that were part of automating a publishing system pipeline. So I picked up a couple of suggested books and learned everything I could about shell scripting. I quickly grew to enjoy working with shell scripts for automation. Yes, there are quirks and oddities in shell scripting but the language is very well document and good practices and style guides have been around for ages. Would you expect to understand any language without studying it first?
- teknopaul 4y agoThis is so true. Torvalds wrote git in shell. You can do great things. Provided... a) agreed, RTFM, (man bash is not long) b) you dont try to do things shells shouldn't do. c) for things you shouldn't do, you write a clean stable cli interface in some other language. Often you can do c) in bash temporarily and port only if/when it gets complicated. (That's what git did)
- masklinn 4y ago> Torvalds wrote git in shell. Did he? Because that’s definitely not what the initial revision of git that’s in the repo shows. It has 1kloc of C, a readme, and a makefile. I struggle to think how you’d even get to the wonky tree format if you used the shell as your implementation language. The first shell scripts landed a few weeks later, in 839a7a06f35bf8cd563a41d6db97f453ab108129 and were convenience wrappers to facilitate interacting with contributors (a wrapper around `merge` to merge a single file, a trivial wrapper around `fsck-cache` to prune, and finally a somewhat more involved original `git pull` which used `rsync` to retrieve all of the remote’s object store then would merge its HEAD with yours). The next two scripts were literally called “example scripts” in their commit messages, and respectively for creating a signed tag and applying a patch.
- 4y ago
- Koshkin 4y ago> stringly-typed Love this
- senzilla 4y agoPOSIX shells does indeed have many problems, including lack of data types and various security limitations. I like the point OP is making that shells should focus on being interactive, but not so scriptable. For scripting, it's better to resort to a simpler solution that doesn't even need complexities like parsing or data types. In a Unix environment all you need is the ability to compose commands together. This is what execline is for: https://skarnet.org/software/execline/ https://skarnet.org/software/execline/
- hnlmorg 4y agoThere are a few modern shells that do include data types and fix a number of security issues that more traditional POSIX-like shells (Bash isn't technically POSIX) suffer from.
- snag 4y ago> cat beef.txt | grep "lasagna" What's wrong with > grep "lasagna" beef.txt ?
- rascul 4y agosort also has -u, so there's two unnecessary pipes. The whole thing could be written: grep lasagna beef.txt | sort -u Some people add extra pipes and cat for what they claim is readability. I don't agree that cat and extra pipes is more readable, but to each their own.
- madacol 4y agoIt has a similar learning curve, but less functionality
- drewg123 4y agoLibtool.. yeah, that's evil. Was it ever really helpful? At a former job we supported Linux, AIX, FreeBSD, Solaris, Mac OS, and Windows. One of the best things I did was rip out libtool, as it was totally unneeded. And if we didn't need it when supporting all those disparate platforms, I really cannot figure out what purpose it serves.
- account42 4y agoIt's the premier porting tool for making cross-compilation harder. I think the main raison d'etre was automatically linking transitive dependencies for static libraries so that builds on system with or without dynamic libraries could work more or less the same. E.g. if you statically link e.g. Freetype than wether you also need to link zlib, libz2 and a bunch of other stuff depends on how Freetype was built but the .a static library contains no information about those dependencies since it is just an archive of object files with a symbol index. libtool records these dependencies in .la files so you don't have to guess. It does have many questionable decisions like silently dropping (some) compiler flags it doesn't know about. I wish I never had to deal with it.
- codazoda 4y agoIf you’re interested in building your own shell, I did that in Go recently. https://github.com/codazoda/goshell https://github.com/codazoda/goshell
- nurspouse 4y agoPsst... xonsh! Relatively terse, and quite readable scripts (they are mostly Python!)
- photochemsyn 4y agoIf you wanted to do this with python: cat beef.txt | grep "lasagna" | sort -n | uniq The correct way would seem to be to write four python programs, named pcat, pgrep, psort, and puniq, and then pipe the output of one program to the other program. The only advantage would be that it should be able to work in any operating system with a python environment. Ideally each program would have the same functionality and resilience as the original bash programs. The argparse module (for accepting the same options at the command line as the origals) and pathlib module (for portability to Windows as well as Linux and Mac) would be useful. Now, is it really worth the trouble of doing this? Possibly, at least as part of an educational Python program. Implementing all of grep's functionality would be a big job, however.
- marcosdumay 4y agoYou can fork and pipe between python functions perfectly well. It just needs a lot of boilerplate (that you can easily abstract away). Also, you will get most of the problems of piping in Bash imported into Python. But the real question is why are you piping those? Almost always, piping is not the end-goal, and this is certainly one of those cases.
- aidenn0 4y agoIt's my opinion that the author is wrong. A lot of the same qualities that make a good REPL for interacting with the system make a good programming language for interacting with the system. C shell was designed solely for interactive use and is a worse REPL experience for it. Not being able to e.g. redirect compound expressions matters for quick one-liners at least as much as it does for writing scripts. I think the author has an underlying assumption that untyped languages are worse for writing programs, and that's what they mean by "terseness" versus "verbosity". If that's the case then they should probably state that outright. I'm only unsure about this assumption because their pandoc example is non-sensical if that's what they are getting at. There is no fundamental reason that: pandoc -t latex \ -f markdown \ --pdf-engine=xelatex \ --table-of-contents \ --toc-depth=2 \ --resource-path=. \ --standalone \ input.md \ -o output.pdf cannot be statically checked any more or less than Pandoc( input = (Path("input.md"), MD), output = (Path("output.pdf"), PDF), pdfEngine = XELATEX, resourcePath=Path("."), completenessMode=STANDALONE, ) And in fact it's trivial to write a mechanical transformation between the two.
- lisper 4y ago> The fundamental problem of shells is they are required to be two things. > A high-frequency REPL, which requires terseness, short command names, little to no syntax, implicit rather than explicit, so as to minimize the duration of REPL cycles. > A programming language, which requires readable and maintainable syntax, static types, modules, visibility, declarations, explicit configuration rather than implicit conventions. > And you can’t do both. You can’t be explicit and implicit, you can’t be terse and readable, you can’t be flexible and robust. Of course you can. Lisp is an existence proof. You get the best of both worlds by starting with a simple dynamic language and adding declarations and type inference on top in a backwards-compatible transparent way. Just because no one does this doesn't mean it can't be done.
- erichocean 4y ago> Lisp is an existence proof So is Forth. Forth is the first "shell" that ever ends up on a new piece of computing hardware. You get a REPL + programming environment from scratch by implementing just 28 words in hand-written assembly.[0] Most programmers can get it running in just a few days, even with no prior knowledge. (Lisp is also famously small to implement.) > Just because no one does this doesn't mean it can't be done. 100% [0] eForth, described in 53 pages here: http://www.exemark.com/FORTH/eForthOverviewv5.pdf http://www.exemark.com/FORTH/eForthOverviewv5.pdf
- H4ZB7 4y agojust like every component of a fully un*x braindamaged OS, the shell only gives you speed where it increase the likelihood of errors (and security vulnerabilities, lots of them). every single thing you can do in the shell is a vuln by default until you write a verbose workaround. since nobody does that, in practice the shell is just insecure and fast(ish) user experience. pl design is a big field and some C/java/sh programmer's opinion doesn't nearly represent it. even python is better than sh in every way; merely showing that it doesn't have a bunch of useful functions in scope by default (duh, it wasn't geared for that use case) doesn't prove that sh is fundamentally a good language for a console (its not).
- lanstin 4y ago1. Shells are a universal interface from orchestration layers in higher level languages to “do stuff on a Unix node”. Your puppet scripts, dockerfiles, ansible modules, all will benefit from understanding shell. 2. The bash chapter in the Architecture of Open Source Applications is really interesting, one finishes it feeling sympathy for the devs. http://www.aosabook.org/en/bash.html http://www.aosabook.org/en/bash.html