10 ms·
Unix shell programming: the next 50 years
- mastrsushi 5y agoThe biggest problem with Python is there's always some divide between it's environment and shell. I don't think Perl had this problem as accessing the shell was built into the language. There needs to be a modern language that looks friendly, is compatible with shell, and is apart of the shells environment. I am one of many people who don't really care for shell, it's old, limited, and just plain ugly. Unfortunately it's still the best way to interact with Unix, so it should be built upon to keep compatibility. Wouldn't it be nice to write shell scripts with objects and data structures?
- rwbhn 5y ago> Wouldn't it be nice to write shell scripts with objects and data structures? That's PowerShell - https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_pipelines?view=powershell-7.1 https://docs.microsoft.com/en-us/powershell/module/microsoft...
- ablyveiled 5y agoBegs the question- is it nice?
- tyingq 5y agoIt is nice, but they chose to leave some odd legacy stuff around. For example, you can make a Win32 GUI app, but the console window still shows. So, you can hide it, but it flashes briefly anyway. Also, there's weird overlap of features, sometimes because it's so easy to call out to dot net. Like, there's Start-Job, but there's also Runspaces and System.Threading.Thread. Makes it hard to find the "right way" to do something.
- majkinetor 5y agoStart-Job is not the same as Runspaces - one uses processes, other one threads. Those who need performant option will know what to use.
- tyingq 5y agoThere's also Start-ThreadJob
- spaetzleesser 5y agoThe idea is very good but they f..ed up the language with a lot of weird quirks. Also, the naming convention with "Get-" first doesn't scale. Auto complete is basically useless when thousands of cmdlets start with "get". And Powershell is super slow when you deal with a lot of data to the point of being unusable. Two years ago I worked on a project where we used Powershell a lot. It worked OK but I think next time around I would use Python.
- cutler 5y agoWhich idea exactly was good? PowerShell, to me, is just a typical Microsoft product full of bloat and redundancy. Microsoft couldn't produce a concise language if their existence depended on it.
- spaetzleesser 5y agoHaving a scripting language deal with objects seems a good idea. Also, easy interop with the .NET framework is good. It would have been much better if they had used a variation of C= (which is a pretty good language)
- da39a3ee 5y agoMicrosoft produced Typescript and Lean. Are those full of bloat and redundancy?
- nextaccountic 5y agoAlso nushell https://github.com/nushell/nushell https://github.com/nushell/nushell - it seems to be less verbose than powershell
- jhgb 5y agoOr Scheme Shell - https://scsh.net/about/about.html https://scsh.net/about/about.html
- majkinetor 5y agoSeems abandoned.
- MathMonkeyMan 5y agostill works
- dbtc 5y agoIs `ruby` any closer? or eshell?
- eikenberry 5y ago> Wouldn't it be nice to write shell scripts with objects and data structures? IMO no, it wouldn't. The simplicity and flexibility of plain text piping beats out the more structured approaches. Sort of reminiscent of how HTTP ate the lunch of most all other RPC protocols (eg. CORBA) even though it wasn't even designed to be an RPC.
- bitwize 5y agoBut you only have the option of sending unstructured, untyped bytes over the pipe, which means you waste CPU cycles parsing and validating at each stage. And all data has a type, so you may as well have strong typing in your shell. HTTP ate everybody else's lunch mainly because corporate firewalls closed off competing protocols' ports but kept port 80 open.
- drran 5y agoMaybe you use a different kind of pipe. My pipes allow me to send binary, typed data, e.g. protobuf. For example: http://temk.github.io/protobuf-utils/ http://temk.github.io/protobuf-utils/ . Do you know a good set of command line utilities to work with binary typed data?
- bitwize 5y agoYou're still sending untyped streams of bytes through the pipes, which must be marshalled and unmarshalled at the endpoints. PowerShell, by contrast, operates directly on objects with strongly typed fields.
- drran 5y agoData are just bytes. You can serialize data into an encoding, or you can send a raw memory snippet. It's implementation detail. Binary data is not a problem for UNIX pipes. Perl has powerful pack/unpack methods to efficiently work with binary data. Do you have tools to use this feature? If not, who will write them? Who will define a standard format to use? Who will extend the standard format, when necessary?
- wudangmonk 5y agoObjects would be a deadend but it would be nice if they had the option to return data structures. No reason why you can't have both text and strutured output.
- geophile 5y agoCheck out marcel (https://marceltheshell.org https://marceltheshell.org), which does this exactly. E.g. to explore the current directory recursively, find files changed in the last day, and then compute the sum of file sizes by extension for the qualifying files: ls -fr | select (f: now() - f.mtime < days(1)) | map (f: (f.suffix, f.size)) | red . + - ls -fr: list recursively, yielding only files. - select (f: ...): f is bound to each file from ls. If a file's modification time is within one day of the current time, then write that file object to the output pipeline. - map (f: ...): f is bound to each file from the select, and mapped to a tuple containing the extension and file size. - red . +: Group by file extension and sum sizes. And shelling out from python is much simpler. Here is the same computation in python from marcel.api import * for ext, size in (ls(file=True, recursive=True) | select(lambda f: now() - f.mtime < days(1)) | map(lambda f: (f.suffix, f.size)) | red(None, r_plus)): print(f'{ext}: {size}) marcel.api imports functions matching the commands (ls, select, and so on), and the pipeline yields an iterator, for neat integration with python loops.
- Fire-Dragon-DoL 5y agoWouldn't ruby fit such bill?
- jrumbut 5y agoOr even Perl!
- catlifeonmars 5y agoA looong time ago I wrote a Ruby shell and named it Rubbish.
- xcfvbnkhnkjq 5y ago>I am one of many people who don't really care for shell, it's old ... and just plain ugly. Egads laddie, it'll happen to you too some day.
- cutler 5y agoSo long as your shell language isn't object-oriented. We don't want PowerShell on UNIX.
- MathMonkeyMan 5y agotoo late! https://github.com/PowerShell/PowerShell https://github.com/PowerShell/PowerShell
- majkinetor 5y agoAmen to that !
- minhmeoke 5y agoTurtle [1] is a reimplementation of the Unix command line environment in Haskell that permits you to run external shell commands but leverages the typesystem to provide structured values and reduce errors. Turtle> cd "/tmp" Turtle> pwd FilePath "/tmp" Turtle> :type pwd pwd :: IO Turtle.FilePath Turtle> touch "file" Turtle> testfile "file" True Turtle> rm "file" Turtle> testfile "file" False Fish [2] and Oilshell [3] are two other efforts to create new shell dialects that improve usability while remaining somewhat backwards-compatible. [1] https://hackage.haskell.org/package/turtle https://hackage.haskell.org/package/turtle https://hackage.haskell.org/package/turtle-1.5.22/docs/Turtle-Tutorial.html https://hackage.haskell.org/package/turtle-1.5.22/docs/Turtl... [2] https://fishshell.com https://fishshell.com [3] https://www.oilshell.org/ https://www.oilshell.org/
- dunefox 5y ago> There needs to be a modern language that looks friendly, is compatible with shell, and is apart of the shells environment. Julia: https://docs.julialang.org/en/v1/manual/running-external-programs/ https://docs.julialang.org/en/v1/manual/running-external-pro... julia> prefixer(prefix, sleep) = `perl -nle '$|=1; print "'$prefix' ", $_; sleep '$sleep';'`; julia> run(pipeline(`perl -le '$|=1; for(0..5){ print; sleep 1 }'`, prefixer("A",2) & prefixer("B",2))); B 0 A 1 B 2 A 3 B 4 A 5
- lizmat 5y agoThen maybe Raku would be a thing for you: https://docs.raku.org/language/create-cli https://docs.raku.org/language/create-cli
- nerdponx 5y agoPython actually has great support for pipelining with the lazy iteration protocol built into the language. The problem arises when you need to chain executables together, rather than just Python code. It turns out that shells do some pretty complicated stuff with file descriptors and forking, stuff that you definitely do not want to DIY in Python. So "streaming" data from one executable to another means you buffer everything in Python runtime, even if you use async. This is fine for a lot of cases, but it feels wasteful and in elegant to me.
- chubot 5y agoPast submissions: https://news.ycombinator.com/item?id=27378444 https://news.ycombinator.com/item?id=27378444 (77 days ago) https://news.ycombinator.com/item?id=27036572 https://news.ycombinator.com/item?id=27036572 (3 months ago) More non-HN comments here: http://www.oilshell.org/blog/2021/06/hotos-shell-panel.html#theres-a-lot-of-interest-in-unix-shell http://www.oilshell.org/blog/2021/06/hotos-shell-panel.html#... (as well as my notes on the HotOS conference event)
- zeotroph 5y agoThis proposes a JIT compiler which can e.g. increase parallelism inside a chain of pipelined commands: We propose a dynamically triggered optimization regime for the shell that we call Jash, short for ‘Just a shell’. Jash inspects each shell command as it comes in to identify candidates for rewriting.
- taeric 5y agoFeels awkward. Piped commands already fork out and run independently. Just... Most folks go out of their way not to launch from a shell.
- rob_c 5y agoComing from HPC and HEP. No the UNIX shell it's not foreign to academia. It's the bread and butter of research at scale and has been since the mainframe... With that in mind I'll confess to not having read the rest of the article because really, that's so far from a correct point as to render discussion mute.
- nickthemagicman 5y agoI think people use Bash/Sh, because there's no other default choices, much like javascript in the browser. If they switched the bash terminal to a python repl, or just reliably offered stable python, that was available on every single distro, and just worked...I bet people would be fine with it. Same with replacing JS in the browser with python or another dynamic language. Tech people are practical and will just learn whatever they need to get things done quickly without introducing additional complexity. But then there's the idea that todays elegant language become tomorrow's kludgy language. So in the long run theres no winning. :)
- edgyquant 5y agoI don't think this is an apt comparison. For the last generation CMD was the "shell" on the most widely installed user OS on the planet and yet everyone who ever used bash immediately thought, "why does windows not just use this?," since it was so much nicer. Microsoft ported Linux to windows because people love the developer experience it provides. That doesn't mean that there isn't a better shell, I just don't think Javascript being the dominant web language is a good comparison.
- nickthemagicman 5y agoI've heard numerous people say PowerShell is better than bash. I think they ported bash just to bring the bash audience over to windows...not because there's any objective quality improvements. I could be wrong. There's a lot to love about bash. Just alot MORE to love about actual fully featured dynamic programming languages i think. It's interesting to think about.
- LeonB 5y agoI believe Microsoft’s move toward Linux is to make Azure competitive. Azure is a “bet the farm” strategy for Microsoft (and has been since before it launched) - they’ll do anything they need to do to make it work. (And so far it’s been a pretty good play!)
- nixpulvis 5y agoI have an idea for a shell I've been very slowly building out where the user can embed other languages in {#! ... } blocks of an otherwise POSIX compliant shell. I haven't solved the hard problems yet and I regularly wonder why I'm bothering myself with this, but it still feels cool sometimes. Even if I can't really use it for every day things yet without tab-complete, proper redirection and a million other missing features. Yes, I think the shell is an amazingly empowering tool, and yes it's also in need of some love. It's hard pulling apart what pieces of the specs still make sense, and what parts don't. But I've tried to learn and love fish shell (for example) and I think it's just proof for me that a new shell language is just going to cause more integrations and translations I need to perform on the fly. I already convert from 12 hour to 24 hours in real time, I don't really need to be translating bash->fish every time I want to do a conditional. Embedding languages inside each other is a topic of theoretical consideration now for me, and I'll admit I haven't found a compelling source for previous research on this yet, but I feel like I might need it one day. Anyway, I'll just be happy if one day I can try and forget about all the weird flags shell options that can be set at any time and impact the evaluation strategies of future expressions directly. But alas, the shell would never stay still.
- Lindby 5y agoYou can do this in any shell already by placing the other language in a separate executable file. With everything in the same file the risk of ignoring the separation of concern principle (each unit should do one thing and do it well).
- hyperpallium2 5y agoIn 100 years, we'll have strong AI writing our bash scripts. Technology adoption is driven by extrinsic need, not intrinsic merit - so backcompatibility is hard to beat. This is especially true of infrastructure, from train tracks to bash scripts.
- ineedasername 5y agoI expect to see that strong AI paging through man files to find the appropriate switches for its needs.
- nix23 5y agoThat seams more reasonable, or analyzing you script AND understands it then gives you optimizing changes.
- nix23 5y ago>In 100 years, we'll have strong AI writing our bash scripts How do you tell the AI what you exactly want? It's just another abstraction...and probably a bad one.
- hyperpallium2 5y agoIn the beginning, it will analyse you to determine your wants. In the end, it will determine your needs, superseding your wants.
- nix23 5y agoYeah, but i wanna go to a Pub but work for my Boss, so it needs to analyze what i want that in reality my boss wants ;)
- coldtea 5y ago>In 100 years, we'll have strong AI writing our bash scripts. Yeah, and in 50 years after 1969 we'd have colonies on Mars. Meanwhile it's been 50 years that we haven't even send a man outside low earth orbit, much less to the moon even.
- 1vuio0pswjnm7 5y agoGlad I learnt the shell before some hyped "modern" scripting language. Shell scripts just keep working year after year, they do not go "out of style" and they are generally fast. I cannot predict the future, the direction of populist computer programming is hilariously nonsensical, so much hype and failed attempts at improvement. However, like the authors of this ACM article, I know for sure the shell will outlive me. I love the UNIX shell. Not for what it is, but what it isn't.
- atoav 5y agoNothing against a good old shell script. But there is IMO a threshold of complexity beyond which shell scripts become unwieldy and where any scripting language starts to become the more rational choice. I still have python scripts from 10 years ago that happily do their job. I just had to swap print foo for print(foo) at one point when I changed from python 2 to 3 everywhere
- MathMonkeyMan 5y agoYou're lucky. The str/bytes/unicode change and the related standard library changes broke almost every python script I'd ever seen or written. It starts in libraries that you depend on -- wrappers around C code and whatnot, and then it just infects everything and breaks the world. I'm happy to be writing all my python as python 3 now. It took ten years.
- pfranz 5y agoThe rule of thumb at a previous job was if the shell script is longer than about a page then consider if another language is more appropriate. That seemed to serve me well and I had dozens and dozens of personal shell scripts to smooth over rough edges of everyday tasks. It's also obvious when you start needing features like more than simple usage or flags or parsing html. I haven't had all that much experience with Python 3, but for the 2.x series there were often tweaks that needed to be made either with code or libraries when moving between new releases with a large code base. Lastly, Python just isn't great for making one-liners. It might be a personal quirk, but that's my favorite way to iterate over an problem. With all that said, I do really like Python and use it heavily.
- cutler 5y agoIf you like Clojure there's babashka.
- Bost 5y agoAlso, you may have want a look at GNU Guix and GNU Guile.
- wudangmonk 5y agoTo me it seems like the REPL is the eventual solution. A shell is pretty much one already, you install programs in the same way one loads libraries in a repl. It lacks a language to glue them together since bash is not up to the task. The reason people want structured data and not just text is to be able to glue programs in the same way you can with functions. The glue should be structured data, none of that object bs. It should also have the option for plain text because sometimes that is all you need and you want simple tasks to remain simple. I'm guessing you'll use some sort of binary spec in order to help in the conversion of data structures between languages. Once you have the communication between programs handled all you need is a decent language to handle the logic. The language doesn't really matter though since the core that allows all of this is not tied to any one language. I'm sure plenty of people would write their own wrapper for their favorite language.
- funnybeam 5y agoAgreed and you’ve just described PowerShell which I think is perfect for this. That said, I do also appreciate bash scripts which are cleaner if you just have a long list of commands to execute and no real logic involved
- stonewareslord 5y ago> since bash is not up to the task What do you mean by this? You can write safe shell scripts; it just takes discipline and learning how to correctly write them. There are a lot of pitfalls but it isn't that hard if people actually learned the language. Edit: to clarify discipline meaning always doing things correctly and never taking shortcuts. For example, you must use: while IFS= read -r || [[ "${REPLY}" ]]; do ((X++)); done < <(command) If you want to loop over lines in a file. Don't use for loops, don't pipe to while, etc. It may be verbose but it won't be buggy
- da39a3ee 5y agoImportant subject, but that’s a pretty terrible article. I do understand the need to have publications by one’s name (I’ve been there too) but still, it’s a bit embarrassing to see this, which is just a blog post really, dressed up as an “academic article”.