24 ms·

Become Shell Literate

512 points by als0 6y ago

jeffbee 6y ago
cpio is the better tool for the job in the last example, compared to tar. You don't have to use xargs, for one. It's safer to use `find -0 | cpio -0o` because it can handle weird filenames that contain newlines or other spaces.
- ed25519FUUU 6y ago
  File names that contain newlines??
  - kaszanka 6y ago
    Yes, just one of the numerous pitfalls you always have to be mindful of when using Unix shell commands.
dllthomas 6y ago
I agree with the thesis but if you're taking this approach with vim it's very much worth being aware of how to use the quickfix and/or location buffers so you can easily step between the things that you've found. If you put a bunch of location-prefixed lines in a vim buffer (either from running a command within vim or by piping to `vim -`) you can get them in your quickfix buffer with `:cbuffer!` and then step between locations with `:cnext` and `:cprev` (one of the few things my top-level .vimrc does is bind these to tab and shift-tab, respectively).
tutfbhuf 6y ago
I wish that shell would be more sane. I don't think that a shell should be a complete programming language. If you need a programming language, then better use one. There is xonsh if you are looking for something like this. I think there should be a better bash with an very clean and consistent interface. Some of the most commonly used utils like awk '{ print $2}', sed, grep, sort, ... should be included out of the box to improve performance and to provide consistency across distributions.
- thyrsus 6y ago
  If the data is small, then expressivness is more important. If the data is large, then multiple processes connected by pipes lets you employ more CPUs.
  - mooreds 6y ago
    Have to pull this old chestnut out: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-... I am not the author of that piece, but the analysis is wonderful.
- enriquto 6y ago
  > I don't think that a shell should be a complete programming language. What do you mean exactly? If you have pipes and lists (e.g., the lines on a file) you are Turing-complete and people can and will make arbitrary programs. How would you like, precisely, to restrict the shell language so that it is not Turing-complete?
  - tutfbhuf 6y ago
    I mean shell should not be python, like xonsh. In my opinion it should not add programming language features (like types, futures, ADTs,...), but instead should clean up the existing built-ins + maybe little bit coreutils as built-ins.
    - enriquto 6y ago
      Never heard about xonsh, it looks really cool. Seems like a step in the good direction (from python to shell). It needs a few steps more to attain the elegant perfection of the shell language, for example, removing this "type" stuff from the variables ;)
- Aloha 6y ago
  It wouldn't be coherently unixy if it was. Small tools that do a limited subset of things, and reliably take/output data from stdin/stdout is the way unix is done. sh is just the glue we use to tack it all together. Bash (and other shells too) is like it is, because it's an accretion of 50 years of history rather than a singular top down design.
mooreds 6y ago
Hear hear! I remember a senior dev at my first job being stymied by the command line; he was very adept in the IDE, but had a hard time navigating directories. It was definitely a disadvantage when it came to getting stuff done. One thing the author didn't cover is how you can share reified knowledge when you write shell scripts. (It's the same with other programming languages, but they aren't as easy to write or modify.) That is really really powerful, because you can not only share it with others but also with your future self. I wrote about this more here: https://letterstoanewdeveloper.com/2019/02/04/learn-the-command-line/ https://letterstoanewdeveloper.com/2019/02/04/learn-the-comm...
- chubot 6y ago
  Yeah the way I think of this is that some people write a text or Markdown file with shell commands for documentation. I INVERT this, and write a shell script with comments :) That way someone else can reuse your knowledge more easily (and your future self as well). Examples: Constructing a big curl command to use the Zulip API, and also using jq for the first time. I used this to easily make a blog post out of a long Zulip thread [1] https://github.com/oilshell/oil/blob/master/services/zulip.sh https://github.com/oilshell/oil/blob/master/services/zulip.s... (oops some tabs snuck in here) Figuring out how to use uftrace (and successfully optimizing Oil with it [2]): https://github.com/oilshell/oil/blob/master/benchmarks/uftrace.sh https://github.com/oilshell/oil/blob/master/benchmarks/uftra... Though one issue is that shell scripts don't really specify their environment, but there is a large number of tools growing around containers that can solve this problem. (Basically Docker is being refactored into something more sane; thank you to OCI and others.) So I hope to integrate the Oil shell and containers more so shell scripts are more reproducible. I mean most of the container tools are already command line tools so in some sense it's already done, but you can imagine closer integration (e.g. not just passing code as text from outside the container to inside the container). ---- I wrote some notes about the documentation issue here: http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#the-0-dispatch-pattern-solves-three-important-problems http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#... And one thing I've wondered is if Oil should literally run shell out of markdown, so you can create executable docs. I can see it being useful, but it might be something you should do with a separate tool that converts markdown code blocks to a shell script... [1] http://www.oilshell.org/blog/2020/11/more-syntax.html http://www.oilshell.org/blog/2020/11/more-syntax.html [2] http://www.oilshell.org/blog/2020/01/parser-benchmarks.html http://www.oilshell.org/blog/2020/01/parser-benchmarks.html
thyrsus 6y ago
TIL "git status -s". So much ad hoc bad parsing no longer necessary!
- Liskni_si 6y ago
  If you really intend to parse the output of "git status" (or any command, really) you want the --porcelain option which is guaranteed to stay somewhat stable and parsable by scripts. Many other commands have such an option. As an example, iproute2 has -json.
chiefalchemist 6y ago
> "Shell literacy is one of the most important skills you ought to possess as a programmer. The Unix shell is one of the most powerful ideas ever put to code, and should be second nature to you as a programmer. No other tool is nearly as effective at commanding your computer to complex tasks quickly — or at storing them as scripts you can use later." I'm not going to disagree. However...the presenter would be wise to spend a paragraph or two about actual benefits. In addition, tell us why I should listen to you. You don't persuade others with subjectivity and finger waving. Even something as simple as "...during my ten years as a software engineer..." Otherwise, this is more rant than educational. Sorry, but I see this pattern far too often and am hoping to nudge someone here away from it.
thakoppno 6y ago
that one tip about using touch to modify the video file metadata resonated with me. at the same time, it is a bit of a hack. what is the minimal non-hack version of this technique? seems abstractly like liking a file-descriptor. thanks for the article. it reminds to get better at awk.
- masklinn 6y ago
  > at the same time, it is a bit of a hack. what is the minimal non-hack version of this technique? seems abstractly like liking a file-descriptor. Using a "proper" media manager and adding the file to a play queue?
- jeffbee 6y ago
  > what is the minimal non-hack version of this technique Rename the file 00WATCHME
  - bradwood 6y ago
    that's not a hack?! :)
ddevault 6y ago
Author here! Some extra tips for HN: If you dislike POSIX sh, consider looking at Plan 9's rc before you look anywhere else (including bash): http://man.9front.org/1/rc http://man.9front.org/1/rc Also, I would advise you to learn about these four tools in depth: sh, sed, awk, and make. Learn when to use each, and don't use one when another would be better.
- enriquto 6y ago
  Thanks for writing this! I enjoyed this article. The "make" tool is really powerful, besides its use for calling compilers. You can write really nifty scripts with make that have automatically optimal parallel scheduling.
- greenie_beans 6y ago
  Thanks !
jancsika 6y ago
I can just never get past the arg/flag inconsistency/complexity across commands. Perhaps if the docs started with a simple example of what has been seen over time to be the most common incantation for each command it might be tolerable. But as it is, I spend more time dealing with idiosyncracies of a command than I do expressively piping stuff. Perhaps if Rust had five mutually incompatible borrow-checkers which get called based on who wrote the code for a particular language construction-- that might get across the frustration I feel when using the shell. (Well, honestly I just StackOverflow for the shell incantation I need and that seems to work well enough.)
- rootlocus 6y ago
  > Perhaps if the docs started with a simple example of what has been seen over time to be the most common incantation for each command it might be tolerable. You might enjoy https://cheat.sh https://cheat.sh which is usable via curl: `curl cheat.sh/grep`
  - regenetron 6y ago
    Thanks for sharing, that’s a great tool.
  - zeppelin101 6y ago
    Very impressive. I just made a function 'cheat' that takes a command name as input and curls this website. Thanks.
- acabal 6y ago
  The inconsistency sometimes bothers me too, but it makes me feel a little better to remember that so many of the CLI commands we take for granted are part of an old historical heritage. The inconsistency is part of that heritage--today's CLI wasn't designed all at once by one group, but rather evolved over 50+ years from many contributors, back when nobody expected that people would still be using `sed` in 2020. For me, reflecting on that history helps dull the annoyance of having to type `grep --extended-regexp` but `sed --regexp-extended`.
gfxgirl 6y ago
Someone should write "become IDE literate" as a response to opening of using vim with no extensions and using grep a lot. I've been using editors that are language aware since at least the late 90s. Depending on the language they'll show me all references, take me to the definition or declaration, stack those jumps so as I follow the links I can pop back a level to where I was. All of this is instant. 1000x faster than opening a shell and trying to grep all the project files for word phrases which, not being language aware, can't tell if one 'foo' is relevant or irrelevant from another 'foo'. They'll also let me refactor various things from renaming a method/class/variable/function and correctly fixing all the related files to doing things like changing class members to getter/setters and other things. The difference is like using a hammer vs using a nail gun. There are times where they hammer is useful but given the job of constructing a building a nail gun will get the job done much faster.
- bentcorner 6y ago
  I do like my IDE but my one complaint with leaning on it a lot is that it lets you write code that is harder to understand if you don't have that IDE readily available. For example, if you name a member "flag" and you have several types with the same member, your IDE can tell you where this flag is used, so it's not a big deal if you wanted to refactor or are trying to track down a bug. But god help you if you're looking at your code in a diff viewer or in your browser in an online repo.
  - morelisp 6y ago
    A better example might be perpetuating typos (and other less obviously wrong bad names), the number of times I've seen some variant of "setIntercetpor" or "closeDataSteam" or... anything that gets actually typed as "most of the first word plus completion."
    - tracyhenry 6y ago
      Your example sounded like results produced with editors that do not understand the underlying language. A proper IDE would not prompt you with closeDataSteam if it's not a member function of the object to which you want to close the data stream.
bredren 6y ago
I find shell literacy more important than ever as I continue to invest in CI/CD using GitHub actions. With multiple contexts, and sometimes complex behavior, leveraging shell seems key to debug and gluing things together.
- chubot 6y ago
  Yup, it seems that shell embedded in YAML is pretty much a pattern now (Github Actions, Gitlab, Kubenetes, Travis CI, sourcehut, etc.) And I think it's growing beyond CI/CD -- that is, Github actions is a more general platform than Travis CI, etc. Good example: https://lobste.rs/s/oeelem/using_github_issues_as_hugo_frontend_with https://lobste.rs/s/oeelem/using_github_issues_as_hugo_front... My pet peeve is that YAML is a terrible syntax for a shell script, i.e. there are if statements in a weird YAML DSL in addition to if statements in shell (which people already complain is hard to remember) People complain about shell syntax, but YAML is even worse. So I would like to make Oil a better language for specifying these "distributed shell scripts": https://news.ycombinator.com/item?id=25343716 https://news.ycombinator.com/item?id=25343716
  - bredren 6y ago
    It is not a very pretty pattern but the value on GA is so high for me these days, I just fight it until I get what I need. I think there are probably very elegant ways to combine yaml and shell scripting, maybe by careful factoring of things into their own actions. I hope so. I will check out oil.
    - chubot 6y ago
      Yes, the lobste.rs thread also has that feedback: Github Actions is a tremendously useful platform, but the syntax and tooling sucks! Since JSON is valid YAML, it's possible to generate JSON from an Oil configuration with embedded and parsed shell, rather than using YAML. (Actually there's no problem generating YAML either, but I suppose it's easier to think about.) Example: I changed my own configs to use .yml.in and .yml and it worked on both Travis CI and sourcehut. https://github.com/oilshell/oil/tree/master/.builds https://github.com/oilshell/oil/tree/master/.builds Some brainstorming here on this feature: https://github.com/oilshell/oil/wiki/Config-Dialect https://github.com/oilshell/oil/wiki/Config-Dialect The benefit would be that it would validate your syntax and possibly your schema locally, not remotely. Some services have command line tools, but I'm sure they don't validate your shell, only the configuration. Another benefit is the ability to reuse code and configuration. For example, my .builds/ dir has a fair bit of redundancy in its configuration, and that could be "factored" with a real language. YAML has a limited << reference mechanism that does some of this. I'm looking for feedback on Oil, as well as the YAML + shell replacement idea. I hope that Oil can be a flexible language that will let people write these sorts of front ends for many cloud services, i.e. "distributed shell scripts". And eventually enable portability across clouds too. https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback
acabal 6y ago
Learning the shell was one of the best things I did as a programmer. It helped a lot that 15+ years ago I committed to running desktop Linux as a daily driver. Especially back then when things were much rougher on the desktop, using Linux as a daily driver means occasionally doing something in the shell. Ultimately, just like learning a programming language, one must have a practical reason, a project, to make it worth the while to learn as you go. For someone interested in learning to use the power of the shell, don't just go read a book or do some exercises or tape a cheat sheet to the wall. Instead, commit to using it for daily tasks for a month, or for maintaining a project entirely in the shell, not even using a GUI file manager. It's tough at first but so worth it!
- scaladev 6y ago
  At least it still gives you the choice. I learned a lot by stripping down my system and removing/replacing a lot of components to the extent that it didn't resemble Fedora (or Debian, or whatever) at all anymore. The amount of stuff you learn after completely trashing your only system and needing it to be restored and usable in a few hours…
  - encom 6y ago
    Funny, I actually learned Linux the opposite way. I started using Gentoo. I know that seems like learning Linux on hard mode, but for me it worked well. I had dabbled a bit before that, but I didn't really get Linux, before getting my hands dirty in this way.
prtkgpt 6y ago
Thank for sharing this. Very well done.
deleted 6y ago
[deleted]
dan-robertson 6y ago
There’s lots of powerful things you can do with a command line but I think it’s not that great an interface for the interactive run-look-modify loop one generally uses with writing shell commands. Some commands are slow and caching their results can be a bit of a pain, if only because you need to use different syntax. Some commands shouldn’t be rerun as part of the cycle as they are side-effecting. Some commands can produce huge amounts of output which is annoying to deal with if you don’t think to run them into a pager. And sometimes certain kinds of data flow can be tricky to write (eg something like 1. List a bunch of things with a command, 2. For each thing pass it as an arg to a second command, 3. Print the result of that command next to the thing, where you want to write something like list-foo | tee /tmp/l | xargs -n1 query-foo | paste /tmp/l -, except this won’t work even if /tmp/l is a named pipe (it works if it’s a regular file and you add a | tac | tac after the tee)). Finally, quoting is a bit of a horrific mess. So here is a proposal for how I’d like a shell to work: 1. It still works as a shell so you can run ordinary commands. 2. By default all pipe operations don’t use a single Unix pipe but rather two pipes where data is transferred from one to the next by the shell (this can be done entirely in the kernel and very cheaply so long as you transfer in pages), and so you can see simple statistics (ie is anything happening), and optionally see some of the data going through 3. When you run a command, the output gets directed into something like a pager so it won’t make your history inaccessible, and furthermore, there’s a way to tack on a pipe into some other command on the end of your pipeline without it needing to rerun the earlier steps 4. Some commands like grep get interactive versions where you can see how the output varies live as you type in the arguments—there’s no need to keep readjusting your regex 5. Something a bit like awk or xargs as a first class feature of the shell language (and therefore some kind of notion of closure that builtin functions can use) to avoid some parts of quoting hell
baby 6y ago
Today I find VSCode is so amazing that you’re really missing out if you avoid using it.
- tonyedgecombe 6y ago
  It's nice but it isn't as good as Visual Studio/Reshaper if you are using C#.
spicyramen 6y ago
Our codebase uses tons of shellscripts and Python. Shell has been great for running installations tasks. Recently we added presubmits and added shellcheck utility to check for style and syntax. Any suggestions for adding unit tests for she'll scripts will be great. I'm planning to unify the syntax across the board
- diarrhea 6y ago
  Are you using shell for installations, or make? I would think builds/installs/tests are more easily done via make, as opposed to a self-made shell implementation.
- IshKebab 6y ago
  Bet you £10 if you try your code from a path containing spaces it doesn't work.
- tommd 6y ago
  I love shellcheck. There's no great reason not to have any new shellcheck errors commented on each pull request. It's almost always right and how much you as the developer cares is the big question.
- efrecon 6y ago
  Try shellspec for tests. https://shellspec.info/ https://shellspec.info/
jgilias 6y ago
To me the main argument for getting used to the shell is that it's a compounding investment. Sure, initially one would be slower using a shell driven workflow as opposed to someone using an IDE. However, the more you stick to it, the more you find out that the skills you've gathered are very transferable and are applicable across languages, projects, types of projects, tasks, types of work, and even whole industries. If you know the shell well and are well versed in a good CLI text editor such as Vim or Emacs, the programming language of the day doesn't matter. Whether you're coding or setting up some environment doesn't matter. Whether you're on your computer, connected to some server, or an embedded device doesn't matter. EDIT: And eventually, you're not slower than someone using an IDE either.
simonw 6y ago
Learning to use the shell and learning SQL are the two skills that have stayed relevant throughout my entire 20 year career. Other tools and languages come and go, but the shell and SQL just keep on providing value.
- jahbrewski 6y ago
  Any resources/recommendations for truly learning SQL? I feel pretty comfortable with the basics, but could certainly stand to improve my SQL toolkit!
  - em500 6y ago
    https://blog.jooq.org/2016/03/17/10-easy-steps-to-a-complete-understanding-of-sql/ https://blog.jooq.org/2016/03/17/10-easy-steps-to-a-complete...
  - simonw 6y ago
    I'm a big fan of Markus Winand: https://use-the-index-luke.com/ https://use-the-index-luke.com/ I've got a bunch of SQL resources I've collected on my blog that you might find helpful: https://simonwillison.net/tags/sql/ https://simonwillison.net/tags/sql/
  - sqlhelpthrow 6y ago
    Haven't tried it, but have heard high praises here in HN for Jennifer Widom's SQL course. It's now on edX. https://cs.stanford.edu/people/widom/ https://cs.stanford.edu/people/widom/ https://www.edx.org/bio/jennifer-widom https://www.edx.org/bio/jennifer-widom
strulovich 6y ago
Piping and shell commands are powerful, but generally a bad experience. You have to plan your command, try to run it and understand how all the piping steps work. If you get it wrong, it’s an annoying experience. As an alternative, try Sublime with multiple selection (or other editors). With multiple selection skills you can transform your lines in a WYSIWYG interactive format which is much easier to work with for me. (To use it well you need to know the following shortcuts: split selection to lines, select next, and the alt+arrows jumps to next/previous word) This won’t mean you don’t need to learn how to use shells, but is still pretty fun to use.
- chubot 6y ago
  Tips on that: - Use "head" to reduce your data first and then quickly iterate on the pipeline. (Or sample [1] if head isn't representative; this is rare in practice) - Use tmux (or at least 2 terminals) with shell. The left side should have your editor, and the right side should have your shell (just like the screenshot in the blog post -- that's what mine looks like too) There is definitely a place for GUIs (IntelliJ CLion beats the GDB console for me) but you can get really far with vim, tmux, and shell. https://unix.stackexchange.com/questions/108581/how-to-randomly-sample-a-subset-of-a-file https://unix.stackexchange.com/questions/108581/how-to-rando...
- DeBraid 6y ago
  > try Sublime with multiple selection (or other editors). With multiple selection skills you can transform ... Yes! This saves me time consistently. I could wrangle a shell script, or where pattern-matching is essential, using text editor to select w/ keyboard shortcuts is VERY powerful.
mraza007 6y ago
I just love using shell and it’s one of the reason I ditched Windows for Linux
luord 6y ago
Fully agreed. So... As long as we're talking about the shell, when will we be able to interact with sourcehut entirely from the command line? Right now I'm particularly looking at managing build secrets. Being able to use it entirely from the command line is the only thing sourcehut lacks for it to become the best code/project/build management tool. At least for me anyway[1]; I'm aware that sourcehut's idiosyncrasies are not for everyone. On that note, is anyone aware of any such platform that can be managed entirely from the command line? The one that comes closest is the github CLI, I think, and even that one still requires a browser flow for logging in and getting the token[2]. [1]: I am aware that I could dive into the code and maybe contribute this myself but a) there might be plans for this already; I believe there's an issue where it was discussed and b) I don't have nearly enough time to commit to it. [2]: I know that this is very nitpicky of me, but I do truly want a tool like this; I think it'd be awesome.
- ddevault 6y ago
  We have a shell for builds.sr.ht over SSH: https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/master/master-shell https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/master/master-... You can use it to, for example, do `ssh builds@builds.sr.ht submit < build.yml` Expaning this to support secret management would be cool.
vfclists 6y ago
The shell is a total mindfuck. Trying to do any non trivial scripting/programming in it will mess up your mind. Oilshell perhaps?
jariel 6y ago
I think we need a visual shell. Git should literally show you a tree, then an animation of what's happening because there's a lot going on and I was shocked at my ignorance once I did see something. Mac Finder is my goto a lot because it's just powerful to see the tree. Could we blend the command line there? And finally - for expert users this is not a problem, for for medium users, the fact that many of the commands have different flags in different orders is a pain. A brand new set of integrated commands - where everything followed a predictable standard would be great.
larkinrichards 6y ago
I spend many hours of each day ~programming~ wrangling text files and I use macos + zsh + textmate2 for my daily drivers. I see shell as an important proficiency because it helps maintain a lower-level understanding of how the "magical" GUI "works," which often helps in debugging obtuse errors, and I'm sad to encounter more engineers who are completely unfamiliar with it. When it comes to examples like that posted by the OP, I like the combination of piping/pasting to mate and multi-caret editing for most scenarios where others would reach for awk/xargs. Here's me following the same example scenario but with multi-caret editing (slowed down slightly): https://user-images.githubusercontent.com/226503/101993951-6480cb80-3c73-11eb-9608-c6912011954b.gif https://user-images.githubusercontent.com/226503/101993951-6... Step by step: 1. git status -s | mate 2. select " D" with arrow keys + shift 3. command-E macos default for "use selection to find" 4. option-command-F to find all (multi-caret editing starts) 5. type "git checkout" to replace " D" 6. command-left to move cursor to start of line, then shift-command-right to select to end of line. 7. copy 8. select all + delete (clear document) 9. paste, 10. press return to insert newlines 11. select all + copy 12. switch back to terminal 13. paste To me, this is many small steps, but each step is more mechanical and flows naturally, and the general flexibility of multi-caret editing means it is applicable more often in my daily work.
- doppioandante 6y ago
  This is exactly how I would have done it, just using my editor of choice (kakoune) instead. I also wrote on Drew's mailing list in case anyone is interested: https://lists.sr.ht/~sircmpwn/public-inbox/%3CCAKW6382rV5iW28YfyK%2BUsAfX3%2BLzYNQXetW3B2qd%3D7cZ5LjNDg%40mail.gmail.com%3E https://lists.sr.ht/~sircmpwn/public-inbox/%3CCAKW6382rV5iW2...
masa331 6y ago
I completely agree, this knowledge can be very powerful. I also want to play Celeste[0] because it looks beautiful. But i have to say lately i dislike posts written in this command tone. Like "Do this to become X"/"Use this technology for Y"/"Don't do Z". Most of the time i end up feeling bad for not knowing something and therefore not being 10x programmer or even feeling antipathy towards the topic or author as i have different views. But in reality these things often don't matter or rather it can vary a lot person by person and context by context. You just don't need to be a shell literate to be a successful programmer. I wonder if more subtle introductions aren't better. Some which would spark interest but wouldn't leave a person with a bad feeling if he can't get into it right away. [0] http://www.celestegame.com/ http://www.celestegame.com/
rakoo 6y ago
It still strikes me that after all this time, the next step in the direction of the author is to use an editor like ACME (https://en.wikipedia.org/wiki/Acme_(text_editor) https://en.wikipedia.org/wiki/Acme_(text_editor)): use the shell to its maximum power, where the output of a command can be edited and used to control the existing interface. Take for instance this video: https://www.youtube.com/watch?v=4djoOiLste0 https://www.youtube.com/watch?v=4djoOiLste0 The author shows how the output of "git status" is just a text, you can modify the text and add "git add" just in front of the files you care about and add the file to the index. There is no linear loop of command input that becomes command output, it's all one big buffer that can feed itself. Consider how there could be another window that perpetually shows the status thanks to a combination of inotify and git status, and you have a git view. Fiddle with the command line argument and you can choose wether to show whitespaces or not, whether to show a summary view or the full diff, or restrict the list to some files. Have another window where you can write some message, and a GitCommit command in the "Tag" of the window will use the whole buffer as a git commit message; boom, you have 60% of what I use git-cola for. A bit more information can be seen by one of its creator here: https://www.youtube.com/watch?v=dP1xVpMPn8M https://www.youtube.com/watch?v=dP1xVpMPn8M. The possibilities are truly endless, and I haven't seen anything that resembles it. There is just no editor that embraces your platform the way Acme does. No, Emacs is not the same, because Emacs doesn't integrate with your OS; Emacs is an OS unto itself. You can't really say it's integrated to the OS when everything is implemented in the language that only Emacs uses.
- sime2009 6y ago
  Your git example is very similar to what can be done in my terminal, [Extraterm](https://extraterm.org/ https://extraterm.org/). See demo here: https://extraterm.org/features.html#reusing-command-output https://extraterm.org/features.html#reusing-command-output Thanks for the tip about Acme. I'm always looking for ideas about how to improve integration between the terminal based work flows and other applications, command line, GUI, or others.
  - rakoo 6y ago
    Wow, I've been looking for a terminal that does this sort of thing for a while, nice job! And it's beautiful !
intrepidhero 6y ago
This week I got my drill bit bound up in a double stud I wanted to run some wire through. The drill went into thermal cut out and wouldn't budge. I was at a loss of how to get the bit out without destroying the stud until I remembered I had a hand drill in the shop. It came from my great-grandfather. It's solid and well maintained and I was able to hand crank that drill bit through the rest of the wood. That's not the first time a simple hand tool has saved the day when a power tool let me down. I agree that IDEs and the tools that accompany them are powerful and can make you more productive. Learning them is an investment that can pay dividends. On the other hand, most of the IDEs I took the time to learn in school are now obsolete. I still use the shell today because those skills are still relevant and have gotten me out of a jam plenty of times. I've chosen to prioritize learning tools that are reliable and lasting even if it costs me some productivity.
gitowiec 6y ago
I agree that a programmer should know how to do things with shell, but when I read: > In my workflow, I use Vim as my editor, and Unix as my “IDE”. I immediately remember how great are Jetbrains IDEs. I just can't imagine how someone could refactor code with such accuracy and without hurdle just using Linux vi or any cli tool...
betwixthewires 6y ago
This guy keeps a Gemini site where he puts all of his articles as well. He's very intelligent and has an interesting perspective, I like his writing a lot. gemini://drewdevault.com/
xx789 6y ago
Sign me up for someone who hates the shell. I've always tried to write scripts in Python or even C++ is so much easier than bash, awk, etc. Bash has no repl, no unit tests not debugger its anachronistic. Don't get me started on emacs.
- tharne 6y ago
  > Bash has no repl, no unit tests not debugger Bash scripts are generally very short. If you need unit tests and a debugger you probably should be using something other than Bash. But saying you hate the shell because it's bad at something it was never designed for is silly.
  - xx789 6y ago
    At work I regularly have to support 500+ line scripts written by random developers, I 'n hate them.
    - hiq 6y ago
      Yeah I think that most people would hate this situation too compared to, say, similar scripts written in Python. But as parent said, the wrong tool for the wrong job is rarely a pleasant experience.
- hiq 6y ago
  Bash has no REPL?
  - xx789 6y ago
    yeah OK its probably the definition of repl.
kaszanka 6y ago
That pipeline sure does look convenient. Let's see how it will handle a path with spaces. $ git status -s | grep '^ D' | awk '{ print $2 }' | xargs git checkout -- xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option $ git status -s D "g h i" I'd be lying if I said I was surprised, to be honest.
- LeoPanthera 6y ago
  You can fix it by replacing the awk with "cut -d ' ' -f 2-" Haven't tested it, but you may also need to throw a "tr '\n' '\0'" in there and call xargs with "-0" to make it happy. This will still break on files with a CR in the filename. (CRs are allowed in unix filenames but, imho, they should not be.)
  - megous 6y ago
    you also want to tell xargs not to run the command on an empty set, otherwise you'll just do 'git checkout --' which will just kill all your local changes, probably.
- hiq 6y ago
  I think people who routinely write this kind of commands in an interactive shell are well aware of the shortcomings, they just deal with it when they need to and not before. One example is that people very rarely handle '\n' or spaces in filenames (e.g. using the -0 option from `xargs`). Whatever works is fine as long as one knows the risks. For a one-off thing like the one mentioned in the article I'd say it's almost ok, although I'd put `git checkout --` in the category of risky commands if you don't have any backup.
- patrec 6y ago
  Yeah, it's annoying that unix filenames and shell quoting are both fundamentally broken and that (as above) people would generally (knowingly) write a a broken-for-spaces-in-filenames version by default because it's easier (and the author no doubt knew he'd not encounter spaces in his repo). Having said that, it's not very hard to fix, I'd probably write something like: git status -s | awk -vORS="\0" 'gsub(/^ D /,"")' | xargs -0 git checkout -- Which is about the same length. This will still break for malicious input (newlines in filename), but for the original use case that's not a concern.
effnorwood 6y ago
It’s a gas station morons
29athrowaway 6y ago
If you want to try something new, try "Facebook path picker" http://facebook.github.io/PathPicker/ http://facebook.github.io/PathPicker/ Pretty useful when paired with git. e.g.: git status | fpp -c git add
tommd 6y ago
And for goodness sakes, use ShellCheck! It can be easy (i.e. there are CI services that will run shellcheck automatically on your PRs and comment bugs).
hiq 6y ago
Nice post, but 2020 and still using an unauthenticated connection (plain netcat) to transfer data including executables (if I understood correctly) is not something you can justify when there are plenty of safe alternatives (e.g. wormhole, syncthing). Maybe the recipient and the sender just compare hashes via a secure channel, but there's no mention of this.
- deleted 6y ago
  [deleted]
- ddevault 6y ago
  The files were transferred via yggdrasil. In any case, it would have to be an unusually clever attacker to identify executables in a tarball being transmitted unwrapped via TCP on a port chosen out of a hat, and intercept it to do something malicious.
  - Nextgrid 6y ago
    Malice is not the only concern. There was a post on here a couple days ago where AT&T was corrupting certain packets, and due to either luck of the TCP checksums being recalculated on the faulty equipment itself the bad packets were actually considered valid by the recipient.
  - hiq 6y ago
    > The files were transferred via yggdrasil. I wish you would have at least mentioned it in your post that you were indeed relying on the security of the network and that nc by itself is not secure. I'm not saying it would have been a real risk in your particular case, but giving these examples on a public website is basically saying "see, you can do it like this". I'm not saying you don't know better, but some of your readers might not, and may replicate your unsafe (without the underlying assumptions) code where the risk is higher. This is especially bad given that there are safe alternatives with equal or better ergonomics.
wglb 6y ago
I do so much of this it would be hard to document all of them. In a very large lisp project, I had an astonishing number of lines of shell code--none more than 10 lines long. (Today I would do production automation with cl-readline and inferior-shell and uiop:launch-program. Also, given that I use emacs, I do lots with dired, and on odd occasions eshell. And I have gathered a bit of emacs lisp code as well.
et1337 6y ago
Shell is simultaneously wonderful and horrendous. Did you know that the left bracket ([) is actually not a part of bash syntax, but instead a binary that lives in /bin? Run “which [“ if you don’t believe me!
- uluyol 6y ago
  And other things which you'd expect to be commands are both commands AND builtins! And they behave differently depending on which you call! Some examples are time and pwd.
sebelk 6y ago
The new religion of devopers says that shell is obsolete...
kmstout 6y ago
The following sets up a key binding to grab interesting lines from the history, throw them into a file, and pop it up in an editor: # put interesting lines from history into a file, open an editor function h2e() { echo "#! /bin/bash" > "$3" history | tail -n"$1" | egrep "$2" | sed -re 's/\s*[0-9]+\s+//' >> "$3" "${EDITOR:-emacs}" "$3" } # prompt for h2e's arguments function h2ei() { read -p 'num. lines [1]: ' lines read -p 'pattern [.]: ' pattern read -p 'file [foo.sh]: ' fn h2e "${lines:-1}" "${pattern:-.}" "${fn:-foo.sh}" } # fix the terminal settings for the duration of a command, revert afterward function insane() { save=$(stty -g) stty sane "$@" stty $save } # bind F12 to our history grabber bind -x '"\e[24~":insane h2ei' It's been quite handy of late.
- nickthemagicman 6y ago
  So you would run "h2ei 10 somepattern out.txt"? Which would put the last 10 lines matching some pattern to the txt file?
  - kmstout 6y ago
    Close--it's those lines matching some pattern within the last 10 lines. Think of it as localize, then filter. I've written it the other way, too, but this seems more natural to me. (Also, a direct h2e invocation would take the arguments from the command line; h2ei will ask every time.) In practice, I will do something, realize that the thing I just did over the last several minutes could be useful--or the core of something more general--and then strike F12 to start h2ei. At the prompts, I'll generally give it a nice round number like 10 or 20 or 50 lines, leave the default pattern, and set the filename to be something evocative in ~/bin. An editor will immediately open on my new shell script, and I can trim it down and continue hacking on it from there. The key idea is that I get an easier path from doing something ad hoc to creating something more useful in the long run.
megous 6y ago
Learning some kind of common shell language is certainly a knowledge that will serve you well in the widest amount of circumstances. From managing headless servers, to local use on a workstation, to scripting within your U-Boot bootloader, to reverse engineering Linux based IoT/devices that oftentimes include busybox. Shell is included in a lot of places.
mechnesium 6y ago
I don’t think this is necessarily true anymore. Today we have much more intuitive tools at our disposable such as vscode. I understand some people were raised on grep and vim, and it is like a second nature to them, but actively encouraging people to use these tools in 2020 in light of the alternatives is not pragmatic.
- ddevault 6y ago
  Much more intuitive, and much less useful.
voidhorse 6y ago
I used to think this way as well, but out of necessity, had to work specifically with a commercial IDE for some time. Turns out, if you are confident learning the keybindings of a robust IDE is worthwhile (e.g. you know that you must use it for some particular project for a decent amount of time) the investment pays off just as well as learning shell commands. A good IDE can do everything a cobbled together shell pipeline can, often in two keystrokes instead of N trial and error command inputs because you forgot, yet again that, e.g. while `grep` takes a pattern argument before a path, `find` does the opposite--in spite of POSIX there's an incredible lack of UX standardization when it comes to unix tooling. People like to pretend that learning shell commands is somehow better because it's more portable, but unless your work entails working across multiple machines and environments daily, it doesn't matter. Most of us do most of our work on one machine, in one environment, so at the end of the day, it really comes down to preference. Personally, I do prefer working in the shell, but if you prefer working in a modern IDE, don't let anyone fool you that you're somehow wasting time or being less proficient than you could be in a shell--if you took the time to learn the IDE keybindings and the IDE is on par with anything Jetbrains puts out, you're not losing in efficiency at all and possibly making gains when it comes to certain tasks.
- dragonwriter 6y ago
  > People like to pretend that learning shell commands is somehow better because it's more portable, No, it's better because if you aren't doing exactly the workflow an IDE or GUI tool designer has envisioned, it is almost invariably much easier to do it in shell (and then make it a script and then bind it to a command in your IDE or GUI tool, if they support that) than to beat the non-shell tool into, first, doing what you want, and then making it easily repeatable. It's also more portable, which, contrary to your description, is very useful for most people, because even if there are some dev-only tasks that “I can only do it in the IDE” is fine for, for many things you may also want to do in a CI pipeline, on a deployment box, or other places where your IDE isn't running. Shell scripts generally work there.
  - tracyhenry 6y ago
    I use IDE mainly for code editing and exploration (e.g. jump from usage to definition), for which I can't see how shell is easier. Modern IDEs have shell built-in so when I finish coding, I can just open up a terminal inside IDE to run shell scripts. Effectively I'm getting the best of both worlds. I never knew that I have to follow some sort of rails laid by IDE designers, which I think it's a very common misconception by people who dislike IDEs.
TeeMassive 6y ago
I got hired in a new company a year ago and we are in the process of migrating from TFS to git. It's quite disheartening to see the lack of skill with a shell. I'm being blocked from pushing and promoting the use of something else than Visual Studio's git pluggin because "other developers are not used to the command line". I proposed Git Extensions and it was more or less the same feedback, too different. It feels like an electrician being told not to use oscilloscopes because it's too hard for others in his team. At least they're in the process of migrating and eventually people will see the light, CLIs are powerful because of their versatility and therefore will server you for most of your career, while "learning" and IDE will server you 10 years at most.
noisy_boy 6y ago
Beyond the point of using shell and scripts as replacement for IDE/development, knowing the shell also helps with general productivity. E.g. I need to remote to office desktop and instead of having to open my phone to read the 2FA pin, I just setup stoken on my desktop and wrote a simple script that invokes stoken and copies the pin to the clipboard. Mapped the script to a shortcut and now I can just press the shortcut from the login page and do Ctrl+V for pasting the pin. Small things like these become very easy once you know a bit of scripting and the time/effort saved over time adds up considerably.
kortilla 6y ago
> I can work with this. I add grep '^ D' to filter out any entries which were not deleted, and pipe it through awk '{ print $2 }' to extract just the filenames. Best hope your file names don’t have spaces in them. One of the downsides of the shell on Unix is that everything is usually text meant for humans so you have to spend an annoying amount of time dealing with delimiters that make it easy on the eyes but a pain in the ass.
- hxegon 6y ago
  couldn't agree more. waiting on nushell to mature so that rich data is the norm for shell stuff. learning to be comfy in shell is still worth it though. It's saved me, at minimum, hundreds of hours of work.
jagged-chisel 6y ago
Can we at least agree on one thing? That sticking with the keyboard is faster than using the mouse. Use keyboard shortcuts in your IDE, type commands into your shell. The killer idea that will make you more efficient is scripting (or creating macros in your IDE) for your own common tasks and assigning them a good name (or a shortcut in your IDE.)

512 points by als0 6y ago

cpio is the better tool for the job in the last example, compared to tar. You don't have to use xargs, for one. It's safer to use `find -0 | cpio -0o` because it can handle weird filenames that contain newlines or other spaces.

ed25519FUUU 6y ago
File names that contain newlines??
- kaszanka 6y ago
  Yes, just one of the numerous pitfalls you always have to be mindful of when using Unix shell commands.

dllthomas 6y ago

I agree with the thesis but if you're taking this approach with vim it's very much worth being aware of how to use the quickfix and/or location buffers so you can easily step between the things that you've found. If you put a bunch of location-prefixed lines in a vim buffer (either from running a command within vim or by piping to `vim -`) you can get them in your quickfix buffer with `:cbuffer!` and then step between locations with `:cnext` and `:cprev` (one of the few things my top-level .vimrc does is bind these to tab and shift-tab, respectively).

tutfbhuf 6y ago

I wish that shell would be more sane. I don't think that a shell should be a complete programming language. If you need a programming language, then better use one. There is xonsh if you are looking for something like this. I think there should be a better bash with an very clean and consistent interface. Some of the most commonly used utils like awk '{ print $2}', sed, grep, sort, ... should be included out of the box to improve performance and to provide consistency across distributions.

thyrsus 6y ago
If the data is small, then expressivness is more important. If the data is large, then multiple processes connected by pipes lets you employ more CPUs.
- mooreds 6y ago
  Have to pull this old chestnut out: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-... I am not the author of that piece, but the analysis is wonderful.
enriquto 6y ago
> I don't think that a shell should be a complete programming language. What do you mean exactly? If you have pipes and lists (e.g., the lines on a file) you are Turing-complete and people can and will make arbitrary programs. How would you like, precisely, to restrict the shell language so that it is not Turing-complete?
- tutfbhuf 6y ago
  I mean shell should not be python, like xonsh. In my opinion it should not add programming language features (like types, futures, ADTs,...), but instead should clean up the existing built-ins + maybe little bit coreutils as built-ins.
  - enriquto 6y ago
    Never heard about xonsh, it looks really cool. Seems like a step in the good direction (from python to shell). It needs a few steps more to attain the elegant perfection of the shell language, for example, removing this "type" stuff from the variables ;)
Aloha 6y ago
It wouldn't be coherently unixy if it was. Small tools that do a limited subset of things, and reliably take/output data from stdin/stdout is the way unix is done. sh is just the glue we use to tack it all together. Bash (and other shells too) is like it is, because it's an accretion of 50 years of history rather than a singular top down design.

mooreds 6y ago

Hear hear! I remember a senior dev at my first job being stymied by the command line; he was very adept in the IDE, but had a hard time navigating directories. It was definitely a disadvantage when it came to getting stuff done. One thing the author didn't cover is how you can share reified knowledge when you write shell scripts. (It's the same with other programming languages, but they aren't as easy to write or modify.) That is really really powerful, because you can not only share it with others but also with your future self. I wrote about this more here: https://letterstoanewdeveloper.com/2019/02/04/learn-the-command-line/ https://letterstoanewdeveloper.com/2019/02/04/learn-the-comm...

chubot 6y ago
Yeah the way I think of this is that some people write a text or Markdown file with shell commands for documentation. I INVERT this, and write a shell script with comments :) That way someone else can reuse your knowledge more easily (and your future self as well). Examples: Constructing a big curl command to use the Zulip API, and also using jq for the first time. I used this to easily make a blog post out of a long Zulip thread [1] https://github.com/oilshell/oil/blob/master/services/zulip.sh https://github.com/oilshell/oil/blob/master/services/zulip.s... (oops some tabs snuck in here) Figuring out how to use uftrace (and successfully optimizing Oil with it [2]): https://github.com/oilshell/oil/blob/master/benchmarks/uftrace.sh https://github.com/oilshell/oil/blob/master/benchmarks/uftra... Though one issue is that shell scripts don't really specify their environment, but there is a large number of tools growing around containers that can solve this problem. (Basically Docker is being refactored into something more sane; thank you to OCI and others.) So I hope to integrate the Oil shell and containers more so shell scripts are more reproducible. I mean most of the container tools are already command line tools so in some sense it's already done, but you can imagine closer integration (e.g. not just passing code as text from outside the container to inside the container). ---- I wrote some notes about the documentation issue here: http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#the-0-dispatch-pattern-solves-three-important-problems http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#... And one thing I've wondered is if Oil should literally run shell out of markdown, so you can create executable docs. I can see it being useful, but it might be something you should do with a separate tool that converts markdown code blocks to a shell script... [1] http://www.oilshell.org/blog/2020/11/more-syntax.html http://www.oilshell.org/blog/2020/11/more-syntax.html [2] http://www.oilshell.org/blog/2020/01/parser-benchmarks.html http://www.oilshell.org/blog/2020/01/parser-benchmarks.html

thyrsus 6y ago

TIL "git status -s". So much ad hoc bad parsing no longer necessary!

Liskni_si 6y ago
If you really intend to parse the output of "git status" (or any command, really) you want the --porcelain option which is guaranteed to stay somewhat stable and parsable by scripts. Many other commands have such an option. As an example, iproute2 has -json.

chiefalchemist 6y ago

> "Shell literacy is one of the most important skills you ought to possess as a programmer. The Unix shell is one of the most powerful ideas ever put to code, and should be second nature to you as a programmer. No other tool is nearly as effective at commanding your computer to complex tasks quickly — or at storing them as scripts you can use later." I'm not going to disagree. However...the presenter would be wise to spend a paragraph or two about actual benefits. In addition, tell us why I should listen to you. You don't persuade others with subjectivity and finger waving. Even something as simple as "...during my ten years as a software engineer..." Otherwise, this is more rant than educational. Sorry, but I see this pattern far too often and am hoping to nudge someone here away from it.

thakoppno 6y ago

that one tip about using touch to modify the video file metadata resonated with me. at the same time, it is a bit of a hack. what is the minimal non-hack version of this technique? seems abstractly like liking a file-descriptor. thanks for the article. it reminds to get better at awk.

masklinn 6y ago
> at the same time, it is a bit of a hack. what is the minimal non-hack version of this technique? seems abstractly like liking a file-descriptor. Using a "proper" media manager and adding the file to a play queue?
jeffbee 6y ago
> what is the minimal non-hack version of this technique Rename the file 00WATCHME
- bradwood 6y ago
  that's not a hack?! :)

ddevault 6y ago

Author here! Some extra tips for HN: If you dislike POSIX sh, consider looking at Plan 9's rc before you look anywhere else (including bash): http://man.9front.org/1/rc http://man.9front.org/1/rc Also, I would advise you to learn about these four tools in depth: sh, sed, awk, and make. Learn when to use each, and don't use one when another would be better.

enriquto 6y ago
Thanks for writing this! I enjoyed this article. The "make" tool is really powerful, besides its use for calling compilers. You can write really nifty scripts with make that have automatically optimal parallel scheduling.
greenie_beans 6y ago
Thanks !

jancsika 6y ago

I can just never get past the arg/flag inconsistency/complexity across commands. Perhaps if the docs started with a simple example of what has been seen over time to be the most common incantation for each command it might be tolerable. But as it is, I spend more time dealing with idiosyncracies of a command than I do expressively piping stuff. Perhaps if Rust had five mutually incompatible borrow-checkers which get called based on who wrote the code for a particular language construction-- that might get across the frustration I feel when using the shell. (Well, honestly I just StackOverflow for the shell incantation I need and that seems to work well enough.)

rootlocus 6y ago
> Perhaps if the docs started with a simple example of what has been seen over time to be the most common incantation for each command it might be tolerable. You might enjoy https://cheat.sh https://cheat.sh which is usable via curl: `curl cheat.sh/grep`
- regenetron 6y ago
  Thanks for sharing, that’s a great tool.
- zeppelin101 6y ago
  Very impressive. I just made a function 'cheat' that takes a command name as input and curls this website. Thanks.
acabal 6y ago
The inconsistency sometimes bothers me too, but it makes me feel a little better to remember that so many of the CLI commands we take for granted are part of an old historical heritage. The inconsistency is part of that heritage--today's CLI wasn't designed all at once by one group, but rather evolved over 50+ years from many contributors, back when nobody expected that people would still be using `sed` in 2020. For me, reflecting on that history helps dull the annoyance of having to type `grep --extended-regexp` but `sed --regexp-extended`.

gfxgirl 6y ago

Someone should write "become IDE literate" as a response to opening of using vim with no extensions and using grep a lot. I've been using editors that are language aware since at least the late 90s. Depending on the language they'll show me all references, take me to the definition or declaration, stack those jumps so as I follow the links I can pop back a level to where I was. All of this is instant. 1000x faster than opening a shell and trying to grep all the project files for word phrases which, not being language aware, can't tell if one 'foo' is relevant or irrelevant from another 'foo'. They'll also let me refactor various things from renaming a method/class/variable/function and correctly fixing all the related files to doing things like changing class members to getter/setters and other things. The difference is like using a hammer vs using a nail gun. There are times where they hammer is useful but given the job of constructing a building a nail gun will get the job done much faster.

bentcorner 6y ago
I do like my IDE but my one complaint with leaning on it a lot is that it lets you write code that is harder to understand if you don't have that IDE readily available. For example, if you name a member "flag" and you have several types with the same member, your IDE can tell you where this flag is used, so it's not a big deal if you wanted to refactor or are trying to track down a bug. But god help you if you're looking at your code in a diff viewer or in your browser in an online repo.
- morelisp 6y ago
  A better example might be perpetuating typos (and other less obviously wrong bad names), the number of times I've seen some variant of "setIntercetpor" or "closeDataSteam" or... anything that gets actually typed as "most of the first word plus completion."
  - tracyhenry 6y ago
    Your example sounded like results produced with editors that do not understand the underlying language. A proper IDE would not prompt you with closeDataSteam if it's not a member function of the object to which you want to close the data stream.

bredren 6y ago

I find shell literacy more important than ever as I continue to invest in CI/CD using GitHub actions. With multiple contexts, and sometimes complex behavior, leveraging shell seems key to debug and gluing things together.

chubot 6y ago
Yup, it seems that shell embedded in YAML is pretty much a pattern now (Github Actions, Gitlab, Kubenetes, Travis CI, sourcehut, etc.) And I think it's growing beyond CI/CD -- that is, Github actions is a more general platform than Travis CI, etc. Good example: https://lobste.rs/s/oeelem/using_github_issues_as_hugo_frontend_with https://lobste.rs/s/oeelem/using_github_issues_as_hugo_front... My pet peeve is that YAML is a terrible syntax for a shell script, i.e. there are if statements in a weird YAML DSL in addition to if statements in shell (which people already complain is hard to remember) People complain about shell syntax, but YAML is even worse. So I would like to make Oil a better language for specifying these "distributed shell scripts": https://news.ycombinator.com/item?id=25343716 https://news.ycombinator.com/item?id=25343716
- bredren 6y ago
  It is not a very pretty pattern but the value on GA is so high for me these days, I just fight it until I get what I need. I think there are probably very elegant ways to combine yaml and shell scripting, maybe by careful factoring of things into their own actions. I hope so. I will check out oil.
  - chubot 6y ago
    Yes, the lobste.rs thread also has that feedback: Github Actions is a tremendously useful platform, but the syntax and tooling sucks! Since JSON is valid YAML, it's possible to generate JSON from an Oil configuration with embedded and parsed shell, rather than using YAML. (Actually there's no problem generating YAML either, but I suppose it's easier to think about.) Example: I changed my own configs to use .yml.in and .yml and it worked on both Travis CI and sourcehut. https://github.com/oilshell/oil/tree/master/.builds https://github.com/oilshell/oil/tree/master/.builds Some brainstorming here on this feature: https://github.com/oilshell/oil/wiki/Config-Dialect https://github.com/oilshell/oil/wiki/Config-Dialect The benefit would be that it would validate your syntax and possibly your schema locally, not remotely. Some services have command line tools, but I'm sure they don't validate your shell, only the configuration. Another benefit is the ability to reuse code and configuration. For example, my .builds/ dir has a fair bit of redundancy in its configuration, and that could be "factored" with a real language. YAML has a limited << reference mechanism that does some of this. I'm looking for feedback on Oil, as well as the YAML + shell replacement idea. I hope that Oil can be a flexible language that will let people write these sorts of front ends for many cloud services, i.e. "distributed shell scripts". And eventually enable portability across clouds too. https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback

acabal 6y ago

Learning the shell was one of the best things I did as a programmer. It helped a lot that 15+ years ago I committed to running desktop Linux as a daily driver. Especially back then when things were much rougher on the desktop, using Linux as a daily driver means occasionally doing something in the shell. Ultimately, just like learning a programming language, one must have a practical reason, a project, to make it worth the while to learn as you go. For someone interested in learning to use the power of the shell, don't just go read a book or do some exercises or tape a cheat sheet to the wall. Instead, commit to using it for daily tasks for a month, or for maintaining a project entirely in the shell, not even using a GUI file manager. It's tough at first but so worth it!

scaladev 6y ago
At least it still gives you the choice. I learned a lot by stripping down my system and removing/replacing a lot of components to the extent that it didn't resemble Fedora (or Debian, or whatever) at all anymore. The amount of stuff you learn after completely trashing your only system and needing it to be restored and usable in a few hours…
- encom 6y ago
  Funny, I actually learned Linux the opposite way. I started using Gentoo. I know that seems like learning Linux on hard mode, but for me it worked well. I had dabbled a bit before that, but I didn't really get Linux, before getting my hands dirty in this way.

prtkgpt 6y ago

Thank for sharing this. Very well done.

deleted 6y ago

[deleted]

dan-robertson 6y ago

There’s lots of powerful things you can do with a command line but I think it’s not that great an interface for the interactive run-look-modify loop one generally uses with writing shell commands. Some commands are slow and caching their results can be a bit of a pain, if only because you need to use different syntax. Some commands shouldn’t be rerun as part of the cycle as they are side-effecting. Some commands can produce huge amounts of output which is annoying to deal with if you don’t think to run them into a pager. And sometimes certain kinds of data flow can be tricky to write (eg something like 1. List a bunch of things with a command, 2. For each thing pass it as an arg to a second command, 3. Print the result of that command next to the thing, where you want to write something like list-foo | tee /tmp/l | xargs -n1 query-foo | paste /tmp/l -, except this won’t work even if /tmp/l is a named pipe (it works if it’s a regular file and you add a | tac | tac after the tee)). Finally, quoting is a bit of a horrific mess. So here is a proposal for how I’d like a shell to work: 1. It still works as a shell so you can run ordinary commands. 2. By default all pipe operations don’t use a single Unix pipe but rather two pipes where data is transferred from one to the next by the shell (this can be done entirely in the kernel and very cheaply so long as you transfer in pages), and so you can see simple statistics (ie is anything happening), and optionally see some of the data going through 3. When you run a command, the output gets directed into something like a pager so it won’t make your history inaccessible, and furthermore, there’s a way to tack on a pipe into some other command on the end of your pipeline without it needing to rerun the earlier steps 4. Some commands like grep get interactive versions where you can see how the output varies live as you type in the arguments—there’s no need to keep readjusting your regex 5. Something a bit like awk or xargs as a first class feature of the shell language (and therefore some kind of notion of closure that builtin functions can use) to avoid some parts of quoting hell

baby 6y ago

Today I find VSCode is so amazing that you’re really missing out if you avoid using it.

tonyedgecombe 6y ago
It's nice but it isn't as good as Visual Studio/Reshaper if you are using C#.

spicyramen 6y ago

Our codebase uses tons of shellscripts and Python. Shell has been great for running installations tasks. Recently we added presubmits and added shellcheck utility to check for style and syntax. Any suggestions for adding unit tests for she'll scripts will be great. I'm planning to unify the syntax across the board

diarrhea 6y ago
Are you using shell for installations, or make? I would think builds/installs/tests are more easily done via make, as opposed to a self-made shell implementation.
IshKebab 6y ago
Bet you £10 if you try your code from a path containing spaces it doesn't work.
tommd 6y ago
I love shellcheck. There's no great reason not to have any new shellcheck errors commented on each pull request. It's almost always right and how much you as the developer cares is the big question.
efrecon 6y ago
Try shellspec for tests. https://shellspec.info/ https://shellspec.info/

jgilias 6y ago

To me the main argument for getting used to the shell is that it's a compounding investment. Sure, initially one would be slower using a shell driven workflow as opposed to someone using an IDE. However, the more you stick to it, the more you find out that the skills you've gathered are very transferable and are applicable across languages, projects, types of projects, tasks, types of work, and even whole industries. If you know the shell well and are well versed in a good CLI text editor such as Vim or Emacs, the programming language of the day doesn't matter. Whether you're coding or setting up some environment doesn't matter. Whether you're on your computer, connected to some server, or an embedded device doesn't matter. EDIT: And eventually, you're not slower than someone using an IDE either.

simonw 6y ago

Learning to use the shell and learning SQL are the two skills that have stayed relevant throughout my entire 20 year career. Other tools and languages come and go, but the shell and SQL just keep on providing value.

jahbrewski 6y ago
Any resources/recommendations for truly learning SQL? I feel pretty comfortable with the basics, but could certainly stand to improve my SQL toolkit!
- em500 6y ago
  https://blog.jooq.org/2016/03/17/10-easy-steps-to-a-complete-understanding-of-sql/ https://blog.jooq.org/2016/03/17/10-easy-steps-to-a-complete...
- simonw 6y ago
  I'm a big fan of Markus Winand: https://use-the-index-luke.com/ https://use-the-index-luke.com/ I've got a bunch of SQL resources I've collected on my blog that you might find helpful: https://simonwillison.net/tags/sql/ https://simonwillison.net/tags/sql/
- sqlhelpthrow 6y ago
  Haven't tried it, but have heard high praises here in HN for Jennifer Widom's SQL course. It's now on edX. https://cs.stanford.edu/people/widom/ https://cs.stanford.edu/people/widom/ https://www.edx.org/bio/jennifer-widom https://www.edx.org/bio/jennifer-widom

strulovich 6y ago

Piping and shell commands are powerful, but generally a bad experience. You have to plan your command, try to run it and understand how all the piping steps work. If you get it wrong, it’s an annoying experience. As an alternative, try Sublime with multiple selection (or other editors). With multiple selection skills you can transform your lines in a WYSIWYG interactive format which is much easier to work with for me. (To use it well you need to know the following shortcuts: split selection to lines, select next, and the alt+arrows jumps to next/previous word) This won’t mean you don’t need to learn how to use shells, but is still pretty fun to use.

chubot 6y ago
Tips on that: - Use "head" to reduce your data first and then quickly iterate on the pipeline. (Or sample [1] if head isn't representative; this is rare in practice) - Use tmux (or at least 2 terminals) with shell. The left side should have your editor, and the right side should have your shell (just like the screenshot in the blog post -- that's what mine looks like too) There is definitely a place for GUIs (IntelliJ CLion beats the GDB console for me) but you can get really far with vim, tmux, and shell. https://unix.stackexchange.com/questions/108581/how-to-randomly-sample-a-subset-of-a-file https://unix.stackexchange.com/questions/108581/how-to-rando...
DeBraid 6y ago
> try Sublime with multiple selection (or other editors). With multiple selection skills you can transform ... Yes! This saves me time consistently. I could wrangle a shell script, or where pattern-matching is essential, using text editor to select w/ keyboard shortcuts is VERY powerful.

mraza007 6y ago

I just love using shell and it’s one of the reason I ditched Windows for Linux

luord 6y ago

Fully agreed. So... As long as we're talking about the shell, when will we be able to interact with sourcehut entirely from the command line? Right now I'm particularly looking at managing build secrets. Being able to use it entirely from the command line is the only thing sourcehut lacks for it to become the best code/project/build management tool. At least for me anyway[1]; I'm aware that sourcehut's idiosyncrasies are not for everyone. On that note, is anyone aware of any such platform that can be managed entirely from the command line? The one that comes closest is the github CLI, I think, and even that one still requires a browser flow for logging in and getting the token[2]. [1]: I am aware that I could dive into the code and maybe contribute this myself but a) there might be plans for this already; I believe there's an issue where it was discussed and b) I don't have nearly enough time to commit to it. [2]: I know that this is very nitpicky of me, but I do truly want a tool like this; I think it'd be awesome.

ddevault 6y ago
We have a shell for builds.sr.ht over SSH: https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/master/master-shell https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/master/master-... You can use it to, for example, do `ssh builds@builds.sr.ht submit < build.yml` Expaning this to support secret management would be cool.

vfclists 6y ago

The shell is a total mindfuck. Trying to do any non trivial scripting/programming in it will mess up your mind. Oilshell perhaps?

jariel 6y ago

I think we need a visual shell. Git should literally show you a tree, then an animation of what's happening because there's a lot going on and I was shocked at my ignorance once I did see something. Mac Finder is my goto a lot because it's just powerful to see the tree. Could we blend the command line there? And finally - for expert users this is not a problem, for for medium users, the fact that many of the commands have different flags in different orders is a pain. A brand new set of integrated commands - where everything followed a predictable standard would be great.

larkinrichards 6y ago

I spend many hours of each day ~programming~ wrangling text files and I use macos + zsh + textmate2 for my daily drivers. I see shell as an important proficiency because it helps maintain a lower-level understanding of how the "magical" GUI "works," which often helps in debugging obtuse errors, and I'm sad to encounter more engineers who are completely unfamiliar with it. When it comes to examples like that posted by the OP, I like the combination of piping/pasting to mate and multi-caret editing for most scenarios where others would reach for awk/xargs. Here's me following the same example scenario but with multi-caret editing (slowed down slightly): https://user-images.githubusercontent.com/226503/101993951-6480cb80-3c73-11eb-9608-c6912011954b.gif https://user-images.githubusercontent.com/226503/101993951-6... Step by step: 1. git status -s | mate 2. select " D" with arrow keys + shift 3. command-E macos default for "use selection to find" 4. option-command-F to find all (multi-caret editing starts) 5. type "git checkout" to replace " D" 6. command-left to move cursor to start of line, then shift-command-right to select to end of line. 7. copy 8. select all + delete (clear document) 9. paste, 10. press return to insert newlines 11. select all + copy 12. switch back to terminal 13. paste To me, this is many small steps, but each step is more mechanical and flows naturally, and the general flexibility of multi-caret editing means it is applicable more often in my daily work.

doppioandante 6y ago
This is exactly how I would have done it, just using my editor of choice (kakoune) instead. I also wrote on Drew's mailing list in case anyone is interested: https://lists.sr.ht/~sircmpwn/public-inbox/%3CCAKW6382rV5iW28YfyK%2BUsAfX3%2BLzYNQXetW3B2qd%3D7cZ5LjNDg%40mail.gmail.com%3E https://lists.sr.ht/~sircmpwn/public-inbox/%3CCAKW6382rV5iW2...

masa331 6y ago

I completely agree, this knowledge can be very powerful. I also want to play Celeste[0] because it looks beautiful. But i have to say lately i dislike posts written in this command tone. Like "Do this to become X"/"Use this technology for Y"/"Don't do Z". Most of the time i end up feeling bad for not knowing something and therefore not being 10x programmer or even feeling antipathy towards the topic or author as i have different views. But in reality these things often don't matter or rather it can vary a lot person by person and context by context. You just don't need to be a shell literate to be a successful programmer. I wonder if more subtle introductions aren't better. Some which would spark interest but wouldn't leave a person with a bad feeling if he can't get into it right away. [0] http://www.celestegame.com/ http://www.celestegame.com/

rakoo 6y ago

It still strikes me that after all this time, the next step in the direction of the author is to use an editor like ACME (https://en.wikipedia.org/wiki/Acme_(text_editor) https://en.wikipedia.org/wiki/Acme_(text_editor)): use the shell to its maximum power, where the output of a command can be edited and used to control the existing interface. Take for instance this video: https://www.youtube.com/watch?v=4djoOiLste0 https://www.youtube.com/watch?v=4djoOiLste0 The author shows how the output of "git status" is just a text, you can modify the text and add "git add" just in front of the files you care about and add the file to the index. There is no linear loop of command input that becomes command output, it's all one big buffer that can feed itself. Consider how there could be another window that perpetually shows the status thanks to a combination of inotify and git status, and you have a git view. Fiddle with the command line argument and you can choose wether to show whitespaces or not, whether to show a summary view or the full diff, or restrict the list to some files. Have another window where you can write some message, and a GitCommit command in the "Tag" of the window will use the whole buffer as a git commit message; boom, you have 60% of what I use git-cola for. A bit more information can be seen by one of its creator here: https://www.youtube.com/watch?v=dP1xVpMPn8M https://www.youtube.com/watch?v=dP1xVpMPn8M. The possibilities are truly endless, and I haven't seen anything that resembles it. There is just no editor that embraces your platform the way Acme does. No, Emacs is not the same, because Emacs doesn't integrate with your OS; Emacs is an OS unto itself. You can't really say it's integrated to the OS when everything is implemented in the language that only Emacs uses.

sime2009 6y ago
Your git example is very similar to what can be done in my terminal, [Extraterm](https://extraterm.org/ https://extraterm.org/). See demo here: https://extraterm.org/features.html#reusing-command-output https://extraterm.org/features.html#reusing-command-output Thanks for the tip about Acme. I'm always looking for ideas about how to improve integration between the terminal based work flows and other applications, command line, GUI, or others.
- rakoo 6y ago
  Wow, I've been looking for a terminal that does this sort of thing for a while, nice job! And it's beautiful !

intrepidhero 6y ago

This week I got my drill bit bound up in a double stud I wanted to run some wire through. The drill went into thermal cut out and wouldn't budge. I was at a loss of how to get the bit out without destroying the stud until I remembered I had a hand drill in the shop. It came from my great-grandfather. It's solid and well maintained and I was able to hand crank that drill bit through the rest of the wood. That's not the first time a simple hand tool has saved the day when a power tool let me down. I agree that IDEs and the tools that accompany them are powerful and can make you more productive. Learning them is an investment that can pay dividends. On the other hand, most of the IDEs I took the time to learn in school are now obsolete. I still use the shell today because those skills are still relevant and have gotten me out of a jam plenty of times. I've chosen to prioritize learning tools that are reliable and lasting even if it costs me some productivity.

gitowiec 6y ago

I agree that a programmer should know how to do things with shell, but when I read: > In my workflow, I use Vim as my editor, and Unix as my “IDE”. I immediately remember how great are Jetbrains IDEs. I just can't imagine how someone could refactor code with such accuracy and without hurdle just using Linux vi or any cli tool...

betwixthewires 6y ago

This guy keeps a Gemini site where he puts all of his articles as well. He's very intelligent and has an interesting perspective, I like his writing a lot. gemini://drewdevault.com/

xx789 6y ago

Sign me up for someone who hates the shell. I've always tried to write scripts in Python or even C++ is so much easier than bash, awk, etc. Bash has no repl, no unit tests not debugger its anachronistic. Don't get me started on emacs.

tharne 6y ago
> Bash has no repl, no unit tests not debugger Bash scripts are generally very short. If you need unit tests and a debugger you probably should be using something other than Bash. But saying you hate the shell because it's bad at something it was never designed for is silly.
- xx789 6y ago
  At work I regularly have to support 500+ line scripts written by random developers, I 'n hate them.
  - hiq 6y ago
    Yeah I think that most people would hate this situation too compared to, say, similar scripts written in Python. But as parent said, the wrong tool for the wrong job is rarely a pleasant experience.
hiq 6y ago
Bash has no REPL?
- xx789 6y ago
  yeah OK its probably the definition of repl.

kaszanka 6y ago

That pipeline sure does look convenient. Let's see how it will handle a path with spaces. $ git status -s | grep '^ D' | awk '{ print $2 }' | xargs git checkout -- xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option $ git status -s D "g h i" I'd be lying if I said I was surprised, to be honest.

LeoPanthera 6y ago
You can fix it by replacing the awk with "cut -d ' ' -f 2-" Haven't tested it, but you may also need to throw a "tr '\n' '\0'" in there and call xargs with "-0" to make it happy. This will still break on files with a CR in the filename. (CRs are allowed in unix filenames but, imho, they should not be.)
- megous 6y ago
  you also want to tell xargs not to run the command on an empty set, otherwise you'll just do 'git checkout --' which will just kill all your local changes, probably.
hiq 6y ago
I think people who routinely write this kind of commands in an interactive shell are well aware of the shortcomings, they just deal with it when they need to and not before. One example is that people very rarely handle '\n' or spaces in filenames (e.g. using the -0 option from `xargs`). Whatever works is fine as long as one knows the risks. For a one-off thing like the one mentioned in the article I'd say it's almost ok, although I'd put `git checkout --` in the category of risky commands if you don't have any backup.
patrec 6y ago
Yeah, it's annoying that unix filenames and shell quoting are both fundamentally broken and that (as above) people would generally (knowingly) write a a broken-for-spaces-in-filenames version by default because it's easier (and the author no doubt knew he'd not encounter spaces in his repo). Having said that, it's not very hard to fix, I'd probably write something like: git status -s | awk -vORS="\0" 'gsub(/^ D /,"")' | xargs -0 git checkout -- Which is about the same length. This will still break for malicious input (newlines in filename), but for the original use case that's not a concern.

effnorwood 6y ago

It’s a gas station morons

29athrowaway 6y ago

If you want to try something new, try "Facebook path picker" http://facebook.github.io/PathPicker/ http://facebook.github.io/PathPicker/ Pretty useful when paired with git. e.g.: git status | fpp -c git add

tommd 6y ago

And for goodness sakes, use ShellCheck! It can be easy (i.e. there are CI services that will run shellcheck automatically on your PRs and comment bugs).

hiq 6y ago

Nice post, but 2020 and still using an unauthenticated connection (plain netcat) to transfer data including executables (if I understood correctly) is not something you can justify when there are plenty of safe alternatives (e.g. wormhole, syncthing). Maybe the recipient and the sender just compare hashes via a secure channel, but there's no mention of this.

deleted 6y ago
[deleted]
ddevault 6y ago
The files were transferred via yggdrasil. In any case, it would have to be an unusually clever attacker to identify executables in a tarball being transmitted unwrapped via TCP on a port chosen out of a hat, and intercept it to do something malicious.
- Nextgrid 6y ago
  Malice is not the only concern. There was a post on here a couple days ago where AT&T was corrupting certain packets, and due to either luck of the TCP checksums being recalculated on the faulty equipment itself the bad packets were actually considered valid by the recipient.
- hiq 6y ago
  > The files were transferred via yggdrasil. I wish you would have at least mentioned it in your post that you were indeed relying on the security of the network and that nc by itself is not secure. I'm not saying it would have been a real risk in your particular case, but giving these examples on a public website is basically saying "see, you can do it like this". I'm not saying you don't know better, but some of your readers might not, and may replicate your unsafe (without the underlying assumptions) code where the risk is higher. This is especially bad given that there are safe alternatives with equal or better ergonomics.

wglb 6y ago

I do so much of this it would be hard to document all of them. In a very large lisp project, I had an astonishing number of lines of shell code--none more than 10 lines long. (Today I would do production automation with cl-readline and inferior-shell and uiop:launch-program. Also, given that I use emacs, I do lots with dired, and on odd occasions eshell. And I have gathered a bit of emacs lisp code as well.

et1337 6y ago

Shell is simultaneously wonderful and horrendous. Did you know that the left bracket ([) is actually not a part of bash syntax, but instead a binary that lives in /bin? Run “which [“ if you don’t believe me!

uluyol 6y ago
And other things which you'd expect to be commands are both commands AND builtins! And they behave differently depending on which you call! Some examples are time and pwd.

sebelk 6y ago

The new religion of devopers says that shell is obsolete...

kmstout 6y ago

The following sets up a key binding to grab interesting lines from the history, throw them into a file, and pop it up in an editor: # put interesting lines from history into a file, open an editor function h2e() { echo "#! /bin/bash" > "$3" history | tail -n"$1" | egrep "$2" | sed -re 's/\s*[0-9]+\s+//' >> "$3" "${EDITOR:-emacs}" "$3" } # prompt for h2e's arguments function h2ei() { read -p 'num. lines [1]: ' lines read -p 'pattern [.]: ' pattern read -p 'file [foo.sh]: ' fn h2e "${lines:-1}" "${pattern:-.}" "${fn:-foo.sh}" } # fix the terminal settings for the duration of a command, revert afterward function insane() { save=$(stty -g) stty sane "$@" stty $save } # bind F12 to our history grabber bind -x '"\e[24~":insane h2ei' It's been quite handy of late.

nickthemagicman 6y ago
So you would run "h2ei 10 somepattern out.txt"? Which would put the last 10 lines matching some pattern to the txt file?
- kmstout 6y ago
  Close--it's those lines matching some pattern within the last 10 lines. Think of it as localize, then filter. I've written it the other way, too, but this seems more natural to me. (Also, a direct h2e invocation would take the arguments from the command line; h2ei will ask every time.) In practice, I will do something, realize that the thing I just did over the last several minutes could be useful--or the core of something more general--and then strike F12 to start h2ei. At the prompts, I'll generally give it a nice round number like 10 or 20 or 50 lines, leave the default pattern, and set the filename to be something evocative in ~/bin. An editor will immediately open on my new shell script, and I can trim it down and continue hacking on it from there. The key idea is that I get an easier path from doing something ad hoc to creating something more useful in the long run.

megous 6y ago

Learning some kind of common shell language is certainly a knowledge that will serve you well in the widest amount of circumstances. From managing headless servers, to local use on a workstation, to scripting within your U-Boot bootloader, to reverse engineering Linux based IoT/devices that oftentimes include busybox. Shell is included in a lot of places.

mechnesium 6y ago

I don’t think this is necessarily true anymore. Today we have much more intuitive tools at our disposable such as vscode. I understand some people were raised on grep and vim, and it is like a second nature to them, but actively encouraging people to use these tools in 2020 in light of the alternatives is not pragmatic.

ddevault 6y ago
Much more intuitive, and much less useful.

voidhorse 6y ago

I used to think this way as well, but out of necessity, had to work specifically with a commercial IDE for some time. Turns out, if you are confident learning the keybindings of a robust IDE is worthwhile (e.g. you know that you must use it for some particular project for a decent amount of time) the investment pays off just as well as learning shell commands. A good IDE can do everything a cobbled together shell pipeline can, often in two keystrokes instead of N trial and error command inputs because you forgot, yet again that, e.g. while `grep` takes a pattern argument before a path, `find` does the opposite--in spite of POSIX there's an incredible lack of UX standardization when it comes to unix tooling. People like to pretend that learning shell commands is somehow better because it's more portable, but unless your work entails working across multiple machines and environments daily, it doesn't matter. Most of us do most of our work on one machine, in one environment, so at the end of the day, it really comes down to preference. Personally, I do prefer working in the shell, but if you prefer working in a modern IDE, don't let anyone fool you that you're somehow wasting time or being less proficient than you could be in a shell--if you took the time to learn the IDE keybindings and the IDE is on par with anything Jetbrains puts out, you're not losing in efficiency at all and possibly making gains when it comes to certain tasks.

dragonwriter 6y ago
> People like to pretend that learning shell commands is somehow better because it's more portable, No, it's better because if you aren't doing exactly the workflow an IDE or GUI tool designer has envisioned, it is almost invariably much easier to do it in shell (and then make it a script and then bind it to a command in your IDE or GUI tool, if they support that) than to beat the non-shell tool into, first, doing what you want, and then making it easily repeatable. It's also more portable, which, contrary to your description, is very useful for most people, because even if there are some dev-only tasks that “I can only do it in the IDE” is fine for, for many things you may also want to do in a CI pipeline, on a deployment box, or other places where your IDE isn't running. Shell scripts generally work there.
- tracyhenry 6y ago
  I use IDE mainly for code editing and exploration (e.g. jump from usage to definition), for which I can't see how shell is easier. Modern IDEs have shell built-in so when I finish coding, I can just open up a terminal inside IDE to run shell scripts. Effectively I'm getting the best of both worlds. I never knew that I have to follow some sort of rails laid by IDE designers, which I think it's a very common misconception by people who dislike IDEs.

TeeMassive 6y ago

I got hired in a new company a year ago and we are in the process of migrating from TFS to git. It's quite disheartening to see the lack of skill with a shell. I'm being blocked from pushing and promoting the use of something else than Visual Studio's git pluggin because "other developers are not used to the command line". I proposed Git Extensions and it was more or less the same feedback, too different. It feels like an electrician being told not to use oscilloscopes because it's too hard for others in his team. At least they're in the process of migrating and eventually people will see the light, CLIs are powerful because of their versatility and therefore will server you for most of your career, while "learning" and IDE will server you 10 years at most.

noisy_boy 6y ago

Beyond the point of using shell and scripts as replacement for IDE/development, knowing the shell also helps with general productivity. E.g. I need to remote to office desktop and instead of having to open my phone to read the 2FA pin, I just setup stoken on my desktop and wrote a simple script that invokes stoken and copies the pin to the clipboard. Mapped the script to a shortcut and now I can just press the shortcut from the login page and do Ctrl+V for pasting the pin. Small things like these become very easy once you know a bit of scripting and the time/effort saved over time adds up considerably.

kortilla 6y ago

> I can work with this. I add grep '^ D' to filter out any entries which were not deleted, and pipe it through awk '{ print $2 }' to extract just the filenames. Best hope your file names don’t have spaces in them. One of the downsides of the shell on Unix is that everything is usually text meant for humans so you have to spend an annoying amount of time dealing with delimiters that make it easy on the eyes but a pain in the ass.

hxegon 6y ago
couldn't agree more. waiting on nushell to mature so that rich data is the norm for shell stuff. learning to be comfy in shell is still worth it though. It's saved me, at minimum, hundreds of hours of work.

jagged-chisel 6y ago

Can we at least agree on one thing? That sticking with the keyboard is faster than using the mouse. Use keyboard shortcuts in your IDE, type commands into your shell. The killer idea that will make you more efficient is scripting (or creating macros in your IDE) for your own common tasks and assigning them a good name (or a shortcut in your IDE.)