6 ms·

Robert Elder's Guide to GNU Coreutils

199 points by signa11 2y ago

kamaal 2y ago
Every programmer worth their salt must learn Perl, Bash and using these command line utils. There are chances you are writing lots of code which just doesn't have to get written. vi, is something that you must absolutely learn. Learning vi(especially with macros) will give you new perspectives about thinking and working with plain text in ways you might not have imagined before. Most of it might be one time work, but it generally saves you lots of spreadsheet and scripting work. Many times you even realise you were doing lots of pointless work over the years. Perl is the next progression in this path, and this will give you full code paradigm to do with anything Unix and text. Like so as long your application runs on Unixy operating systems, Perl can do it. You can often ship just one file to the server, be guaranteed it will run with backwards compatibility, it will run fast. You can also develop fast and change at will. Learning bash gives you means to dump a series a commands into a file and repeating them. But many times what saves you time and effort is learning to use for loop on the command line. This helps you do a lot of bulk work in repetition.
- synergy20 2y ago
  agree all but perl, I found perl hard to write and hard to read
- forrestthewoods 2y ago
  > Every programmer worth their salt must learn Perl, Bash and using these command line utils. Naw. As a game and VR programmer I only barely need to acknowledge that POSIX exists. These tools are not even Top 100 in usefulness.
  - pxc 2y ago
    When I was a student, I once interned for a company of Windows developers, a Microsoft shop whose work was all in .NET. I remember that there were some tasks I was able to do, having to do with processing many, many little text files, that everyone was surprised I was able to do, or surprised I was able to do quickly. Every time, it was something that I put together in a few minutes with the usual Unix tools thanks to Cygwin. It's probably true that game development is really different from, say, web development, when it comes to the relevance of Unix. But I think it's also true that often, when you don't know common Unix utils, you don't know what you're missing. I don't think any programmer could master them and regret it.
    - dale_glass 2y ago
      These days, Windows caught up very well with PowerShell. Actually more UNIX-y than UNIX in some respects.
      - pxc 2y ago
        I love PowerShell, but I've yet to meet someone who 'lives in' PowerShell the way so many 'live in' Bash and the various GNU CLI utilities, which I think inhibits familiarity of the kind most convenient for whipping up quick scripts like that. That said, PowerShell is much more a 'real programming language' than the shell languages that came before it. PowerShell scripts are a lot more maintainable that scripts written for those other shells. And there are some really powerful things built on top of shell (like DSC, for example) that don't to my knowledge have analogues written directly in other shell languages. Plus PowerShell has an actual package management story, code signing capabilities, and other modern niceties. But for me and I suspect most others, it falls down as an interactive environment for rwasons of performance and verbosity, and Windows environments themselves are unfortunately still harder to automate or inspect programmatically because most applications don't provide or document PowerShell-friendly interfaces. So I don't think it's surprising that despite all of PowerShell's incredible advancemwnts, you rarely meet Windows users who have invested in it as much as the average longtime Unix user invests in their respective CLI utilities. But for, say, game developers who say 'Unix is not my environment; I'm not sure I wanna bother with all of that CLI and scripting stuff', they do nowadays have an alternative in that arena which is outstanding in a lot of positive ways. I don't anyone would regret deeply learning PowerShell, either. Hell, my team at work is all Unix guys. I probably have the most experience PowerShell experience of any of us, which is way less than my GNU command line environment experience. But if someone with deep PowerShell experience joined our team, I'd absolutely encourage them to use PowerShell on Linux wherever we write scripts, and be happy to learn whatever I needed to to keep up. PowerShell is a nice enough language that I think it's very reasonable to bring it with you to other operating systems as well.
    - forrestthewoods 2y ago
      I don't regret knowing Unix tools. But I very frequently regret that other people chose to write non-trivial operations in a bash script instead of a real programming language! If Unix utils didn't exist and everyone just used Python instead I'd honestly probably prefer that.
- RodgerTheGreat 2y ago
  Eh; Perl just feels to me like a cancerously overgrown AWK. For simple tasks, AWK is tremendously more elegant. For anything too complex for AWK- which is a rather high ceiling- there are a whole host of scripting languages I'd select before Perl. AWK is available in some form on many more posix-ish environments than Perl and the most modern implementations are lightning-fast.
  - kamaal 2y ago
    To make awk work for non trivial manipulations you have use cat, awk, | and > in several sequences. This is not by any means elegant. What happens when there is a line that needs some error checking and handled a little differently? Or just a bad line? Once you realise that error tracing, and fixing is just not possible for non trivial stuff you are better off now skipping the entire ceremony and simply use Perl. As matter of fact, Larry Wall invented Perl while trying to the very same things you mention with C and awk. Then you realise you are dealing with so many adhoc things, without dependency checks, error handling and then you actually need a fully blown programming language to do this sort of work.
    - RodgerTheGreat 2y ago
      I've written AWK programs that perform decidedly non-trivial manipulations on data, and none of the problems you describe are real. It is absolutely not the case that one must shell out to a pipeline of other commands to get work done; you can write structured programs in AWK just as you would in any other scripting language, and operate in multiple passes over data. It's a very amusing irony if a Perl enthusiast genuinely believes that AWK can only be used in the form of cryptic one-liners, especially in the context of POSIX tools one "must" learn.
- petepete 2y ago
  Didn't realise vi had macros, I thought they were a more recent addition.
  - kamaal 2y ago
    vi macros are those magical things that one must just know. Once you know how to use them, its really one those things with regards to text manipulation that helps you convert manual O(n^k) tasks to O(1) task. And a lot of things in our industry can be reduced to text manipulation. For the next steps: Perl.
    - petepete 2y ago
      vi is not vim. I don't think vi has macros.
      - kamaal 2y ago
        vi does have it. https://math.berkeley.edu/~gbergman/misc/hacks/vi.html https://math.berkeley.edu/~gbergman/misc/hacks/vi.html
      - fuzztester 2y ago
        vi does have macros. it had it from at least 1990 or so (when i first used vi), and possibly from much earlier.
- yjftsjthsd-h 2y ago
  > Every programmer worth their salt must learn Perl, Bash and using these command line utils. There are chances you are writing lots of code which just doesn't have to get written. I have written a rather lot of sh (I target POSIX sh, not bash) and I'm decently familiar with common cli tools (coreutils, moreutils, etc.). IME awk+sed cover most of the "fancy" text manipulation just fine; what am I missing out on if I skip perl?
  - kamaal 2y ago
    Perl was made to ensure you don't have to learn the dozens of things you'd have to learn and slap together with all sorts of adhoc hacks. Everything comes in one package. The full language can't be captured in this comment, of course. But compared to Bash, you get a C like syntax(Bash comes from Algol, you have to learn a new syntax), you get error handling, full master set regexes, heredocs, qw, associative arrays, arrays, built-in string manipulation, unix interfaces, DB drivers, first class integration with command line tooling etc etc. Think of it like a unix native language. Like the immediate extension to the operating system itself. Apart from that Perl is very stable, fast, and very deeply committed to backwards compatibility. You can be sure the scripts you wrote years back will run on new Perls. awk+sed are simple match-do engines, you can't do counting(atleast not easily), you can't do other myriad stuff like working with >2-3 files at one time(iterating simultaneously, in sub-loops etc). If you are dealing with large files, in Perl its fairly simple to build a index by doing one go on the whole file, then use the key to map to the value where you want to go in the file using seek/tell(with pack/unpack), This makes frequent look ups very computationally cheap. In awk/sed you can't do error handling, you can't do reading and writing to multiple files at the same time et al easily atleast. In Perl, you get dbm drivers and functions install so its relatively straight forward to build look up applications of all kinds. Perl also comes with OO, so if your code gets lengthier you can can fit it into neat design patterns. Last but not the least. Perl has CPAN. Its just one of those things. Only real competition to CPAN till date is NPM. Not even Java/Python have a library ecosystem as large as Perl.
  - mbivert 2y ago
    IME (used Perl daily at work for years) not much. If you need something more robust or sophisticated than sh(1)& cie, it's best to head up to a statically typed language instead -- I tend to favor go, in part because of the cross-compilable fat binary thing. The lack of static typing is especially annoying when things are getting beefy enough to consider moving away from sh(1). For prototyping, if you know Perl already, why not; but even there, I personally reach out to sh(1) & friends instead.
- nox101 2y ago
  Perl is really really bad. Bash is also really really bad. Perl is unreadable and bash is massively insecure unless you chant the magic incantations which even the pros forget. At most you should learn Perl to learn now not to design a language and then you should learn bash to learn why such a large percentage of systems are vulnerable to various injections. Even Apple had a bash error where they had an OS upgrade that deleted your hard drive if you had a space in the volume name which I'm making the assumption was a bash script because spaces in filename don't generally matter in any sane language.
  - BoingBoomTschak 2y ago
    That's true, but shellcheck exists, at least.
  - kamaal 2y ago
    Hating Perl is some what like hating vim, or emacs, or lisp macros. You can live without it. But you will do a lot of pointless work over long periods of time. There are a lots of people who are ok with that. Me personally, I attempt to live a life where I can do O(1) or O(as little steps it takes) in anything I touch. I stubbornly refuse to do work that can be done using a computer.
    - nox101 2y ago
      I wrote an entire blogging system in Perl. knowing perl doesn't make me want to use it. Rather it makes me want to run away screaming. it doesn't compare to vim, emacs, or lisp. Perl is objectively bad
      - oguz-ismail 2y ago
        >Perl is objectively bad Nah, I think it's fine.
        tempfile 2y ago
        perhaps it's objectively bad, but subjectively good?
- froh 2y ago
  perl I'd agreed with up to somewhere 2010. Python it is, these days, for maintainability. one shot quick regex things? sed. anything else python. and I don't say "switch", btw, if you're proficient I there is no value and no need ditching it. however if you're new to the CLI game, then I'd _not_ suggest perl but python instead.
  - kamaal 2y ago
    Python unfortunately still doesn't provide a lot of things like heredocs, qw, multiline lambdas, many other unixy things. Its regex capabilities are also not Perl standard. Scripting is a lot about text, but its not only about text. You need to neatly integrate with many other OS things. More importantly the language needs to be designed such that such tasks can be done with ease. Python is more like good a small application programming language. Not that great if you do heavy OS, text or command line interaction work. It feels like a less flexible Java in that regard.
    - fluidcruft 2y ago
      My experience (switching from heavy use of shell) is Python is really good at text processing and building data structure very quickly. Python is a bit wordy for small things but it become extremely concice and reusable once things get a tiny bit larger. Generally I find doing things in Python is straightforward whereas using shell is more of a low level puzzle game of conversions and reformats and fragility. The puzzles in Python are higher level using better data types and containers. But also it's much easier to wrap things up into modules and reuse code in Python. And yeah Python is slow... but not compared to bash (unless you're using things like parallel, but Python isn't necessarily horrible for that either) That's been my experience anyway.
- mofosyne 2y ago
  One issue might be discoverability of these programs
- pjmlp 2y ago
  The only reason I care about vi, is that back in the day it would be only sane thing installed by default in many UNIX boxes, unless I would like to have some fun with ed. And until IDEs became common in UNIX land, I would rather be on Emacs camp.
SuperNinKenDo 2y ago
Robert Elder's Guide to GNU Coreutils is my favourite guide to GNU Coreutils.
- robertelder 2y ago
  This is my favourite comment.
forrestthewoods 2y ago
> To name a few them this includes cat, chmod, chown, cp, date, df, du, ln, ls, mkdir, mv, od, pr, rm, rmdir, stat, su, sum, tty, wc, and who. C programmers really need to be banned from naming things. Naming things is a certified “hard problem”… but it’s not that hard!
- vincent-manis 2y ago
  Actually, I'm very glad that Thompson, Ritchie, and friends used these strange names for the commands they implemented (supposedly because they didn't like to type long words). This leaves longer, more meaningful names open for me to use in naming commands I write. In any case, (a) UNIX (as it was spelled then) antedates C, so it's more that the UNIX spelling conventions influenced C's development; and (b) lots of earlier and later programming languages have had mystical abbreviations. For example, ALGOL68 replaced “integer” with “int”, for reasons I have never understood; Wirth's original CDC 6600 had a data type “alfa”, which was a 10-character string; William Wulf's Bliss language had library routines with names like “getnum” rather than “get_number”. In PL/I, you can write “proc” for procedure, and “dec” for “decimal”. Me, I prefer to program in a language that has primitive procedure names like “call-with-current-continuation”.
  - SuperNinKenDo 2y ago
    That seems backwards to me, short commands should be where users place their daily workflow of aliases. Longer names should be where the kind of static, shell scripting "API" sits. Imagine a world where we could set 'mv' and 'ls' to something other than the `mv` and `ls` binaries without breaking our systems...
    - AyyEye 2y ago
      That's the NixOS world.
  - Aloha 2y ago
    You must really like powershell then. I love most of the language, but I'm convinced it was written by people who type 300 wpm.
nick__m 2y ago
The comm command is my favorite lesser know coreutils command https://blog.robertelder.org/intro-to-comm-command/ https://blog.robertelder.org/intro-to-comm-command/
jeffbee 2y ago
Brilliant to begin every video with "blah is my favorite command." I watched the video for 'paste' first and I thought it wonderful that someone else also thinks paste is their favorite.
- robertelder 2y ago
  This is my favorite observation.
- jftuga 2y ago
  I like using paste to convert a list of lines into a single line, comma-delimited list... Contrived example, for brevity: awk 'BEGIN { while (i < 10) print i++ }' | paste -sd, - or maybe: ls *txt | paste -sd, -
  - paradox460 2y ago
    Fwiw you can do that with just awk, no pipe and paste required
  - maleldil 2y ago
    I like using paste to sum over numbers where each number is in a distinct line. E.g.: printf "1\n2\n3" | paste -sd+ | bc 6 I suppose you could do that more cleanly with awk, but this has worked whenever I needed it. I find it easier to type.
  - tacostakohashi 2y ago
    I'd just do that with tr '\n' , It's fascinating that there are so many different ways to do things, even within the limited confines of coreutils!
  - monista 2y ago
    seq 10 | paste -sd, -
asicsp 2y ago
I wrote a book focusing only on text processing commands (includes exercises as well): https://learnbyexample.github.io/cli_text_processing_coreutils/ https://learnbyexample.github.io/cli_text_processing_coreuti... You can practice some of the exercises using this interactive TUI app: https://github.com/learnbyexample/TUI-apps/tree/main/CLI-Exercises https://github.com/learnbyexample/TUI-apps/tree/main/CLI-Exe... See also: https://ratfactor.com/slackware/pkgblog/coreutils https://ratfactor.com/slackware/pkgblog/coreutils, https://maizure.org/projects/decoded-gnu-coreutils/ https://maizure.org/projects/decoded-gnu-coreutils/ and https://www.pixelbeat.org/docs/coreutils-testing.html https://www.pixelbeat.org/docs/coreutils-testing.html
- r366y6 2y ago
  Awesome! Thank you very much :)
azalemeth 2y ago
Some of these examples are genuinely hilarious -- I honestly laughed aloud at the example file given for csplit (https://blog.robertelder.org/intro-to-csplit-command/ https://blog.robertelder.org/intro-to-csplit-command/), one command I don't routinely use but now shall certainly do so more...
- fuzztester 2y ago
  ha ha, i laughed at that one too, the "procrastinate on work until the last day" example. csplit stand for context split, iirc. there is also the split command. it splits on the basis of number of lines, which can be specified in various ways. didn't know about the '--filter' flag earlier. it allows you to filter each output file through some other command. useful. https://blog.robertelder.org/intro-to-split-command/ https://blog.robertelder.org/intro-to-split-command/ https://man7.org/linux/man-pages/man1/split.1.html https://man7.org/linux/man-pages/man1/split.1.html
- tveyben 2y ago
  Indeed they are - look at the EXIT turorial, just awesome…!!!! I gotta reserve a weekend to go through this vast and humoristisk set of knowledge. Thank you Robert!
- fluidcruft 2y ago
  https://blog.robertelder.org/intro-to-runcon-command/ https://blog.robertelder.org/intro-to-runcon-command/ was entertaining
stabbles 2y ago
It's funny that it uses `xxd` in the example here [1] echo "Hello World!" | xxd which is not part of coreutils ... but vim [2] confusingly. But also hexdump is not part of coreutils. They have `od`, which nobody uses because it looks like this: echo "Hello World!" | od -A x -t x1z -v [1] https://blog.robertelder.org/intro-to-mkfifo-command/ https://blog.robertelder.org/intro-to-mkfifo-command/ [2] https://github.com/vim/vim/blob/d33afe12c6639d70fca82230df6b9fdee7365423/src/xxd/xxd.c https://github.com/vim/vim/blob/d33afe12c6639d70fca82230df6b...
haunter 2y ago
Hard to tell where is the border between the shell and the utility with [
blueflow 2y ago
Want to get to the real source of truth for your system? Do `ls /usr/share/man/man1/`[1] to list the commands, and then use `man 1 $command` The manpage is opened with your $PAGER, so you can use your favourite search and navigation commands. Ideally your system has a `man 1 intro` to guide you to things. [1] Paths might be different for Nix or MacOS users
- kryptiskt 2y ago
  It's a bit ironic since the GNU policy is to prefer info to man pages. But I have to agree that man is a lot better for practical purposes.
k3vinw 2y ago
Piping tail to grep is another example I was hoping to see. A simple use case would be to show a filtered live stream of incoming error messages printed to a log.
deleted 2y ago
[deleted]
fourier54 2y ago
This is my favourite HN post
deleted 2y ago
[deleted]
classified 2y ago
Videos are the worst possible medium to learn about command line tools. But if all you have is a hammer...

199 points by signa11 2y ago

Every programmer worth their salt must learn Perl, Bash and using these command line utils. There are chances you are writing lots of code which just doesn't have to get written. vi, is something that you must absolutely learn. Learning vi(especially with macros) will give you new perspectives about thinking and working with plain text in ways you might not have imagined before. Most of it might be one time work, but it generally saves you lots of spreadsheet and scripting work. Many times you even realise you were doing lots of pointless work over the years. Perl is the next progression in this path, and this will give you full code paradigm to do with anything Unix and text. Like so as long your application runs on Unixy operating systems, Perl can do it. You can often ship just one file to the server, be guaranteed it will run with backwards compatibility, it will run fast. You can also develop fast and change at will. Learning bash gives you means to dump a series a commands into a file and repeating them. But many times what saves you time and effort is learning to use for loop on the command line. This helps you do a lot of bulk work in repetition.

synergy20 2y ago
agree all but perl, I found perl hard to write and hard to read
forrestthewoods 2y ago
> Every programmer worth their salt must learn Perl, Bash and using these command line utils. Naw. As a game and VR programmer I only barely need to acknowledge that POSIX exists. These tools are not even Top 100 in usefulness.
- pxc 2y ago
  When I was a student, I once interned for a company of Windows developers, a Microsoft shop whose work was all in .NET. I remember that there were some tasks I was able to do, having to do with processing many, many little text files, that everyone was surprised I was able to do, or surprised I was able to do quickly. Every time, it was something that I put together in a few minutes with the usual Unix tools thanks to Cygwin. It's probably true that game development is really different from, say, web development, when it comes to the relevance of Unix. But I think it's also true that often, when you don't know common Unix utils, you don't know what you're missing. I don't think any programmer could master them and regret it.
  - dale_glass 2y ago
    These days, Windows caught up very well with PowerShell. Actually more UNIX-y than UNIX in some respects.
    - pxc 2y ago
      I love PowerShell, but I've yet to meet someone who 'lives in' PowerShell the way so many 'live in' Bash and the various GNU CLI utilities, which I think inhibits familiarity of the kind most convenient for whipping up quick scripts like that. That said, PowerShell is much more a 'real programming language' than the shell languages that came before it. PowerShell scripts are a lot more maintainable that scripts written for those other shells. And there are some really powerful things built on top of shell (like DSC, for example) that don't to my knowledge have analogues written directly in other shell languages. Plus PowerShell has an actual package management story, code signing capabilities, and other modern niceties. But for me and I suspect most others, it falls down as an interactive environment for rwasons of performance and verbosity, and Windows environments themselves are unfortunately still harder to automate or inspect programmatically because most applications don't provide or document PowerShell-friendly interfaces. So I don't think it's surprising that despite all of PowerShell's incredible advancemwnts, you rarely meet Windows users who have invested in it as much as the average longtime Unix user invests in their respective CLI utilities. But for, say, game developers who say 'Unix is not my environment; I'm not sure I wanna bother with all of that CLI and scripting stuff', they do nowadays have an alternative in that arena which is outstanding in a lot of positive ways. I don't anyone would regret deeply learning PowerShell, either. Hell, my team at work is all Unix guys. I probably have the most experience PowerShell experience of any of us, which is way less than my GNU command line environment experience. But if someone with deep PowerShell experience joined our team, I'd absolutely encourage them to use PowerShell on Linux wherever we write scripts, and be happy to learn whatever I needed to to keep up. PowerShell is a nice enough language that I think it's very reasonable to bring it with you to other operating systems as well.
  - forrestthewoods 2y ago
    I don't regret knowing Unix tools. But I very frequently regret that other people chose to write non-trivial operations in a bash script instead of a real programming language! If Unix utils didn't exist and everyone just used Python instead I'd honestly probably prefer that.
RodgerTheGreat 2y ago
Eh; Perl just feels to me like a cancerously overgrown AWK. For simple tasks, AWK is tremendously more elegant. For anything too complex for AWK- which is a rather high ceiling- there are a whole host of scripting languages I'd select before Perl. AWK is available in some form on many more posix-ish environments than Perl and the most modern implementations are lightning-fast.
- kamaal 2y ago
  To make awk work for non trivial manipulations you have use cat, awk, | and > in several sequences. This is not by any means elegant. What happens when there is a line that needs some error checking and handled a little differently? Or just a bad line? Once you realise that error tracing, and fixing is just not possible for non trivial stuff you are better off now skipping the entire ceremony and simply use Perl. As matter of fact, Larry Wall invented Perl while trying to the very same things you mention with C and awk. Then you realise you are dealing with so many adhoc things, without dependency checks, error handling and then you actually need a fully blown programming language to do this sort of work.
  - RodgerTheGreat 2y ago
    I've written AWK programs that perform decidedly non-trivial manipulations on data, and none of the problems you describe are real. It is absolutely not the case that one must shell out to a pipeline of other commands to get work done; you can write structured programs in AWK just as you would in any other scripting language, and operate in multiple passes over data. It's a very amusing irony if a Perl enthusiast genuinely believes that AWK can only be used in the form of cryptic one-liners, especially in the context of POSIX tools one "must" learn.
petepete 2y ago
Didn't realise vi had macros, I thought they were a more recent addition.
- kamaal 2y ago
  vi macros are those magical things that one must just know. Once you know how to use them, its really one those things with regards to text manipulation that helps you convert manual O(n^k) tasks to O(1) task. And a lot of things in our industry can be reduced to text manipulation. For the next steps: Perl.
  - petepete 2y ago
    vi is not vim. I don't think vi has macros.
    - kamaal 2y ago
      vi does have it. https://math.berkeley.edu/~gbergman/misc/hacks/vi.html https://math.berkeley.edu/~gbergman/misc/hacks/vi.html
    - fuzztester 2y ago
      vi does have macros. it had it from at least 1990 or so (when i first used vi), and possibly from much earlier.
yjftsjthsd-h 2y ago
> Every programmer worth their salt must learn Perl, Bash and using these command line utils. There are chances you are writing lots of code which just doesn't have to get written. I have written a rather lot of sh (I target POSIX sh, not bash) and I'm decently familiar with common cli tools (coreutils, moreutils, etc.). IME awk+sed cover most of the "fancy" text manipulation just fine; what am I missing out on if I skip perl?
- kamaal 2y ago
  Perl was made to ensure you don't have to learn the dozens of things you'd have to learn and slap together with all sorts of adhoc hacks. Everything comes in one package. The full language can't be captured in this comment, of course. But compared to Bash, you get a C like syntax(Bash comes from Algol, you have to learn a new syntax), you get error handling, full master set regexes, heredocs, qw, associative arrays, arrays, built-in string manipulation, unix interfaces, DB drivers, first class integration with command line tooling etc etc. Think of it like a unix native language. Like the immediate extension to the operating system itself. Apart from that Perl is very stable, fast, and very deeply committed to backwards compatibility. You can be sure the scripts you wrote years back will run on new Perls. awk+sed are simple match-do engines, you can't do counting(atleast not easily), you can't do other myriad stuff like working with >2-3 files at one time(iterating simultaneously, in sub-loops etc). If you are dealing with large files, in Perl its fairly simple to build a index by doing one go on the whole file, then use the key to map to the value where you want to go in the file using seek/tell(with pack/unpack), This makes frequent look ups very computationally cheap. In awk/sed you can't do error handling, you can't do reading and writing to multiple files at the same time et al easily atleast. In Perl, you get dbm drivers and functions install so its relatively straight forward to build look up applications of all kinds. Perl also comes with OO, so if your code gets lengthier you can can fit it into neat design patterns. Last but not the least. Perl has CPAN. Its just one of those things. Only real competition to CPAN till date is NPM. Not even Java/Python have a library ecosystem as large as Perl.
- mbivert 2y ago
  IME (used Perl daily at work for years) not much. If you need something more robust or sophisticated than sh(1)& cie, it's best to head up to a statically typed language instead -- I tend to favor go, in part because of the cross-compilable fat binary thing. The lack of static typing is especially annoying when things are getting beefy enough to consider moving away from sh(1). For prototyping, if you know Perl already, why not; but even there, I personally reach out to sh(1) & friends instead.
nox101 2y ago
Perl is really really bad. Bash is also really really bad. Perl is unreadable and bash is massively insecure unless you chant the magic incantations which even the pros forget. At most you should learn Perl to learn now not to design a language and then you should learn bash to learn why such a large percentage of systems are vulnerable to various injections. Even Apple had a bash error where they had an OS upgrade that deleted your hard drive if you had a space in the volume name which I'm making the assumption was a bash script because spaces in filename don't generally matter in any sane language.
- BoingBoomTschak 2y ago
  That's true, but shellcheck exists, at least.
- kamaal 2y ago
  Hating Perl is some what like hating vim, or emacs, or lisp macros. You can live without it. But you will do a lot of pointless work over long periods of time. There are a lots of people who are ok with that. Me personally, I attempt to live a life where I can do O(1) or O(as little steps it takes) in anything I touch. I stubbornly refuse to do work that can be done using a computer.
  - nox101 2y ago
    I wrote an entire blogging system in Perl. knowing perl doesn't make me want to use it. Rather it makes me want to run away screaming. it doesn't compare to vim, emacs, or lisp. Perl is objectively bad
    - oguz-ismail 2y ago
      >Perl is objectively bad Nah, I think it's fine.
      - tempfile 2y ago
        perhaps it's objectively bad, but subjectively good?
froh 2y ago
perl I'd agreed with up to somewhere 2010. Python it is, these days, for maintainability. one shot quick regex things? sed. anything else python. and I don't say "switch", btw, if you're proficient I there is no value and no need ditching it. however if you're new to the CLI game, then I'd _not_ suggest perl but python instead.
- kamaal 2y ago
  Python unfortunately still doesn't provide a lot of things like heredocs, qw, multiline lambdas, many other unixy things. Its regex capabilities are also not Perl standard. Scripting is a lot about text, but its not only about text. You need to neatly integrate with many other OS things. More importantly the language needs to be designed such that such tasks can be done with ease. Python is more like good a small application programming language. Not that great if you do heavy OS, text or command line interaction work. It feels like a less flexible Java in that regard.
  - fluidcruft 2y ago
    My experience (switching from heavy use of shell) is Python is really good at text processing and building data structure very quickly. Python is a bit wordy for small things but it become extremely concice and reusable once things get a tiny bit larger. Generally I find doing things in Python is straightforward whereas using shell is more of a low level puzzle game of conversions and reformats and fragility. The puzzles in Python are higher level using better data types and containers. But also it's much easier to wrap things up into modules and reuse code in Python. And yeah Python is slow... but not compared to bash (unless you're using things like parallel, but Python isn't necessarily horrible for that either) That's been my experience anyway.
mofosyne 2y ago
One issue might be discoverability of these programs
pjmlp 2y ago
The only reason I care about vi, is that back in the day it would be only sane thing installed by default in many UNIX boxes, unless I would like to have some fun with ed. And until IDEs became common in UNIX land, I would rather be on Emacs camp.

SuperNinKenDo 2y ago

Robert Elder's Guide to GNU Coreutils is my favourite guide to GNU Coreutils.

robertelder 2y ago
This is my favourite comment.

forrestthewoods 2y ago

> To name a few them this includes cat, chmod, chown, cp, date, df, du, ln, ls, mkdir, mv, od, pr, rm, rmdir, stat, su, sum, tty, wc, and who. C programmers really need to be banned from naming things. Naming things is a certified “hard problem”… but it’s not that hard!

vincent-manis 2y ago
Actually, I'm very glad that Thompson, Ritchie, and friends used these strange names for the commands they implemented (supposedly because they didn't like to type long words). This leaves longer, more meaningful names open for me to use in naming commands I write. In any case, (a) UNIX (as it was spelled then) antedates C, so it's more that the UNIX spelling conventions influenced C's development; and (b) lots of earlier and later programming languages have had mystical abbreviations. For example, ALGOL68 replaced “integer” with “int”, for reasons I have never understood; Wirth's original CDC 6600 had a data type “alfa”, which was a 10-character string; William Wulf's Bliss language had library routines with names like “getnum” rather than “get_number”. In PL/I, you can write “proc” for procedure, and “dec” for “decimal”. Me, I prefer to program in a language that has primitive procedure names like “call-with-current-continuation”.
- SuperNinKenDo 2y ago
  That seems backwards to me, short commands should be where users place their daily workflow of aliases. Longer names should be where the kind of static, shell scripting "API" sits. Imagine a world where we could set 'mv' and 'ls' to something other than the `mv` and `ls` binaries without breaking our systems...
  - AyyEye 2y ago
    That's the NixOS world.
- Aloha 2y ago
  You must really like powershell then. I love most of the language, but I'm convinced it was written by people who type 300 wpm.

nick__m 2y ago

The comm command is my favorite lesser know coreutils command https://blog.robertelder.org/intro-to-comm-command/ https://blog.robertelder.org/intro-to-comm-command/

jeffbee 2y ago

Brilliant to begin every video with "blah is my favorite command." I watched the video for 'paste' first and I thought it wonderful that someone else also thinks paste is their favorite.

robertelder 2y ago
This is my favorite observation.
jftuga 2y ago
I like using paste to convert a list of lines into a single line, comma-delimited list... Contrived example, for brevity: awk 'BEGIN { while (i < 10) print i++ }' | paste -sd, - or maybe: ls *txt | paste -sd, -
- paradox460 2y ago
  Fwiw you can do that with just awk, no pipe and paste required
- maleldil 2y ago
  I like using paste to sum over numbers where each number is in a distinct line. E.g.: printf "1\n2\n3" | paste -sd+ | bc 6 I suppose you could do that more cleanly with awk, but this has worked whenever I needed it. I find it easier to type.
- tacostakohashi 2y ago
  I'd just do that with tr '\n' , It's fascinating that there are so many different ways to do things, even within the limited confines of coreutils!
- monista 2y ago
  seq 10 | paste -sd, -

asicsp 2y ago

I wrote a book focusing only on text processing commands (includes exercises as well): https://learnbyexample.github.io/cli_text_processing_coreutils/ https://learnbyexample.github.io/cli_text_processing_coreuti... You can practice some of the exercises using this interactive TUI app: https://github.com/learnbyexample/TUI-apps/tree/main/CLI-Exercises https://github.com/learnbyexample/TUI-apps/tree/main/CLI-Exe... See also: https://ratfactor.com/slackware/pkgblog/coreutils https://ratfactor.com/slackware/pkgblog/coreutils, https://maizure.org/projects/decoded-gnu-coreutils/ https://maizure.org/projects/decoded-gnu-coreutils/ and https://www.pixelbeat.org/docs/coreutils-testing.html https://www.pixelbeat.org/docs/coreutils-testing.html

r366y6 2y ago
Awesome! Thank you very much :)

azalemeth 2y ago

Some of these examples are genuinely hilarious -- I honestly laughed aloud at the example file given for csplit (https://blog.robertelder.org/intro-to-csplit-command/ https://blog.robertelder.org/intro-to-csplit-command/), one command I don't routinely use but now shall certainly do so more...

fuzztester 2y ago
ha ha, i laughed at that one too, the "procrastinate on work until the last day" example. csplit stand for context split, iirc. there is also the split command. it splits on the basis of number of lines, which can be specified in various ways. didn't know about the '--filter' flag earlier. it allows you to filter each output file through some other command. useful. https://blog.robertelder.org/intro-to-split-command/ https://blog.robertelder.org/intro-to-split-command/ https://man7.org/linux/man-pages/man1/split.1.html https://man7.org/linux/man-pages/man1/split.1.html
tveyben 2y ago
Indeed they are - look at the EXIT turorial, just awesome…!!!! I gotta reserve a weekend to go through this vast and humoristisk set of knowledge. Thank you Robert!
fluidcruft 2y ago
https://blog.robertelder.org/intro-to-runcon-command/ https://blog.robertelder.org/intro-to-runcon-command/ was entertaining

stabbles 2y ago

It's funny that it uses `xxd` in the example here [1] echo "Hello World!" | xxd which is not part of coreutils ... but vim [2] confusingly. But also hexdump is not part of coreutils. They have `od`, which nobody uses because it looks like this: echo "Hello World!" | od -A x -t x1z -v [1] https://blog.robertelder.org/intro-to-mkfifo-command/ https://blog.robertelder.org/intro-to-mkfifo-command/ [2] https://github.com/vim/vim/blob/d33afe12c6639d70fca82230df6b9fdee7365423/src/xxd/xxd.c https://github.com/vim/vim/blob/d33afe12c6639d70fca82230df6b...

haunter 2y ago

Hard to tell where is the border between the shell and the utility with [

blueflow 2y ago

Want to get to the real source of truth for your system? Do `ls /usr/share/man/man1/`[1] to list the commands, and then use `man 1 $command` The manpage is opened with your $PAGER, so you can use your favourite search and navigation commands. Ideally your system has a `man 1 intro` to guide you to things. [1] Paths might be different for Nix or MacOS users

kryptiskt 2y ago
It's a bit ironic since the GNU policy is to prefer info to man pages. But I have to agree that man is a lot better for practical purposes.

k3vinw 2y ago

Piping tail to grep is another example I was hoping to see. A simple use case would be to show a filtered live stream of incoming error messages printed to a log.

deleted 2y ago

[deleted]

fourier54 2y ago

This is my favourite HN post

deleted 2y ago

[deleted]

classified 2y ago

Videos are the worst possible medium to learn about command line tools. But if all you have is a hammer...