5 ms·

That's correct. Articles/tutorials or an evangelizing fan often show the end result: the cool command/pipeline that does something cool and useful. The obvious

by pdkl95 8y ago

That's correct. Articles/tutorials or an evangelizing fan often show the end result: the cool command/pipeline that does something cool and useful. The obvious question when someone unfamiliar with unix upon seeing something like the pipeline in this article: comm -1 -3 <(ls -1 dataset-directory | \ grep '\d\d\d\d_A.csv' | \ cut -c 1-4 | \ python3 parse.py | \ uniq \ ) \ <(seq 500) is "Why would I want to write a complicated mess like that?" Just use ${FAVORITE_PROG_LANG:-Perl, Ruby, or whatever}". For many tasks, a short paragraph of code in a "normal" programming language is probably easier to write and is almost certainly a more robust, easier to maintain solution. However, this assumes that you knew what the problem was and that qualities like maintainability are a goal. Bernhardt's (and my) point is that sometimes you don't know what the goal is yet. Sometimes you just need to do a small, one-off task where a half-assed solutions might be appropriate... iff it's the right half of the ass. Unix shell gets that right for a really useful set of tasks. This works because you are free to utilize that powerful features incrementally, as needed. The interactive nature of the shell lets you explore the problem. The "better" version in a "proper" programming language doesn't exist when you don't yet know the exact nature of the problem. A half-assed bit of shell code that slowly evolved into something useful might be the step between "I have some data" and a larger "real" programming project. That said, there is also wisdom in learning to recognize when your needs have outgrown "small, half-assed" solutions. If the project is growing and adding layers of complexity, it's probably time to switch to a more appropriate tool.

SomethingOrNot 8y ago
I generalized interactivity to the Unix that most people seem familiar with. “The interactive nature of the shell” isn’t that impressive in this day and age. Certainly not shells like Bash (Fish is probably better, but then again that’s very cutting edge shell (“for the ’90s”)). Irrespective of the shell this just boils down to executing code, editing text, executing code, repeat. I suspect people started doing that once they got updating displays, if not sooner.
- TeMPOraL 8y ago
  How is that not impressive for vast majority of developers? For the past couple decades, the only other even remotely mainstream place where you could get a comparable experience was a Lisp REPL. And maaaybe Matlab, later on. Recently, projects like R, Jupyer, and (AFAIK) Julia have been introducing people to interactive development, but those are specific to scientific computing. For general programming, this approach is pretty much unknown outside of Lisp and Unix shell worlds.
  - fwip 8y ago
    The alternative is to write a program or script that does part of the job, run it (compiling if necessary), and see what happens. Then modify. This loop is definitely slower than the shell or other REPL though.
    - TeMPOraL 8y ago
      It's much slower, and doesn't lend itself as well for building the program up from small, independently tested and refined pieces. The speed of that feedback loop really matters - the slower it is, the larger chunks you'll be writing before testing. I currently believe the popularity of TDD is primarily a symptom of not having a decent REPL (though REPL doesn't replace unit tests, especially in terms of regression testing). BTW. there's another nice feature of Lisp-style interactive development - you're mutating a living program. You can change the data or define and redefine functions and classes as the program is executing them, without pausing the program. The other end of your REPL essentially becomes a small OS. This matters less when you're building a terminal utility, but it's useful for server and GUI software, and leads to wonders like this: https://www.youtube.com/watch?v=gj5IzggEWKE https://www.youtube.com/watch?v=gj5IzggEWKE It's a different way of approaching programming, and I encourage everyone to try it out.
      - pdkl95 8y ago
        > leads to wonders like this The engineer in me that learned about computers on a 286 with 4MB of RAM and a Hercules graphics card screams in shock and horror at the thought of letting a Cray-2's worth of computing power burn in the background. The hacker in me thinks the engineer in me should shut up and realize that live-editing shader programs is fun[1] and a great way to play with interesting math[2]. [1] http://glslsandbox.com/e#41482.0 http://glslsandbox.com/e#41482.0 [2] http://glslsandbox.com/e#52411.1 http://glslsandbox.com/e#52411.1
Apes 8y ago
To be fair, it's possible to make this block simpler and more readable than what you have there. The problem with a lot of bash scripts I've seen is that they just duck-tape layer after layer of complexity on top of each other, instead of breaking things into smaller, composable pieces. Here's a quick refactor for the block that I would say is simpler and easier to maintain. function xform() { local dir="$1" ls -1 "$dir" | grep '\d\d\d\d_A.csv' | cut -c 1-4 | python3 parse.py | uniq } comm -1 -3 <(xform dataset-directory) <(seq 500)
vram22 8y ago
>is "Why would I want to write a complicated mess like that?" Just use ${FAVORITE_PROG_LANG:-Perl, Ruby, or whatever}". Some discussion on the pros and cons of those two approaches, here: More shell, less egg: http://www.leancrew.com/all-this/2011/12/more-shell-less-egg/ http://www.leancrew.com/all-this/2011/12/more-shell-less-egg... I had written quick solutions to that problem in both Python and Unix shell (bash), here: The Bentley-Knuth problem and solutions: https://jugad2.blogspot.com/2012/07/the-bentley-knuth-problem-and-solutions.html https://jugad2.blogspot.com/2012/07/the-bentley-knuth-proble...
- svat 8y ago
  That's not a “discussion on the pros and cons of those two approaches”; that's a skewed story about just one part of a particular review of an exercise done in a particular historical context. (More on that here: https://news.ycombinator.com/item?id=18699718 https://news.ycombinator.com/item?id=18699718) Not that there isn't some merit to McIllroy's criticism (I know some of the frustration from trying to read Knuth's programs carefully), but at least link to the original context instead of a blog post that tells a partial story: https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-1.pdf https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-... https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-2.pdf https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-... (One of the places where McIlroy admits his criticism was "a little unfair": https://www.princeton.edu/~hos/mike/transcripts/mcilroy.htm https://www.princeton.edu/~hos/mike/transcripts/mcilroy.htm) BTW, there's a wonderful book called “Exercises in Programming Style” (a review here: https://henrikwarne.com/2018/03/13/exercises-in-programming-style/ https://henrikwarne.com/2018/03/13/exercises-in-programming-...) that illustrates many different solutions to that problem (though as it happens it does not include Knuth's WEB program or McIllroy's Unix pipeline).
  - vram22 8y ago
    Thanks. >(More on that here: https://news.ycombinator.com/item?id=18699718 https://news.ycombinator.com/item?id=18699718) >BTW, there's a wonderful book called “Exercises in Programming Style” (a review here: https://henrikwarne.com/2018/03/13/exercises-in-programming-... https://henrikwarne.com/2018/03/13/exercises-in-programming-...) that illustrates many different solutions to that problem (though as it happens it does not include Knuth's WEB program or McIllroy's Unix pipeline). I'm the same person who referred to my post with two solutions (in Python and shell) in that thread, here: https://news.ycombinator.com/item?id=18699656 https://news.ycombinator.com/item?id=18699656 in reply to which, Henrik Warne talked about the book you mention above.
thaumaturgy 8y ago
Just yesterday I needed to extract, sort, and categorize the user agent strings for 6 months' traffic to a handful of sites (attempting to convince a company to abandon TLS 1.0/1.1). The first half of the job was exactly the process you described: start with one log file, craft a grep for it, craft a `grep -o` for the relevant part of each relevant line, add `sort | uniq -c | sort -r`, switch to zgrep for the archived rotated files, and so on. The other half of the ass was done in a different language, using the output from the shell, because I needed to do a thousand or so lookups against a website and parse the results. Composable shell tools is a very under-appreciated toolbox, IMO.
- svsucculents 8y ago
  The one thing that I hate about 'half assed' comments when people refer to the unix shell is that the half ass really refers to users knowledge of how to actually construct efficient pipelines. How many times I've seen 'cat frick.csv | sed 's/foo/bar/g' | sort -n | uniq | cut -d "," -f 3' type nonsense masquerading as deeply effective and efficient extraction and transformation is mind blowing. Your example included.
js2 8y ago
FYI, you don't need the backslash if you end the line with a pipe... it's implied in that case.

by pdkl95 8y ago

SomethingOrNot 8y ago

I generalized interactivity to the Unix that most people seem familiar with. “The interactive nature of the shell” isn’t that impressive in this day and age. Certainly not shells like Bash (Fish is probably better, but then again that’s very cutting edge shell (“for the ’90s”)). Irrespective of the shell this just boils down to executing code, editing text, executing code, repeat. I suspect people started doing that once they got updating displays, if not sooner.

TeMPOraL 8y ago
How is that not impressive for vast majority of developers? For the past couple decades, the only other even remotely mainstream place where you could get a comparable experience was a Lisp REPL. And maaaybe Matlab, later on. Recently, projects like R, Jupyer, and (AFAIK) Julia have been introducing people to interactive development, but those are specific to scientific computing. For general programming, this approach is pretty much unknown outside of Lisp and Unix shell worlds.
- fwip 8y ago
  The alternative is to write a program or script that does part of the job, run it (compiling if necessary), and see what happens. Then modify. This loop is definitely slower than the shell or other REPL though.
  - TeMPOraL 8y ago
    It's much slower, and doesn't lend itself as well for building the program up from small, independently tested and refined pieces. The speed of that feedback loop really matters - the slower it is, the larger chunks you'll be writing before testing. I currently believe the popularity of TDD is primarily a symptom of not having a decent REPL (though REPL doesn't replace unit tests, especially in terms of regression testing). BTW. there's another nice feature of Lisp-style interactive development - you're mutating a living program. You can change the data or define and redefine functions and classes as the program is executing them, without pausing the program. The other end of your REPL essentially becomes a small OS. This matters less when you're building a terminal utility, but it's useful for server and GUI software, and leads to wonders like this: https://www.youtube.com/watch?v=gj5IzggEWKE https://www.youtube.com/watch?v=gj5IzggEWKE It's a different way of approaching programming, and I encourage everyone to try it out.
    - pdkl95 8y ago
      > leads to wonders like this The engineer in me that learned about computers on a 286 with 4MB of RAM and a Hercules graphics card screams in shock and horror at the thought of letting a Cray-2's worth of computing power burn in the background. The hacker in me thinks the engineer in me should shut up and realize that live-editing shader programs is fun[1] and a great way to play with interesting math[2]. [1] http://glslsandbox.com/e#41482.0 http://glslsandbox.com/e#41482.0 [2] http://glslsandbox.com/e#52411.1 http://glslsandbox.com/e#52411.1

Apes 8y ago

To be fair, it's possible to make this block simpler and more readable than what you have there. The problem with a lot of bash scripts I've seen is that they just duck-tape layer after layer of complexity on top of each other, instead of breaking things into smaller, composable pieces. Here's a quick refactor for the block that I would say is simpler and easier to maintain. function xform() { local dir="$1" ls -1 "$dir" | grep '\d\d\d\d_A.csv' | cut -c 1-4 | python3 parse.py | uniq } comm -1 -3 <(xform dataset-directory) <(seq 500)

vram22 8y ago

>is "Why would I want to write a complicated mess like that?" Just use ${FAVORITE_PROG_LANG:-Perl, Ruby, or whatever}". Some discussion on the pros and cons of those two approaches, here: More shell, less egg: http://www.leancrew.com/all-this/2011/12/more-shell-less-egg/ http://www.leancrew.com/all-this/2011/12/more-shell-less-egg... I had written quick solutions to that problem in both Python and Unix shell (bash), here: The Bentley-Knuth problem and solutions: https://jugad2.blogspot.com/2012/07/the-bentley-knuth-problem-and-solutions.html https://jugad2.blogspot.com/2012/07/the-bentley-knuth-proble...

svat 8y ago
That's not a “discussion on the pros and cons of those two approaches”; that's a skewed story about just one part of a particular review of an exercise done in a particular historical context. (More on that here: https://news.ycombinator.com/item?id=18699718 https://news.ycombinator.com/item?id=18699718) Not that there isn't some merit to McIllroy's criticism (I know some of the frustration from trying to read Knuth's programs carefully), but at least link to the original context instead of a blog post that tells a partial story: https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-1.pdf https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-... https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-2.pdf https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-... (One of the places where McIlroy admits his criticism was "a little unfair": https://www.princeton.edu/~hos/mike/transcripts/mcilroy.htm https://www.princeton.edu/~hos/mike/transcripts/mcilroy.htm) BTW, there's a wonderful book called “Exercises in Programming Style” (a review here: https://henrikwarne.com/2018/03/13/exercises-in-programming-style/ https://henrikwarne.com/2018/03/13/exercises-in-programming-...) that illustrates many different solutions to that problem (though as it happens it does not include Knuth's WEB program or McIllroy's Unix pipeline).
- vram22 8y ago
  Thanks. >(More on that here: https://news.ycombinator.com/item?id=18699718 https://news.ycombinator.com/item?id=18699718) >BTW, there's a wonderful book called “Exercises in Programming Style” (a review here: https://henrikwarne.com/2018/03/13/exercises-in-programming-... https://henrikwarne.com/2018/03/13/exercises-in-programming-...) that illustrates many different solutions to that problem (though as it happens it does not include Knuth's WEB program or McIllroy's Unix pipeline). I'm the same person who referred to my post with two solutions (in Python and shell) in that thread, here: https://news.ycombinator.com/item?id=18699656 https://news.ycombinator.com/item?id=18699656 in reply to which, Henrik Warne talked about the book you mention above.

thaumaturgy 8y ago

Just yesterday I needed to extract, sort, and categorize the user agent strings for 6 months' traffic to a handful of sites (attempting to convince a company to abandon TLS 1.0/1.1). The first half of the job was exactly the process you described: start with one log file, craft a grep for it, craft a `grep -o` for the relevant part of each relevant line, add `sort | uniq -c | sort -r`, switch to zgrep for the archived rotated files, and so on. The other half of the ass was done in a different language, using the output from the shell, because I needed to do a thousand or so lookups against a website and parse the results. Composable shell tools is a very under-appreciated toolbox, IMO.

svsucculents 8y ago
The one thing that I hate about 'half assed' comments when people refer to the unix shell is that the half ass really refers to users knowledge of how to actually construct efficient pipelines. How many times I've seen 'cat frick.csv | sed 's/foo/bar/g' | sort -n | uniq | cut -d "," -f 3' type nonsense masquerading as deeply effective and efficient extraction and transformation is mind blowing. Your example included.

js2 8y ago

FYI, you don't need the backslash if you end the line with a pipe... it's implied in that case.