3 ms·

Hello! I had started tutorials on command line text processing (https://github.com/learnbyexample/Command-line-text-processing https://github.com/learnbyexampl

by asicsp 6y ago

Hello! I had started tutorials on command line text processing (https://github.com/learnbyexample/Command-line-text-processing https://github.com/learnbyexample/Command-line-text-processi...) more than three years back. I learnt a lot writing them and continue to learn more with experience. I finished first version of cookbook on Perl one-liners recently. With that, five of the major chapters from that repo are now accessible as better formatted ebooks, updated for newer software versions, exercises, solutions, etc. You can download pdf/epub versions of the ebooks using the links below (free until this Sunday) * Perl one-liners cookbook: https://gumroad.com/l/perl-oneliners https://gumroad.com/l/perl-oneliners or https://leanpub.com/perl-oneliners https://leanpub.com/perl-oneliners * Five book bundle of grep/sed/awk/perl/ruby one-liners: https://gumroad.com/l/oneliners https://gumroad.com/l/oneliners or https://leanpub.com/b/oneliners https://leanpub.com/b/oneliners --- I'd highly appreciate your feedback and hope that you find these resources useful. Happy learning and stay safe :)

rStar 6y ago
nice links! thank you!
kqr 6y ago
How do you picture someone using this? The reason I ask is that I've never figured out a good way to make use of this type of material. Reading idly means I retain almost nothing. I try to avoid spaced repetition on things I don't actually use (that has turned out to be a good filter to avoid over-adding flashcards). Whenever I have a problem that can be solved by a Perl one-liner, there are two obstacles to using something like your book: 1. Figuring out that my problem is solvable with a Perl one-liner in the first place, and 2. Finding the right patterns in the book to piece together to achieve my goal. Have you considered any sort of tiered index to allow one to classify a problem at hand and look up the relevant structures int he book?
- rattray 6y ago
  I agree – having read the intro and skimmed some other sections, they seem like great foundational introductions and I'm absolutely excited to use perl instead of GNU-like utilities... but I'm very nervous that I wouldn't be able to quickly find how to do it with perl later. An additional "problems to solve with a perl one-liner" index of some kind might be quite helpful for that...
- weare138 6y ago
  >Whenever I have a problem that can be solved by a Perl one-liner, there are two obstacles to using something like your book: 1. Figuring out that my problem is solvable with a Perl one-liner in the first place, and 2. Finding the right patterns in the book to piece together to achieve my goal. The same argument can be made about any programming language. What makes Perl one-liners different?
  - kqr 6y ago
    My comment was not about a programming language, but about a book of example one-liners. I have the same problem with any book of one-liners, no matter the subject language.
- asicsp 6y ago
  >Have you considered any sort of tiered index to allow one to classify a problem at hand and look up the relevant structures int he book? I have wondered a few times if I can build some sort of guide/flowchart that allows one to pick tool(s) for a particular text processing job (for ex: when to choose head/tail/grep/sed/awk/pr/paste/etc). And then within the tool(s) chosen, how to break down a problem and select particular features of that tool. It has remained a fantasy as I haven't tried to actually start doing something about this. I dunno if this is feasible and/or whether I'll be able to do it. >The reason I ask is that I've never figured out a good way to make use of this type of material. There are several prerequisites mentioned in the Preface before you can use this book. You'll have to be familiar with cli environment as well as Perl basics. After that, I feel the best way would be to start using these one-liners in your everyday tasks. If your workflow doesn't suit this type of usage, you'll have to be either motivated enough to explore or take a risk to change/modify your workflow. I provide several examples for each topic and have exercises at the end of chapters. Broadly speaking, there are three main tasks (tools mentioned in parentheses is the fundamental task of that tool) * Filter an input record based on regexp, string or numeric comparisons (grep) * Search and replace (sed) * Field based filtering/substitution (awk) awk and perl can do all of these tasks. However, a tool specialized for a task is usually faster and has custom options for ease of use for those tasks. So, my best advice would be to practice and find reasons to incorporate cli based solutions in your everyday workflow.
  - kqr 6y ago
    I consider myself an intermediate-or-advanced user of both sed and awk. Those are small enough languages that I was able to study them for a few hours and then know roughly how to solve any problem with them. Perl one-liners are different in that they rely on a large set of flags and language shortcuts which are harder to remember when your main language isn't Perl. So it's not about getting used to the POSIX utilities and text processing, it's more about it being infeasible to "just memorise" all these Perl shortcuts. The traditional solution when "just memory" won't cut it is some sort of index to speed up the search. The guide/flowchart thing is just what I'm talking about. Botanists use a tiered index to identify plants: they subdivide the flora first by large differences in characteristics and then down to differences you need a magnifier to see. Each observation cuts the potential space by a significant fraction. Something similar would be useful for this. E.g. "is your input structured or free text?" "Are records separated by linebreak or something else?" "Are there fields in the records?" "Are you looking to perform an aggregation?" And so on. I don't know. It would be neat for many reasons!

by asicsp 6y ago

rStar 6y ago

nice links! thank you!

kqr 6y ago

How do you picture someone using this? The reason I ask is that I've never figured out a good way to make use of this type of material. Reading idly means I retain almost nothing. I try to avoid spaced repetition on things I don't actually use (that has turned out to be a good filter to avoid over-adding flashcards). Whenever I have a problem that can be solved by a Perl one-liner, there are two obstacles to using something like your book: 1. Figuring out that my problem is solvable with a Perl one-liner in the first place, and 2. Finding the right patterns in the book to piece together to achieve my goal. Have you considered any sort of tiered index to allow one to classify a problem at hand and look up the relevant structures int he book?

rattray 6y ago
I agree – having read the intro and skimmed some other sections, they seem like great foundational introductions and I'm absolutely excited to use perl instead of GNU-like utilities... but I'm very nervous that I wouldn't be able to quickly find how to do it with perl later. An additional "problems to solve with a perl one-liner" index of some kind might be quite helpful for that...
weare138 6y ago
>Whenever I have a problem that can be solved by a Perl one-liner, there are two obstacles to using something like your book: 1. Figuring out that my problem is solvable with a Perl one-liner in the first place, and 2. Finding the right patterns in the book to piece together to achieve my goal. The same argument can be made about any programming language. What makes Perl one-liners different?
- kqr 6y ago
  My comment was not about a programming language, but about a book of example one-liners. I have the same problem with any book of one-liners, no matter the subject language.
asicsp 6y ago
>Have you considered any sort of tiered index to allow one to classify a problem at hand and look up the relevant structures int he book? I have wondered a few times if I can build some sort of guide/flowchart that allows one to pick tool(s) for a particular text processing job (for ex: when to choose head/tail/grep/sed/awk/pr/paste/etc). And then within the tool(s) chosen, how to break down a problem and select particular features of that tool. It has remained a fantasy as I haven't tried to actually start doing something about this. I dunno if this is feasible and/or whether I'll be able to do it. >The reason I ask is that I've never figured out a good way to make use of this type of material. There are several prerequisites mentioned in the Preface before you can use this book. You'll have to be familiar with cli environment as well as Perl basics. After that, I feel the best way would be to start using these one-liners in your everyday tasks. If your workflow doesn't suit this type of usage, you'll have to be either motivated enough to explore or take a risk to change/modify your workflow. I provide several examples for each topic and have exercises at the end of chapters. Broadly speaking, there are three main tasks (tools mentioned in parentheses is the fundamental task of that tool) * Filter an input record based on regexp, string or numeric comparisons (grep) * Search and replace (sed) * Field based filtering/substitution (awk) awk and perl can do all of these tasks. However, a tool specialized for a task is usually faster and has custom options for ease of use for those tasks. So, my best advice would be to practice and find reasons to incorporate cli based solutions in your everyday workflow.
- kqr 6y ago
  I consider myself an intermediate-or-advanced user of both sed and awk. Those are small enough languages that I was able to study them for a few hours and then know roughly how to solve any problem with them. Perl one-liners are different in that they rely on a large set of flags and language shortcuts which are harder to remember when your main language isn't Perl. So it's not about getting used to the POSIX utilities and text processing, it's more about it being infeasible to "just memorise" all these Perl shortcuts. The traditional solution when "just memory" won't cut it is some sort of index to speed up the search. The guide/flowchart thing is just what I'm talking about. Botanists use a tiered index to identify plants: they subdivide the flora first by large differences in characteristics and then down to differences you need a magnifier to see. Each observation cuts the potential space by a significant fraction. Something similar would be useful for this. E.g. "is your input structured or free text?" "Are records separated by linebreak or something else?" "Are there fields in the records?" "Are you looking to perform an aggregation?" And so on. I don't know. It would be neat for many reasons!