4 ms·

Awk, like shell, is a fraught programming environment, full of silent failure and hidden gotchas. Even for one-liners. I use shell because I have to, not beca

Awk, like shell, is a fraught programming environment, full of silent failure and hidden gotchas. Even for one-liners. I use shell because I have to, not because I like it. I dread maintaining shell scripts which have a bunch of awk and sed in them. The Unix ideal of small single-purpose tools and text processing is separable from these old warhorses.

rb808 7y ago
scripts are the antithesis of modern software development. No Unit testing, No CI, no monitoring, no consistency, often no source control. Appeals to me as a hacker but only my scripts are intelligible - which is a bad sign. :)
- sansnomme 7y ago
  PowerShell begs to differ.
- umvi 7y ago
  I'm not actually convinced unit testing is all that valuable, unless the unit under test has a very clear input -> output transformation (like algorithms, string utilities, etc). If it doesn't (and most units don't), unit tests just encumber you.
  - Twisol 7y ago
    The value of unit tests when creating software is perhaps debatable, but I find their greatest value to be in maintaining software. If you lock in your expectations of a software component, when it's time to make changes (due to shifting requirements or what not), you can add tests capturing your intent and make sure that you aren't breaking existing expectations. Or at least, that you know which expectations your change will break.
  - Eikon 7y ago
    I’m really glad to read that. I never understood the whole religion around unit tests. Integration tests are often far easier to write and far more valuable. Like you said, unit tests are really nice when testing for a known, expected output. Unit tests that are effectively testing mocks and crazy stubs because your method has side effects? Not for me.
    - coliveira 7y ago
      It is easy to explain: unit tests give a documented proof that you care about code robustness. It is used more for social and psychological effect than for its advantages. In fact, outside a few domains, unit tests make it harder to evolve software because the more tests you write, the harder it is to make changes that move the design in a different direction. This is, by the way, my main problem with verbose techniques of programming: the more you have to code, the harder to make needed changes.
    - gbacon 7y ago
      Integration tests can be easier to write, are necessary, but can also be much slower to run. Yes, it’s possible to write bad unit tests; the same is true of integration tests. You need both unit tests and integration tests.
  - hnick 7y ago
    Yes I worked in a print production industry and unit testing was almost impossible. To generate a string of text reliably and correctly is not difficult, but to position it on the page respecting other elements, flow, overlap, current branded font and size, and white space - unit testing is basically useless. And these are the errors we faced most.
- wnoise 7y ago
  Every single ones of those things can be done when using scripts.
- arendtio 7y ago
  Actually, that is not true. I have written a tool (also a script) myself that allows you to write unit tests, manage your scripts in git, load scripts in other scripts, etc. Maybe I will post a Show HN in the coming weeks, but at the moment I would like to round up some edges before posting it. In my experience, the biggest problem is that there are many different runtime environments out there that differ in detail and make it hard to write scripts that run everywhere. But programmers not applying all their skills (e.g. writing tests) to build scripts are also part of the problem.
rhombocombus 7y ago
While I agree that the silent failures and "opaqueness" can be off-putting, once you understand the tool and how to implement it in your workflow it is wonderfully efficient. Aside from being syntactically terse I haven't found any compelling reasons not to use it.
- rectang 7y ago
  I've achieved a high level of expertise in Perl. Even though I can, I won't write complex Perl one liners; instead, I will write tested, documented utilities which someone who has not achieved such a level of mastery has a shot at grokking, maintaining and debugging. Even better if I can write such utilities in a more modern programming language like Go or Rust, if the organization has personnel with expertise in such languages.
  - Chris2048 7y ago
    What if the task is one-off? would you still care to write a proper utility? As an example, you have some data you need to fix as input to some program. you incrementally try to fix it with perl: 1) run program with data, observe errors, infer what needs fixing -> 2) write perl to fix data, modify data -> 3) repeat from (1) until no errors You have no expectation you'll ever need to fix data corrupted in the same way ever again.
  - 0xff00ffee 7y ago
    rectang: "Awk, like shell, is a fraught programming environment, full of silent failure and hidden gotchas. Even for one-liners..." Also rectang: "Even better if I can write such utilities in a more modern programming language like Go or Rust..." Thank you for demonstrating how Linux decays due to lack of experience and NIH syndrome.
- jolmg 7y ago
  > Aside from being syntactically terse I haven't found any compelling reasons not to use it. For me, that's THE reason to use it. It's terseness is what allows it to be efficient enough to be used primarily interactively.
jiggawatts 7y ago
PowerShell is a lot better at this. It was designed for reading by humans, not for saving precious bytes over 300-baud terminal connections. So for example, selecting columns is easy, and can be done by name. Here's a useful little snippet demonstrating converting and processing the CSV output of the legacy "whoami" Windows command. It lists the groups a user is a member of without poking domain controllers using LDAP. It always works, including cross-forest scenarios. WHOAMI /GROUPS /FO CSV ` | ConvertFrom-Csv ` | Where-Object Attributes -NotLike '*used for deny only*' ` | Select-Object -ExpandProperty 'Group Name' It looks like a lot of typing, but everything tab-completes and the end-result is human readable. I find that people that prefer terseness over verbose syntax are selfish. They simply don't care about the future maintainers of their scripts. PowerShell can also natively parse JSON and XML into object hierarchies and then select from them. That's difficult in UNIX. The Invoke-RestMethod command is particularly brilliant, because instead of a stream of bytes it returns an object that can be manipulated directly. $p = Invoke-RestMethod 'https://ipapi.co/8.8.8.8/json/' echo $p.org echo $p.country_name PowerShell Core is available on Linux and is great for ad-hoc tabular data processing! Give it a go...
- LilBytes 7y ago
  PowerShell Core (PS5/open source) is crushing it in this space over Bash\AWK specifically. I'm a pretty advocate for how it works in these use cases now. Generally I still use Bash\Sed\AWK for the one off instances when ever I need something done there and then. But if I'm writing something that's going to be involved in my CI\CD with a high chance of needing to be maintained. I'll nearly always write it in PowerShell.
- shiftless 7y ago
  Sorry, the reason I moved to OpenBSD and Linux From Scratch was to avoid getting sucked into the dark gaping asshole of Lucifer. Microsoft shitware is not allowed on my system. Does the phrase "Embrace, Extend, Extinguish" ring a bell? Between RedHat, Ubuntu, and Poettering (in no particular order), they have already done great damage to the world of Linux. Microsoft, needless to say, is a cancer. > I find that people that prefer terseness over verbose syntax are selfish. They simply don't care about the future maintainers of their scripts. Nor do you, apparently, since the poor souls who follow in your footsteps will invariably be stuck within the literal hell on earth that is the Microsoft ecosystem. Enjoy your regularly scheduled malware updates.
kortex 7y ago
I wish there were a set of tools like coreutils but like..."less surprising" I guess? Like I spend 98% of my focus-time in algol-likes. 1-indexing drives me batty. I always have to try a few times to craft craft cut or tail expressions to match [i:j] slice notation. And anything more complex than that I typically just use python.

rb808 7y ago

scripts are the antithesis of modern software development. No Unit testing, No CI, no monitoring, no consistency, often no source control. Appeals to me as a hacker but only my scripts are intelligible - which is a bad sign. :)

sansnomme 7y ago
PowerShell begs to differ.
umvi 7y ago
I'm not actually convinced unit testing is all that valuable, unless the unit under test has a very clear input -> output transformation (like algorithms, string utilities, etc). If it doesn't (and most units don't), unit tests just encumber you.
- Twisol 7y ago
  The value of unit tests when creating software is perhaps debatable, but I find their greatest value to be in maintaining software. If you lock in your expectations of a software component, when it's time to make changes (due to shifting requirements or what not), you can add tests capturing your intent and make sure that you aren't breaking existing expectations. Or at least, that you know which expectations your change will break.
- Eikon 7y ago
  I’m really glad to read that. I never understood the whole religion around unit tests. Integration tests are often far easier to write and far more valuable. Like you said, unit tests are really nice when testing for a known, expected output. Unit tests that are effectively testing mocks and crazy stubs because your method has side effects? Not for me.
  - coliveira 7y ago
    It is easy to explain: unit tests give a documented proof that you care about code robustness. It is used more for social and psychological effect than for its advantages. In fact, outside a few domains, unit tests make it harder to evolve software because the more tests you write, the harder it is to make changes that move the design in a different direction. This is, by the way, my main problem with verbose techniques of programming: the more you have to code, the harder to make needed changes.
  - gbacon 7y ago
    Integration tests can be easier to write, are necessary, but can also be much slower to run. Yes, it’s possible to write bad unit tests; the same is true of integration tests. You need both unit tests and integration tests.
- hnick 7y ago
  Yes I worked in a print production industry and unit testing was almost impossible. To generate a string of text reliably and correctly is not difficult, but to position it on the page respecting other elements, flow, overlap, current branded font and size, and white space - unit testing is basically useless. And these are the errors we faced most.
wnoise 7y ago
Every single ones of those things can be done when using scripts.
arendtio 7y ago
Actually, that is not true. I have written a tool (also a script) myself that allows you to write unit tests, manage your scripts in git, load scripts in other scripts, etc. Maybe I will post a Show HN in the coming weeks, but at the moment I would like to round up some edges before posting it. In my experience, the biggest problem is that there are many different runtime environments out there that differ in detail and make it hard to write scripts that run everywhere. But programmers not applying all their skills (e.g. writing tests) to build scripts are also part of the problem.

rhombocombus 7y ago

While I agree that the silent failures and "opaqueness" can be off-putting, once you understand the tool and how to implement it in your workflow it is wonderfully efficient. Aside from being syntactically terse I haven't found any compelling reasons not to use it.

rectang 7y ago
I've achieved a high level of expertise in Perl. Even though I can, I won't write complex Perl one liners; instead, I will write tested, documented utilities which someone who has not achieved such a level of mastery has a shot at grokking, maintaining and debugging. Even better if I can write such utilities in a more modern programming language like Go or Rust, if the organization has personnel with expertise in such languages.
- Chris2048 7y ago
  What if the task is one-off? would you still care to write a proper utility? As an example, you have some data you need to fix as input to some program. you incrementally try to fix it with perl: 1) run program with data, observe errors, infer what needs fixing -> 2) write perl to fix data, modify data -> 3) repeat from (1) until no errors You have no expectation you'll ever need to fix data corrupted in the same way ever again.
- 0xff00ffee 7y ago
  rectang: "Awk, like shell, is a fraught programming environment, full of silent failure and hidden gotchas. Even for one-liners..." Also rectang: "Even better if I can write such utilities in a more modern programming language like Go or Rust..." Thank you for demonstrating how Linux decays due to lack of experience and NIH syndrome.
jolmg 7y ago
> Aside from being syntactically terse I haven't found any compelling reasons not to use it. For me, that's THE reason to use it. It's terseness is what allows it to be efficient enough to be used primarily interactively.

jiggawatts 7y ago

PowerShell is a lot better at this. It was designed for reading by humans, not for saving precious bytes over 300-baud terminal connections. So for example, selecting columns is easy, and can be done by name. Here's a useful little snippet demonstrating converting and processing the CSV output of the legacy "whoami" Windows command. It lists the groups a user is a member of without poking domain controllers using LDAP. It always works, including cross-forest scenarios. WHOAMI /GROUPS /FO CSV ` | ConvertFrom-Csv ` | Where-Object Attributes -NotLike '*used for deny only*' ` | Select-Object -ExpandProperty 'Group Name' It looks like a lot of typing, but everything tab-completes and the end-result is human readable. I find that people that prefer terseness over verbose syntax are selfish. They simply don't care about the future maintainers of their scripts. PowerShell can also natively parse JSON and XML into object hierarchies and then select from them. That's difficult in UNIX. The Invoke-RestMethod command is particularly brilliant, because instead of a stream of bytes it returns an object that can be manipulated directly. $p = Invoke-RestMethod 'https://ipapi.co/8.8.8.8/json/' echo $p.org echo $p.country_name PowerShell Core is available on Linux and is great for ad-hoc tabular data processing! Give it a go...

LilBytes 7y ago
PowerShell Core (PS5/open source) is crushing it in this space over Bash\AWK specifically. I'm a pretty advocate for how it works in these use cases now. Generally I still use Bash\Sed\AWK for the one off instances when ever I need something done there and then. But if I'm writing something that's going to be involved in my CI\CD with a high chance of needing to be maintained. I'll nearly always write it in PowerShell.
shiftless 7y ago
Sorry, the reason I moved to OpenBSD and Linux From Scratch was to avoid getting sucked into the dark gaping asshole of Lucifer. Microsoft shitware is not allowed on my system. Does the phrase "Embrace, Extend, Extinguish" ring a bell? Between RedHat, Ubuntu, and Poettering (in no particular order), they have already done great damage to the world of Linux. Microsoft, needless to say, is a cancer. > I find that people that prefer terseness over verbose syntax are selfish. They simply don't care about the future maintainers of their scripts. Nor do you, apparently, since the poor souls who follow in your footsteps will invariably be stuck within the literal hell on earth that is the Microsoft ecosystem. Enjoy your regularly scheduled malware updates.

kortex 7y ago

I wish there were a set of tools like coreutils but like..."less surprising" I guess? Like I spend 98% of my focus-time in algol-likes. 1-indexing drives me batty. I always have to try a few times to craft craft cut or tail expressions to match [i:j] slice notation. And anything more complex than that I typically just use python.