5 ms·

Every programmer worth their salt must learn Perl, Bash and using these command line utils. There are chances you are writing lots of code which just doesn't ha

by kamaal 2y ago

Every programmer worth their salt must learn Perl, Bash and using these command line utils. There are chances you are writing lots of code which just doesn't have to get written. vi, is something that you must absolutely learn. Learning vi(especially with macros) will give you new perspectives about thinking and working with plain text in ways you might not have imagined before. Most of it might be one time work, but it generally saves you lots of spreadsheet and scripting work. Many times you even realise you were doing lots of pointless work over the years. Perl is the next progression in this path, and this will give you full code paradigm to do with anything Unix and text. Like so as long your application runs on Unixy operating systems, Perl can do it. You can often ship just one file to the server, be guaranteed it will run with backwards compatibility, it will run fast. You can also develop fast and change at will. Learning bash gives you means to dump a series a commands into a file and repeating them. But many times what saves you time and effort is learning to use for loop on the command line. This helps you do a lot of bulk work in repetition.

synergy20 2y ago
agree all but perl, I found perl hard to write and hard to read
forrestthewoods 2y ago
> Every programmer worth their salt must learn Perl, Bash and using these command line utils. Naw. As a game and VR programmer I only barely need to acknowledge that POSIX exists. These tools are not even Top 100 in usefulness.
- pxc 2y ago
  When I was a student, I once interned for a company of Windows developers, a Microsoft shop whose work was all in .NET. I remember that there were some tasks I was able to do, having to do with processing many, many little text files, that everyone was surprised I was able to do, or surprised I was able to do quickly. Every time, it was something that I put together in a few minutes with the usual Unix tools thanks to Cygwin. It's probably true that game development is really different from, say, web development, when it comes to the relevance of Unix. But I think it's also true that often, when you don't know common Unix utils, you don't know what you're missing. I don't think any programmer could master them and regret it.
  - dale_glass 2y ago
    These days, Windows caught up very well with PowerShell. Actually more UNIX-y than UNIX in some respects.
    - pxc 2y ago
      I love PowerShell, but I've yet to meet someone who 'lives in' PowerShell the way so many 'live in' Bash and the various GNU CLI utilities, which I think inhibits familiarity of the kind most convenient for whipping up quick scripts like that. That said, PowerShell is much more a 'real programming language' than the shell languages that came before it. PowerShell scripts are a lot more maintainable that scripts written for those other shells. And there are some really powerful things built on top of shell (like DSC, for example) that don't to my knowledge have analogues written directly in other shell languages. Plus PowerShell has an actual package management story, code signing capabilities, and other modern niceties. But for me and I suspect most others, it falls down as an interactive environment for rwasons of performance and verbosity, and Windows environments themselves are unfortunately still harder to automate or inspect programmatically because most applications don't provide or document PowerShell-friendly interfaces. So I don't think it's surprising that despite all of PowerShell's incredible advancemwnts, you rarely meet Windows users who have invested in it as much as the average longtime Unix user invests in their respective CLI utilities. But for, say, game developers who say 'Unix is not my environment; I'm not sure I wanna bother with all of that CLI and scripting stuff', they do nowadays have an alternative in that arena which is outstanding in a lot of positive ways. I don't anyone would regret deeply learning PowerShell, either. Hell, my team at work is all Unix guys. I probably have the most experience PowerShell experience of any of us, which is way less than my GNU command line environment experience. But if someone with deep PowerShell experience joined our team, I'd absolutely encourage them to use PowerShell on Linux wherever we write scripts, and be happy to learn whatever I needed to to keep up. PowerShell is a nice enough language that I think it's very reasonable to bring it with you to other operating systems as well.
  - forrestthewoods 2y ago
    I don't regret knowing Unix tools. But I very frequently regret that other people chose to write non-trivial operations in a bash script instead of a real programming language! If Unix utils didn't exist and everyone just used Python instead I'd honestly probably prefer that.
RodgerTheGreat 2y ago
Eh; Perl just feels to me like a cancerously overgrown AWK. For simple tasks, AWK is tremendously more elegant. For anything too complex for AWK- which is a rather high ceiling- there are a whole host of scripting languages I'd select before Perl. AWK is available in some form on many more posix-ish environments than Perl and the most modern implementations are lightning-fast.
- kamaal 2y ago
  To make awk work for non trivial manipulations you have use cat, awk, | and > in several sequences. This is not by any means elegant. What happens when there is a line that needs some error checking and handled a little differently? Or just a bad line? Once you realise that error tracing, and fixing is just not possible for non trivial stuff you are better off now skipping the entire ceremony and simply use Perl. As matter of fact, Larry Wall invented Perl while trying to the very same things you mention with C and awk. Then you realise you are dealing with so many adhoc things, without dependency checks, error handling and then you actually need a fully blown programming language to do this sort of work.
  - RodgerTheGreat 2y ago
    I've written AWK programs that perform decidedly non-trivial manipulations on data, and none of the problems you describe are real. It is absolutely not the case that one must shell out to a pipeline of other commands to get work done; you can write structured programs in AWK just as you would in any other scripting language, and operate in multiple passes over data. It's a very amusing irony if a Perl enthusiast genuinely believes that AWK can only be used in the form of cryptic one-liners, especially in the context of POSIX tools one "must" learn.
petepete 2y ago
Didn't realise vi had macros, I thought they were a more recent addition.
- kamaal 2y ago
  vi macros are those magical things that one must just know. Once you know how to use them, its really one those things with regards to text manipulation that helps you convert manual O(n^k) tasks to O(1) task. And a lot of things in our industry can be reduced to text manipulation. For the next steps: Perl.
  - petepete 2y ago
    vi is not vim. I don't think vi has macros.
    - kamaal 2y ago
      vi does have it. https://math.berkeley.edu/~gbergman/misc/hacks/vi.html https://math.berkeley.edu/~gbergman/misc/hacks/vi.html
    - fuzztester 2y ago
      vi does have macros. it had it from at least 1990 or so (when i first used vi), and possibly from much earlier.
yjftsjthsd-h 2y ago
> Every programmer worth their salt must learn Perl, Bash and using these command line utils. There are chances you are writing lots of code which just doesn't have to get written. I have written a rather lot of sh (I target POSIX sh, not bash) and I'm decently familiar with common cli tools (coreutils, moreutils, etc.). IME awk+sed cover most of the "fancy" text manipulation just fine; what am I missing out on if I skip perl?
- kamaal 2y ago
  Perl was made to ensure you don't have to learn the dozens of things you'd have to learn and slap together with all sorts of adhoc hacks. Everything comes in one package. The full language can't be captured in this comment, of course. But compared to Bash, you get a C like syntax(Bash comes from Algol, you have to learn a new syntax), you get error handling, full master set regexes, heredocs, qw, associative arrays, arrays, built-in string manipulation, unix interfaces, DB drivers, first class integration with command line tooling etc etc. Think of it like a unix native language. Like the immediate extension to the operating system itself. Apart from that Perl is very stable, fast, and very deeply committed to backwards compatibility. You can be sure the scripts you wrote years back will run on new Perls. awk+sed are simple match-do engines, you can't do counting(atleast not easily), you can't do other myriad stuff like working with >2-3 files at one time(iterating simultaneously, in sub-loops etc). If you are dealing with large files, in Perl its fairly simple to build a index by doing one go on the whole file, then use the key to map to the value where you want to go in the file using seek/tell(with pack/unpack), This makes frequent look ups very computationally cheap. In awk/sed you can't do error handling, you can't do reading and writing to multiple files at the same time et al easily atleast. In Perl, you get dbm drivers and functions install so its relatively straight forward to build look up applications of all kinds. Perl also comes with OO, so if your code gets lengthier you can can fit it into neat design patterns. Last but not the least. Perl has CPAN. Its just one of those things. Only real competition to CPAN till date is NPM. Not even Java/Python have a library ecosystem as large as Perl.
- mbivert 2y ago
  IME (used Perl daily at work for years) not much. If you need something more robust or sophisticated than sh(1)& cie, it's best to head up to a statically typed language instead -- I tend to favor go, in part because of the cross-compilable fat binary thing. The lack of static typing is especially annoying when things are getting beefy enough to consider moving away from sh(1). For prototyping, if you know Perl already, why not; but even there, I personally reach out to sh(1) & friends instead.
nox101 2y ago
Perl is really really bad. Bash is also really really bad. Perl is unreadable and bash is massively insecure unless you chant the magic incantations which even the pros forget. At most you should learn Perl to learn now not to design a language and then you should learn bash to learn why such a large percentage of systems are vulnerable to various injections. Even Apple had a bash error where they had an OS upgrade that deleted your hard drive if you had a space in the volume name which I'm making the assumption was a bash script because spaces in filename don't generally matter in any sane language.
- BoingBoomTschak 2y ago
  That's true, but shellcheck exists, at least.
- kamaal 2y ago
  Hating Perl is some what like hating vim, or emacs, or lisp macros. You can live without it. But you will do a lot of pointless work over long periods of time. There are a lots of people who are ok with that. Me personally, I attempt to live a life where I can do O(1) or O(as little steps it takes) in anything I touch. I stubbornly refuse to do work that can be done using a computer.
  - nox101 2y ago
    I wrote an entire blogging system in Perl. knowing perl doesn't make me want to use it. Rather it makes me want to run away screaming. it doesn't compare to vim, emacs, or lisp. Perl is objectively bad
    - oguz-ismail 2y ago
      >Perl is objectively bad Nah, I think it's fine.
      - tempfile 2y ago
        perhaps it's objectively bad, but subjectively good?
froh 2y ago
perl I'd agreed with up to somewhere 2010. Python it is, these days, for maintainability. one shot quick regex things? sed. anything else python. and I don't say "switch", btw, if you're proficient I there is no value and no need ditching it. however if you're new to the CLI game, then I'd _not_ suggest perl but python instead.
- kamaal 2y ago
  Python unfortunately still doesn't provide a lot of things like heredocs, qw, multiline lambdas, many other unixy things. Its regex capabilities are also not Perl standard. Scripting is a lot about text, but its not only about text. You need to neatly integrate with many other OS things. More importantly the language needs to be designed such that such tasks can be done with ease. Python is more like good a small application programming language. Not that great if you do heavy OS, text or command line interaction work. It feels like a less flexible Java in that regard.
  - fluidcruft 2y ago
    My experience (switching from heavy use of shell) is Python is really good at text processing and building data structure very quickly. Python is a bit wordy for small things but it become extremely concice and reusable once things get a tiny bit larger. Generally I find doing things in Python is straightforward whereas using shell is more of a low level puzzle game of conversions and reformats and fragility. The puzzles in Python are higher level using better data types and containers. But also it's much easier to wrap things up into modules and reuse code in Python. And yeah Python is slow... but not compared to bash (unless you're using things like parallel, but Python isn't necessarily horrible for that either) That's been my experience anyway.
mofosyne 2y ago
One issue might be discoverability of these programs
pjmlp 2y ago
The only reason I care about vi, is that back in the day it would be only sane thing installed by default in many UNIX boxes, unless I would like to have some fun with ed. And until IDEs became common in UNIX land, I would rather be on Emacs camp.

by kamaal 2y ago

synergy20 2y ago

agree all but perl, I found perl hard to write and hard to read

forrestthewoods 2y ago

> Every programmer worth their salt must learn Perl, Bash and using these command line utils. Naw. As a game and VR programmer I only barely need to acknowledge that POSIX exists. These tools are not even Top 100 in usefulness.

pxc 2y ago
When I was a student, I once interned for a company of Windows developers, a Microsoft shop whose work was all in .NET. I remember that there were some tasks I was able to do, having to do with processing many, many little text files, that everyone was surprised I was able to do, or surprised I was able to do quickly. Every time, it was something that I put together in a few minutes with the usual Unix tools thanks to Cygwin. It's probably true that game development is really different from, say, web development, when it comes to the relevance of Unix. But I think it's also true that often, when you don't know common Unix utils, you don't know what you're missing. I don't think any programmer could master them and regret it.
- dale_glass 2y ago
  These days, Windows caught up very well with PowerShell. Actually more UNIX-y than UNIX in some respects.
  - pxc 2y ago
    I love PowerShell, but I've yet to meet someone who 'lives in' PowerShell the way so many 'live in' Bash and the various GNU CLI utilities, which I think inhibits familiarity of the kind most convenient for whipping up quick scripts like that. That said, PowerShell is much more a 'real programming language' than the shell languages that came before it. PowerShell scripts are a lot more maintainable that scripts written for those other shells. And there are some really powerful things built on top of shell (like DSC, for example) that don't to my knowledge have analogues written directly in other shell languages. Plus PowerShell has an actual package management story, code signing capabilities, and other modern niceties. But for me and I suspect most others, it falls down as an interactive environment for rwasons of performance and verbosity, and Windows environments themselves are unfortunately still harder to automate or inspect programmatically because most applications don't provide or document PowerShell-friendly interfaces. So I don't think it's surprising that despite all of PowerShell's incredible advancemwnts, you rarely meet Windows users who have invested in it as much as the average longtime Unix user invests in their respective CLI utilities. But for, say, game developers who say 'Unix is not my environment; I'm not sure I wanna bother with all of that CLI and scripting stuff', they do nowadays have an alternative in that arena which is outstanding in a lot of positive ways. I don't anyone would regret deeply learning PowerShell, either. Hell, my team at work is all Unix guys. I probably have the most experience PowerShell experience of any of us, which is way less than my GNU command line environment experience. But if someone with deep PowerShell experience joined our team, I'd absolutely encourage them to use PowerShell on Linux wherever we write scripts, and be happy to learn whatever I needed to to keep up. PowerShell is a nice enough language that I think it's very reasonable to bring it with you to other operating systems as well.
- forrestthewoods 2y ago
  I don't regret knowing Unix tools. But I very frequently regret that other people chose to write non-trivial operations in a bash script instead of a real programming language! If Unix utils didn't exist and everyone just used Python instead I'd honestly probably prefer that.

RodgerTheGreat 2y ago

Eh; Perl just feels to me like a cancerously overgrown AWK. For simple tasks, AWK is tremendously more elegant. For anything too complex for AWK- which is a rather high ceiling- there are a whole host of scripting languages I'd select before Perl. AWK is available in some form on many more posix-ish environments than Perl and the most modern implementations are lightning-fast.

kamaal 2y ago
To make awk work for non trivial manipulations you have use cat, awk, | and > in several sequences. This is not by any means elegant. What happens when there is a line that needs some error checking and handled a little differently? Or just a bad line? Once you realise that error tracing, and fixing is just not possible for non trivial stuff you are better off now skipping the entire ceremony and simply use Perl. As matter of fact, Larry Wall invented Perl while trying to the very same things you mention with C and awk. Then you realise you are dealing with so many adhoc things, without dependency checks, error handling and then you actually need a fully blown programming language to do this sort of work.
- RodgerTheGreat 2y ago
  I've written AWK programs that perform decidedly non-trivial manipulations on data, and none of the problems you describe are real. It is absolutely not the case that one must shell out to a pipeline of other commands to get work done; you can write structured programs in AWK just as you would in any other scripting language, and operate in multiple passes over data. It's a very amusing irony if a Perl enthusiast genuinely believes that AWK can only be used in the form of cryptic one-liners, especially in the context of POSIX tools one "must" learn.

petepete 2y ago

Didn't realise vi had macros, I thought they were a more recent addition.

kamaal 2y ago
vi macros are those magical things that one must just know. Once you know how to use them, its really one those things with regards to text manipulation that helps you convert manual O(n^k) tasks to O(1) task. And a lot of things in our industry can be reduced to text manipulation. For the next steps: Perl.
- petepete 2y ago
  vi is not vim. I don't think vi has macros.
  - kamaal 2y ago
    vi does have it. https://math.berkeley.edu/~gbergman/misc/hacks/vi.html https://math.berkeley.edu/~gbergman/misc/hacks/vi.html
  - fuzztester 2y ago
    vi does have macros. it had it from at least 1990 or so (when i first used vi), and possibly from much earlier.

yjftsjthsd-h 2y ago

> Every programmer worth their salt must learn Perl, Bash and using these command line utils. There are chances you are writing lots of code which just doesn't have to get written. I have written a rather lot of sh (I target POSIX sh, not bash) and I'm decently familiar with common cli tools (coreutils, moreutils, etc.). IME awk+sed cover most of the "fancy" text manipulation just fine; what am I missing out on if I skip perl?

kamaal 2y ago
Perl was made to ensure you don't have to learn the dozens of things you'd have to learn and slap together with all sorts of adhoc hacks. Everything comes in one package. The full language can't be captured in this comment, of course. But compared to Bash, you get a C like syntax(Bash comes from Algol, you have to learn a new syntax), you get error handling, full master set regexes, heredocs, qw, associative arrays, arrays, built-in string manipulation, unix interfaces, DB drivers, first class integration with command line tooling etc etc. Think of it like a unix native language. Like the immediate extension to the operating system itself. Apart from that Perl is very stable, fast, and very deeply committed to backwards compatibility. You can be sure the scripts you wrote years back will run on new Perls. awk+sed are simple match-do engines, you can't do counting(atleast not easily), you can't do other myriad stuff like working with >2-3 files at one time(iterating simultaneously, in sub-loops etc). If you are dealing with large files, in Perl its fairly simple to build a index by doing one go on the whole file, then use the key to map to the value where you want to go in the file using seek/tell(with pack/unpack), This makes frequent look ups very computationally cheap. In awk/sed you can't do error handling, you can't do reading and writing to multiple files at the same time et al easily atleast. In Perl, you get dbm drivers and functions install so its relatively straight forward to build look up applications of all kinds. Perl also comes with OO, so if your code gets lengthier you can can fit it into neat design patterns. Last but not the least. Perl has CPAN. Its just one of those things. Only real competition to CPAN till date is NPM. Not even Java/Python have a library ecosystem as large as Perl.
mbivert 2y ago
IME (used Perl daily at work for years) not much. If you need something more robust or sophisticated than sh(1)& cie, it's best to head up to a statically typed language instead -- I tend to favor go, in part because of the cross-compilable fat binary thing. The lack of static typing is especially annoying when things are getting beefy enough to consider moving away from sh(1). For prototyping, if you know Perl already, why not; but even there, I personally reach out to sh(1) & friends instead.

nox101 2y ago

Perl is really really bad. Bash is also really really bad. Perl is unreadable and bash is massively insecure unless you chant the magic incantations which even the pros forget. At most you should learn Perl to learn now not to design a language and then you should learn bash to learn why such a large percentage of systems are vulnerable to various injections. Even Apple had a bash error where they had an OS upgrade that deleted your hard drive if you had a space in the volume name which I'm making the assumption was a bash script because spaces in filename don't generally matter in any sane language.

BoingBoomTschak 2y ago
That's true, but shellcheck exists, at least.
kamaal 2y ago
Hating Perl is some what like hating vim, or emacs, or lisp macros. You can live without it. But you will do a lot of pointless work over long periods of time. There are a lots of people who are ok with that. Me personally, I attempt to live a life where I can do O(1) or O(as little steps it takes) in anything I touch. I stubbornly refuse to do work that can be done using a computer.
- nox101 2y ago
  I wrote an entire blogging system in Perl. knowing perl doesn't make me want to use it. Rather it makes me want to run away screaming. it doesn't compare to vim, emacs, or lisp. Perl is objectively bad
  - oguz-ismail 2y ago
    >Perl is objectively bad Nah, I think it's fine.
    - tempfile 2y ago
      perhaps it's objectively bad, but subjectively good?

froh 2y ago

perl I'd agreed with up to somewhere 2010. Python it is, these days, for maintainability. one shot quick regex things? sed. anything else python. and I don't say "switch", btw, if you're proficient I there is no value and no need ditching it. however if you're new to the CLI game, then I'd _not_ suggest perl but python instead.

kamaal 2y ago
Python unfortunately still doesn't provide a lot of things like heredocs, qw, multiline lambdas, many other unixy things. Its regex capabilities are also not Perl standard. Scripting is a lot about text, but its not only about text. You need to neatly integrate with many other OS things. More importantly the language needs to be designed such that such tasks can be done with ease. Python is more like good a small application programming language. Not that great if you do heavy OS, text or command line interaction work. It feels like a less flexible Java in that regard.
- fluidcruft 2y ago
  My experience (switching from heavy use of shell) is Python is really good at text processing and building data structure very quickly. Python is a bit wordy for small things but it become extremely concice and reusable once things get a tiny bit larger. Generally I find doing things in Python is straightforward whereas using shell is more of a low level puzzle game of conversions and reformats and fragility. The puzzles in Python are higher level using better data types and containers. But also it's much easier to wrap things up into modules and reuse code in Python. And yeah Python is slow... but not compared to bash (unless you're using things like parallel, but Python isn't necessarily horrible for that either) That's been my experience anyway.

mofosyne 2y ago

One issue might be discoverability of these programs

pjmlp 2y ago

The only reason I care about vi, is that back in the day it would be only sane thing installed by default in many UNIX boxes, unless I would like to have some fun with ed. And until IDEs became common in UNIX land, I would rather be on Emacs camp.