13 ms·
> it always blows my mind that so few people really know how to use it at a decent level Not nearly as surprising as it is to me that now that most developers
by ajross 8y ago
> it always blows my mind that so few people really know how to use it at a decent level
Not nearly as surprising as it is to me that now that most developers have forgotten perl, they're turning to awk as an inspiring example of a bygone era.
Seriously: perl basically replaced awk in the mid-90's. It absorbed all the great lessons and added two or three dozen innovations. But it had scary syntax, so everyone used python (which did not replace awk very effectively) and forgot perl. So now we're back at awk. And the scary syntax is all in the Rust world.
- kqr 8y agoWhat I like about AWK is that it is described by a few pages of the POSIX standard. If Perl was that ubiquitous and simple, I would prefer it over AWK.
- ajross 8y agoTo be clear: I like that about it too. But in the real world, we want tooling that does more stuff. In the mid-90's, "everyone" knew perl, because it was the best choice for problems in this space. And in a world where everyone knows perl, there is no place for awk. But now we live in a world where no one knows perl, and languages like python and javascript are clumsy and weird in this space. And that makes awk look clever and elegant. All I'm saying is that in perl-world (which was a real place!), awk wasn't clever and elegant, it was stale and primitive. And that to me is more surprising than awk's cleverness.
- fiddlerwoaroof 8y agoI think people underestimate the importance of medium-powered tools. Perl is great, but awks limitations make it easier to write in a way that the next person can maintain.
- fiddlerwoaroof 8y agoThat being said, the couple times I’ve paired with a perl wizard were revelatory.
- kqr 8y agoI totally see your point. And in Perl-world, I would probably use Perl too – I mean, if both tools are equally ubiquitous, why not use the most powerful one? I entered the field at the tail end of the Perl era, so I've only toyed with it a long time ago.
- kamaal 8y ago>>And that makes awk look clever and elegant. Eventually the cycle will repeat again. The moment you will have non trivial text work to do, awk will have to give way to Perl. The rise of Python was really use case for dealing with standard interfaces like DBs/XMLs/JSONs becoming common. Python hasn't actually replaced Perl in any meaningful way.
- oblio 8y agoMost modern distributions write their tools in Python instead of Perl. Proof: the really long transition process Fedora (and therefore RHEL), Debian and Ubuntu went through to migrate from Python 2 to 3. They could have done it faster if not for their system tools written in Python 2. In the web space, Python is not huge, but it definitely supplanted the niche Perl used to have. No one I know writes web stuff in Perl anymore.
- mozumder 8y agoI hated perl in the mid-90's, and stuck with grep/awk/sed intead, because it was a little more documented-structured. Meanwhile, if you randomly mashed on your keyboard, it would output a perl script. I jumped to Python as soon as I found out about it in the late 90's, because it was exactly what I was looking for in self-documenting structuring. It was great for creating parsers with state-machines. I was also the user of my scripts, and I didn't want to have to relearn what I coded a year or so earlier. Python let me pick that up, and to a lesser extent, awk. State machine programming was really self-documenting in Python.
- singingfish 8y agoI’m sick of the “perl is line noise” narrative. https://pastebin.com/C9hPS8xR https://pastebin.com/C9hPS8xR
- ams6110 8y ago> in a world where everyone knows perl, there is no place for awk. And yet in 2018, I never think about Perl anymore, but use sed, awk, and grep daily.
- mbubb 8y agoThis is a great point
- tannhaeuser 8y agoThat, and awk programs are almost perfectly portable across awk implementations: nawk, gawk, mawk (can't vouch for busybox-awk as I haven't tested it, though I've heard it's good as well). This is of practical importance for portable scripts, since Debian uses mawk [1], RH ships gawk, and Mac OS uses nawk. [1] mawk has recently received new commits by its original author after a hiatus of 20 years or so; see https://github.com/mikebrennan000 https://github.com/mikebrennan000
- dannas 8y agoHere's a picture of the AWK book next to the Camel book https://twitter.com/heinrichhartman/status/771706433452769280 https://twitter.com/heinrichhartman/status/77170643345276928... "#awk doc shows you how to implement a rel-DB and a compiler. #perl doc talks 20p+ about nested data structures."
- FractalLP 8y agoThat is one thing I like better in Python than in Perl. In Perl, I was having difficulty with nested data structures and then realized this was something I did in Python daily without even knowing it was a thing on a conscious level. Who hasn't made some weird dictionary with its values being lists or tuples or something like that?
- ajross 8y agoThis is exactly backwards to my eyes. Perl's autovivification behavior (assigning through a empty reference to something autocreates the object of the right type) makes nested data much, much cleaner. Who hasn't made some weird dictionary with its values being lists and forgotten to check for and write code to create it when it doesn't exist, and had to fix that later after a runtime crash? In perl that's literally covered by the one-line assignment you used to set the field you parsed. This is why it's sad that everyone's forgotten perl.
- FractalLP 8y agoSorry, but I didn't catch your example. Can you explain a little more? What does Perl do better there?
- kqr 8y agoDefaultdicts to the rescue! But I think it should be an explicit choice. If you only intend to have lists at some keys, and then accidentally mistype a key, it shouldn't (in my opinion) silently create a list, effectively hiding the bug.
- kamaal 8y agoMost of awk's 'advanced and hard to do' use cases in Perl go like open (my $FILEHANDLE, '<', $file) or die "Cannot open $file\n"; while(<FILEHANDLE>) { chomp; #Do you stuff here } close(FILEHANDLE); Also perl can do various other things awk just can't. For example removing something from a file and then talking to a database or a web service, or do other stuff like parse a JSON or XML. Deal with multiple files, or other advanced use cases. Unicode work, advanced regexes etc etc. In fact the whole point of Perl was Larry wall reaching the upper limits of what one could do with awk, sed and other utilities and having to use them all over the place in C. Then realizing there was a use case for a whole new language.
- greenleafjacob 8y agoIn fact the boilerplate you describe is mostly redundant if you use perl -p since perl -p results in your program running with this wrapper: LINE: while (<>) { ... # your program goes here } continue { print or die "-p destination: $!\n"; }
- flukus 8y agoCorrect me if I'm wrong, but this will only be line based won't it? As far as I can tell there is no equivalent of FS/RS/OFS/ORS that make awk the record (not line) based language it is.
- Annatar 8y agoFor example removing something from a file and then talking to a database or a web service, ... Deal with multiple files, or other advanced use cases. I do exactly this every day with AWK. Solving exactly these kinds of problems features prominently in the AWK programming language book.
- kamaal 8y agoIt goes further. Awk is when you entire use case fits into the category of iterating lines in a file. Perl is that plus more things.
- olskool 8y agoPython was never intended to replace Perl. Perl was designed to extract stuff from text files. Python was designed as a scripting language for system programming. IM(Seldom Humble)O Python beat Perl for two reasons (a) batteries included (CPAN became a clusterfuck) (b) C API lead to stuff like SciPy, NumPy and Pandas. FWIW I've used both Perl and Python professionally and Python rules.
- marklgr 8y agoTo me, the main reason of Python success vs Perl is that it's much easier to learn, so anyone can jump in even with limited programming experience.
- mannykannot 8y agoThat is probably true with regard to its adoption, but, as a Perl fan and user, I have to say that in addition, the limitations of Perl 5's syntax begin to show once you get in to collections of collections, and passing them to functions. If you are doing this sort of thing every day, it becomes second nature, but as a casual, ad-hoc user, it is no longer in my working memory. Maybe Perl 6 fixes these things, but learning it is too far down on my to-do list, where it sits just below Ruby. If I have a problem that can be solved by looping through the lines of a text file and applying some combination of regular expression matching and substitution, the split and join functions, simple arrays and hash tables, I reach for Perl 5.
- reacweb 8y agoPerl is a little older than Python. Most of the ideology of Python was a reaction against Perl (https://wiki.python.org/moin/TOOWTDI https://wiki.python.org/moin/TOOWTDI). Python has always tried to be as good as Perl on everything. I use very much Perl for what it was intended: parsing files (often log files and configuration files), producing reports and launching commands (shell replacement). I have tried to use Python for the same tasks, in particular when some of the files were in xml (xml parsing in Python is nicer than in Perl). Regular expression usage is easier in Perl where multithreading is easier in Python. IMHO, the main handicap of Python vs Perl is the lack of autovivification. Python is very good for teaching (whiteboard interview) or as a scripting language around some big libraries (like tensorflow). At my work, the small glue scripts are almost always in shell or in Perl. The Python applications are being rewritten in Java because of maintenance issues (mainly caused by lack of proper typing). Python does not rule here.
- wenc 8y agoSecond this. Perl replaced both awk and sed. Data scientists marvel at awk/sed but Perl is somehow forgotten. Perl May not be as suited to writing complex programs, but for text processing tasks, it is much more elegant than awk or sed. Perl -pie is a very powerful idiom. I would rather use a subset of Perl than awk.
- asicsp 8y agoyeah.. I like the `s///` syntax in sed/perl than the `sub/gsub` syntax.. plus the regex is lacking in awk.. no backreference(gawk provides, but only in replacement section).. and perl has non-greedy, lookarounds, code in replacement section, etc other nice features I'd like in awk is `tr` and `join`
- ams6110 8y agoWhy add 'tr' and 'join' to awk when they exist on their own? That's part of why people avoid perl. It's very capable, but that wide scope is counter to the unix philosophy that prefers simple, focused utilities that can be combined in pipelines.
- asicsp 8y agolooks like I didn't word it properly... you could ask why have sub/gsub when there is sed... that's because you need that for specific field or string in addition to other processing.. similarly, having tr for specific string/field is useful.. I meant join as in perl's join - to construct a string out of array values with specified separator Some examples: * https://stackoverflow.com/questions/48920626/sort-rows-in-csv-file-without-header-first-column https://stackoverflow.com/questions/48920626/sort-rows-in-cs... * https://stackoverflow.com/questions/45571828/execute-bash-command-inside-awk-and-print-command-output https://stackoverflow.com/questions/45571828/execute-bash-co... * https://stackoverflow.com/questions/48925359/sorting-groups-of-lines https://stackoverflow.com/questions/48925359/sorting-groups-...
- jclulow 8y agoAh, but which subset?
- xyrouter 8y agoIf a 25 year old programmer were to start learning Perl now, what would you recommend: Perl 5 or Perl 6?
- nur0n 8y agoPerl 5 and Perl 6 are two different beasts. Perl 5 is closer in spirit to Awk.
- cygx 8y agoDepends on what you're looking for: In many ways, p6 is the superior language. But as an AWK competitor specifically, p5 might be the better choice for performance reasons alone (while p6 has been improving, as far as regex performance goes, it just isn't there yet [1]). You might even be able to write tighter code in p5 (for one, p6 regexes return objects, not strings, so in cases where you actually want the latter, you'll have to throw in boilerplate). [1] https://gist.github.com/cygx/9c94eefdf6300f726bc698655555d73b#file-graph-md https://gist.github.com/cygx/9c94eefdf6300f726bc698655555d73...
- zimpenfish 8y agoIf you want to get paid for doing it, Perl 5. If you're just mucking about with personal projects, Perl 6.
- FractalLP 8y agoPerl6 is mostly complete. Even though it has been getting significant weekly performance improvements (they have a weekly summary of these things), it still has a long way to go. It has a lot of really cool things such as simple to use concurrency, grammars, gradual typing...etc. It is a big language, but not super hard to learn. A bunch of books have been released on it in the past two years. Perl5 is used by companies now. Some are choosing to write new software in it, but there is a lot of legacy apps. Remember that they are two different languages with some similarities. Perl6 has really good support for making big apps I would say as it has really good OO support out of the box in addition to making FP concepts easy too. Perl5 has atrocious support baked in, but they have the excellent libraries that everyone uses to give them world class support for OO. Still, it feels more natural in Perl6. On another note, Groovy is another great and fast language that runs on the JVM and works very well in the scripting space with a lot of DSLs for GUI, XML, DB...cool stuff.
- conner_bw 8y ago> forgot perl Or maybe, hiding out in PHP? Steeling syntax, releasing versions bigger than 5, smoking regex benchmarks... https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/regexredux.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- walshemj 8y agoAnd Perl is 1000% better than awk/sed when my company got switched to unix based systems we all got sent on a weeks crash course this included sed and awk. I have only used sed once to edit the passwd file on an early linix system when vi wasn't installed and awk never. Though having being trained in sed and awk helped with picking up Perl
- TurboHaskal 8y agoPerl deprecates awk & sed [0]. Whereas Python is a bad substitute for those, and because of that, we're back on Awk. I have the feeling that Python simply succeeded as it seems to be a language designed for people that do not enjoy programming, which sadly seem to be the majority in our industry. The Perl hate is so prevalent in most companies, that you can get into serious issues should you even write a one-liner in it. So here I am, reluctantly using awk/sed and asking too much of Bash/Python. [0] https://www.manning.com/books/minimal-perl https://www.manning.com/books/minimal-perl
- blowski 8y ago> a language designed for people that do not enjoy programming, which sadly seem to be the majority in our industry Could you expand a bit on that comment?
- mjburgess 8y agoIf I can add my interpretation of his comment: Consider which language is sold to people-off-the-street as "How to Program!" it's python. Most of these people do not enjoy the cognitive effort, detailed typing and symbolic work, algorithmic thinking, architectural thinking, linguistic thinking that is "programming". At best, the enjoy the end result, but not the process. Python taught in this way at least, fools people into thinking that programming is about something far simpler and more toylike than it is. Many "hard" disciplines do this: sell children and lay people on toy experiments that momentarily captive but have no relationship to what a practitioner of that discipline does. The older system of recruitment would be to polarized hard, ie., to throw people in at the deep-end to scare off all the people who would waste time/resources training. Who remained really wanted to do that thing (eg. physics, programming, ...). Today we're doing something perhaps vaguely immoral: selling people on a career that has no relationship to the sales pitch.
- blowski 8y agoTo me, that sounds like gatekeeping - as in "it's only programming when it's insanely difficult". Somebody playing football in a Sunday pub league is still playing football, and still loves playing football, even if they're not at the same standard of Lionel Messi. For me at least, I feel the same about programming. I love understanding somebody's problem, and building a solution for them that solves that problem. I normally use PHP (and sometimes Python or Ruby or JavaScript) because they make it easier for me to focus on the problem, rather than language details. I can't always solve the problem because it's too difficult, and perhaps some of my solutions are not 'optimal'. But I feel hurt by the idea that because I don't have a strong understanding of how Python works at a really deep level, I'm not a real programmer. I also think that's a great way to piss people off who are just getting into the industry, and may one day become great programmers - even Linus Torvalds was a junior once. I'd encourage them to keep going, keep learning, and keep helping people solve their problems (and getting paid good money for doing that).
- carlmr 8y ago>And the scary syntax is all in the Rust world. There's something like necessary complexity you can't easily abstract away. I find Rust does a fine job at cleaning up syntax. I'm not a fan of snake_case. Other than that I can't think of anything that's more difficult than the underlying concept in Rust. And it's still close to C (braces and functions) and ML syntax (type after colon and variable name, let bindings) in many ways. Especially compared with a similarly complex language like C++. Now that is scary syntax, if you're not used to it from 20 years of using C++ and developing Stockholm Syndrome.
- ajross 8y agoOh, don't be silly. Exactly the same point you're making about rust can be made about both C++ and perl by expert practitioners. Syntactic complexity is linear with expressive power, that's why it's complex to begin with. You just "like" rust, so you view it as a good tradeoff there and not in other languages that you "dislike". My point above was that this decision is basically one of fashion and not technology. And the proof is how the general consensus about awk has evolved over 20 years as perl has declined.
- quiq 8y ago> Syntactic complexity is linear with expressive power Have you taken a serious look at a lisp? You might be pleasently surprised. Everything that can be said about lisp has probably already been said, but I'd argue that sexprs have a much higher expressive/complexity ratio than say the C++ grammar. Somewhat relative xkcd[1] [1] https://www.explainxkcd.com/wiki/index.php/224:_Lisp https://www.explainxkcd.com/wiki/index.php/224:_Lisp
- carlmr 8y agoI like C++, but it's not the language that you would design if you hadn't accumulated so much cruft over the years. Rust didn't have to support C compatibility and compatibility with earlier C++ standards. So Rust could be designed properly for modern use cases. There's necessary complexity to express certain concepts, but C++ has accumulated a lot of unnecessary complexity over the years. Rust, in terms of GC-less languages with mostly zero-cost abstractions is the simplest language that I've seen. And having a GCless language with memory safety is not just fashion. It's pretty much the greatest single advancement in language design since the GC itself.
- snarfy 8y agoThe cargo cult continues. It's disgusting how much we all rely on intuition and gut feelings when evaluating large swathes of technology. The internet hates perl so people criticize it without even using it. People will use what everybody else is using without actually trying out the options. There's too much information and so we must go with what others have said, and it all becomes hearsay. Keeping up is more like wizardry than engineering. Go with what the crowd says because I can't possibly install all of those libraries, play with the examples, and give my own evaluation. Hey look a new js framework just came out...
- busterarm 8y agoLet's be honest -- people only use what everyone else is using because it lowers the bar to getting hired to 90+% of programming jobs. I can count the number of tool-agnostic development teams that I've met on one hand. Many more have claimed they are when they are not. If you aren't aiming for the top 10% of jobs (vague quality metric that you can interpret as you wish), then you want to have above-average knowledge of _just_ Python, Go, React, Docker and Kubernetes. The situation only changes when high profile current/ex-Googlers (or similar) start talking about a language/tool a lot. Then the mass hops on board that train too.
- adamc 8y agoI agree with the thrust of your comment but not the specifics. There are still a ton of Java and JavaScript (not necessarily React) jobs out there in the "bottom 90%". And I don't think "high-profile" people talking has much effect. Paul Graham talked up lisp for a while, and I was certainly interested (I like lisps), but... there are still precious few jobs that use Lisp. Most people learn technologies that they either need currently or are in use in jobs they know of and might conceivably get.
- busterarm 8y agoEven at shops just hiring for Java or JavaScript, I don't think knowing them is a career advantage. They don't make you more hirable than you would be otherwise. That's really the only reason I left those out. Paul Graham, while a thought-leader of sorts, isn't generally thought of as someone working at either the edge of tech or in large-scale systems. That's why nobody wants to chase the tech he's using vs what Google/Facebook/etc are.
- tannhaeuser 8y ago> And the scary syntax is all in the Rust world. JavaScript is heading in the direction of Haskell, Scala, and (to a lesser extent) Rust, too [1]. [1]: https://medium.freecodecamp.org/here-are-three-upcoming-changes-to-javascript-that-youll-love-387bce1bfb0b https://medium.freecodecamp.org/here-are-three-upcoming-chan...
- jhbadger 8y agoAWK seems far more modern than Perl, though. Defining a function with you know, "function" and actually having parameters with names feels like a 21st century language. Calling things "sub" and having it work off a an array called @_ doesn't. Yes, I know there are packages to add named parameters to Perl, plus Perl6 has them out of the box, but it is weird that things went backwards in a lot of ways when Perl replaced AWK in the mid 1990s.
- kqr 8y agoYou have a really good point. AWK feels incredibly modern for being so old.
- kazinator 8y agoUntil you find that it has no local variables other than function parameters. In a function, local variables can be simulated by defining additional arguments ... and not passing those. Everything that isn't a function parameter is a global! Awk is stupidly confused whether it is a two-namespace or one-namespace language (think Lisp-2 vs. Lisp-1). For instance, this works: function x() { print "x called" } function foo(x) { x() # function x, not parameter x. } but in other situations the two spaces are conflated. For instance sin = 3 is a syntax error because sin is the built-in sinusoid function, even though you're using it as a variable, which shouldn't interfere with function use like sin(3.14).
- coliveira 8y agoActually what you show is that there is a clear rule. First, the symbol is treated as a function. If it is not in the function space, then it is used as a variable name. To make a finer distinction awk would need some form of local variable declaration, which it clearly hasn't.
- kazinator 8y ago> First, the symbol is treated as a function. If it is not in the function space, then it is used as a variable name. That is simply not the case. x() unambiguously treats x as a function, and will fail if there is no such function, even if there is a variable x. $ awk 'function foo(x) { x() } BEGIN { foo(42) }' awk: cmd. line:2: fatal: function `x' not defined > To make a finer distinction awk would need some form of local variable declaration, which it clearly hasn't. Also not the case. The purely syntactic context distinguishes whether or not the identifier is being used as a variable or function. Awk sometimes uses it, sometimes not. This doesn't diagnose: function x() { } function foo(x) { x() # function call allowed x = 3 # assignment allowed } BEGIN { foo(42) } But: function foo(x) { } BEGIN { foo = 3 } fatal: function `foo' called with space between name and `(', or used as a variable or an array Why isn't it a problem that the function x() is used as a variable?
- justwalt 8y agoI didn’t know Perl was capable of filling the same niche as awk, and I think awk is absolutely wonderful. How does Perl improve on it?
- MichaelMoser123 8y agoPerl5 and perl6 are very different languages - however after perl5 there was nobody left to listen... (Disclaimer: I got used to perl5 and I still think it's quite reasonable)
- vasili111 8y agoWhat about tcl? Is it better than perl for text proccessing
- deleted 8y ago[deleted]