5 ms·

What are you using Perl/awk for that it's needed so often? I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently

What are you using Perl/awk for that it's needed so often? I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them. Is this a certain kind of engineering job that requires searching through text files so often and requiring specialized tools? I've managed my whole career with ctrl-f, and highlight-all matches.

eternauta3k 7y ago
> I've managed my whole career with ctrl-f, and highlight-all matches. Imagine you have to do this for 100 files.
- saberience 7y ago
  Fair :)
mongol 7y ago
Analysis of large log files. Granted, today Splunk and other tools are more used.
Timpy 7y ago
I'm sure use cases vary to some degree, but awk clicked for me when I had some data that wasn't formatted well. I didn't use it in a script, I just worked in the shell and kept iterating until it looked good, then redirected it into a text file. I use it for quick one-offs like this all the time now. I've also used awk to get button presses from an input stream on a MIDI controller. For me, I found that the up front cost of learning a few awk commands quickly made it's return on investment.
- mauvehaus 7y ago
  This. It's often a good enough tool for solving problems that technically require a specialized tool. Except learning the specialized tool takes an order of magnitude more time than iterating on awk to get a good-enough solution. I'm not proud to admit it, but I've used awk in a couple places where I should have properly used `expect' instead. Except, I don't know expect well and haven't gotten to the point where it was worth the investment in learning it.
Wilem82 7y ago
Using any kind of management position as justification for anything tech-related won't have the effect you're probably aiming for. Dev going management is always downshifting.
submeta 7y ago
There are hundreds of use-cases. It is not important what the use-cases are. - There are two types of developers: Those who use hacking to solve all kinds of tasks that need hundreds of manual steps. And those who never think of automating smaller steps. The second kind of devs can be very good software engineers, but they prefer IDEs instead of tweaking Vim or Emacs. The first kind of dev will look for possiblities to automate repeated steps, opportunities for tweaking, transforming, hacking. For the fun of it.
- dan-robertson 7y ago
  I think this argument is unfairly dismissive. I think it depends a lot on the sort of software one works with. A main use of awk and other unix tools for me is as-hoc data munging, looking through production logs, taking bits of data from different sources and comparing them. If you make eg gui applications and sell them or work as a contractor developing in-house applications for clients, you probably don’t see a lot of production logs like that or process them that way. Any data that you expect to process ought to go into a database and then you can use sql (where joins work much better than the unix join command and you can use the actual structure of the data instead of trying to tease it out with the smallest simplest code you can think of). You might be using a debugger or backtracks or reproduction in test to deal with/investigate production issues rather than starting with logs. On the other hand, many other companies will have big production systems that run on lots of different virtual machines and produce lots of logs and have issues that are hard to reproduce (or maybe the logs are easier), and have lots of data spread about random places such that it may be easier to do the hacky thing for one-off cases or to prototype or whatever rather than Doing things more thoroughly or properly. And this set up will also lead to tools that are designed to fit into the rest of the unix ETL tack by the way they output or input data. On the second point, I think that even as a die-hard emacs user I wouldn’t recommend that anyone try to use emacs as an ide for something like java (or maybe c++). I’d probably try to use emacs a myself but I think my java-editing experience would be worse than for those people who do it with an IDE. More realistically, I’d try to avoid writing java if at all possible.
indymike 7y ago
Well, you might have have got the job done faster with awk (and other tools like it)... ETL and anything dealing with importing large data sets come to mind.
scruple 7y ago
Are you working in a Linux development environment? Targeting Linux systems? You're not going to come across much of a use for awk if you're in the Windows world. I'm making an assumption based on C#. I've used perl and awk extensively for all sorts of things. Log parsing embedded device logs to generate reports was a big one for me, that's where I really learned both languages. And they are superb tools for that task. We also used awk quite a bit for configuration parsing as a sort of intermediary between different processes and scripts on those devices. As I moved into web development, I've found myself using both much less. I haven't touched perl more than once in the last 6 years -- I was given a script to maintain recently but it only needed an hour or two of my time to add a small feature to. That perl script is part of a legacy build system. I still use awk every couple of months but it's just part of some pipeline on the command line.
mauvehaus 7y ago
I don't, and that's half the problem :-) If I did, I'd probably remember enough perl that I'd have never bit the bullet and learned awk! To answer the serious question, it's often enough for a first order solution to any problem where you have line and field separated text. Pulling instances of "something weird" out of log files, for instance, is a great use for awk, especially if the fingerprint of "something weird" is scattered through fields in a line in a way that makes grep cumbersome. Or if it spans a couple of lines, especially if you're dealing with a log from a multi-threaded app where there might be irrelevant lines interleaved with the ones you care about. I haven't written much production-grade awk, but I've often used it as a tool to understand a problem well enough to write a production-grade solution to a problem or fix bugs in application code.
masto 7y ago
In all that time, have you strictly been writing code, on computers that came set up out of the box, that runs on computers managed by other people? This is not an insult, I’m aware such things exist, but I seem to travel in different circles. It may come from a certain era of being into computers as a hobby for a long time before I could call myself a software engineer, and having to do what we now call DevOps necessarily by working for companies that didn’t have the resources to hire a fleet of system administrators. As a result I may live in a bubble where the friends and colleagues I associate with are as comfortable with a command line as they are with an IDE, and frequently jump between the two. I mention DevOps because it is hard to think of a situation where you’re just coding Java and C# all day that truly requires awk-or-similar. But just the other day I had a situation where I copied the contents of a CD-ROM to a web server — drag and drop on my Mac — and ended up with the files all in upper case. The program I was trying to run was failing because it requested everything with lower-case URLs. There were hundreds and hundreds of files in a bunch of directories, so I fixed it with Perl in about 30 seconds. I’m curious if that’s the sort of problem you never encounter, or if it is, what you reach for.
scottlocklin 7y ago
People (me anyway) used perl back in the dark ages of the 90s for the same sorts of things people use python for now. It was really the only option for open source general purpose interpreters for quite a while. People still use it in biotech I think, or were 5 years ago; the text processing capabilities are useful for genomic data. Awk/sed and the ETL stack that comes with every Unix-like OS (aka od, tr, cut, sort) and all that are superb data wrangling tools. Log file parsing, data cleaning, even actual data science at scale can be done with these tools. They're extremely efficient, as they're designed from an era when pretty much all interesting data was comparable to or much larger than memory size. As such, you can do a lot of stuff with them that most people don't imagine is even possible. FWIIW for high end data scientists; I don't consider knowledge of these tools to be optional at all. Anyone who hasn't used them in their career hasn't worked on serious problems, and is probably the kind of educated idiot who will suggest you do a job in a giant Hadoop cluster that you could easily do on one machine.
kshacker 7y ago
So I just wrote one for personal use. I was looking for a duplicate file/photo finder and read some reviews about losing data so I said why would I trust a third party for my data and wrote a dupe finder. Use a hash (md5) to make a match (it is not perfect but it works) and then sort and print to show me the dupes. The first version was 45 lines of code although I could have reduced lines if I tried. You don't need awk strictly but I used it to prettify my output and add some filters. Current code with lots more bells and whistles is less than 150 lines (including white space) of bash and awk. And it works for me and I trust it. I could set up a database, write multiple layers of code ... but really?
andrewl 7y ago
The author offers one answer to this in his opening text: I say it's useful on servers because log files, dump files, or whatever text format servers end up dumping to disk will tend to grow large, and you'll have many of them per server. If you ever get into the situation where you have to analyze gigabytes of files from 50 different servers without tools like Splunk or its equivalents, it would feel fairly bad to have to download all these files locally to then drive some forensics on them. This personally happens to me when some Erlang nodes tend to die and leave a crash dump of 700MB to 4GB behind, or on smaller individual servers (say a VPS) where I need to quickly go through logs, looking for a common pattern.
DLA 7y ago
You ever deal with very large sets of large text files? You really CTRL-f through multi-GB files? Have you never needed to transform data between formats or yank out subsets of data for processing? You may have just working on business systems that did not deal with significant data volumes or data you did not control (coming from some other organization, for example).

eternauta3k 7y ago

> I've managed my whole career with ctrl-f, and highlight-all matches. Imagine you have to do this for 100 files.

saberience 7y ago
Fair :)

mongol 7y ago

Analysis of large log files. Granted, today Splunk and other tools are more used.

Timpy 7y ago

I'm sure use cases vary to some degree, but awk clicked for me when I had some data that wasn't formatted well. I didn't use it in a script, I just worked in the shell and kept iterating until it looked good, then redirected it into a text file. I use it for quick one-offs like this all the time now. I've also used awk to get button presses from an input stream on a MIDI controller. For me, I found that the up front cost of learning a few awk commands quickly made it's return on investment.

mauvehaus 7y ago
This. It's often a good enough tool for solving problems that technically require a specialized tool. Except learning the specialized tool takes an order of magnitude more time than iterating on awk to get a good-enough solution. I'm not proud to admit it, but I've used awk in a couple places where I should have properly used `expect' instead. Except, I don't know expect well and haven't gotten to the point where it was worth the investment in learning it.

Wilem82 7y ago

Using any kind of management position as justification for anything tech-related won't have the effect you're probably aiming for. Dev going management is always downshifting.

submeta 7y ago

There are hundreds of use-cases. It is not important what the use-cases are. - There are two types of developers: Those who use hacking to solve all kinds of tasks that need hundreds of manual steps. And those who never think of automating smaller steps. The second kind of devs can be very good software engineers, but they prefer IDEs instead of tweaking Vim or Emacs. The first kind of dev will look for possiblities to automate repeated steps, opportunities for tweaking, transforming, hacking. For the fun of it.

dan-robertson 7y ago
I think this argument is unfairly dismissive. I think it depends a lot on the sort of software one works with. A main use of awk and other unix tools for me is as-hoc data munging, looking through production logs, taking bits of data from different sources and comparing them. If you make eg gui applications and sell them or work as a contractor developing in-house applications for clients, you probably don’t see a lot of production logs like that or process them that way. Any data that you expect to process ought to go into a database and then you can use sql (where joins work much better than the unix join command and you can use the actual structure of the data instead of trying to tease it out with the smallest simplest code you can think of). You might be using a debugger or backtracks or reproduction in test to deal with/investigate production issues rather than starting with logs. On the other hand, many other companies will have big production systems that run on lots of different virtual machines and produce lots of logs and have issues that are hard to reproduce (or maybe the logs are easier), and have lots of data spread about random places such that it may be easier to do the hacky thing for one-off cases or to prototype or whatever rather than Doing things more thoroughly or properly. And this set up will also lead to tools that are designed to fit into the rest of the unix ETL tack by the way they output or input data. On the second point, I think that even as a die-hard emacs user I wouldn’t recommend that anyone try to use emacs as an ide for something like java (or maybe c++). I’d probably try to use emacs a myself but I think my java-editing experience would be worse than for those people who do it with an IDE. More realistically, I’d try to avoid writing java if at all possible.

indymike 7y ago

Well, you might have have got the job done faster with awk (and other tools like it)... ETL and anything dealing with importing large data sets come to mind.

scruple 7y ago

Are you working in a Linux development environment? Targeting Linux systems? You're not going to come across much of a use for awk if you're in the Windows world. I'm making an assumption based on C#. I've used perl and awk extensively for all sorts of things. Log parsing embedded device logs to generate reports was a big one for me, that's where I really learned both languages. And they are superb tools for that task. We also used awk quite a bit for configuration parsing as a sort of intermediary between different processes and scripts on those devices. As I moved into web development, I've found myself using both much less. I haven't touched perl more than once in the last 6 years -- I was given a script to maintain recently but it only needed an hour or two of my time to add a small feature to. That perl script is part of a legacy build system. I still use awk every couple of months but it's just part of some pipeline on the command line.

mauvehaus 7y ago

I don't, and that's half the problem :-) If I did, I'd probably remember enough perl that I'd have never bit the bullet and learned awk! To answer the serious question, it's often enough for a first order solution to any problem where you have line and field separated text. Pulling instances of "something weird" out of log files, for instance, is a great use for awk, especially if the fingerprint of "something weird" is scattered through fields in a line in a way that makes grep cumbersome. Or if it spans a couple of lines, especially if you're dealing with a log from a multi-threaded app where there might be irrelevant lines interleaved with the ones you care about. I haven't written much production-grade awk, but I've often used it as a tool to understand a problem well enough to write a production-grade solution to a problem or fix bugs in application code.

masto 7y ago

In all that time, have you strictly been writing code, on computers that came set up out of the box, that runs on computers managed by other people? This is not an insult, I’m aware such things exist, but I seem to travel in different circles. It may come from a certain era of being into computers as a hobby for a long time before I could call myself a software engineer, and having to do what we now call DevOps necessarily by working for companies that didn’t have the resources to hire a fleet of system administrators. As a result I may live in a bubble where the friends and colleagues I associate with are as comfortable with a command line as they are with an IDE, and frequently jump between the two. I mention DevOps because it is hard to think of a situation where you’re just coding Java and C# all day that truly requires awk-or-similar. But just the other day I had a situation where I copied the contents of a CD-ROM to a web server — drag and drop on my Mac — and ended up with the files all in upper case. The program I was trying to run was failing because it requested everything with lower-case URLs. There were hundreds and hundreds of files in a bunch of directories, so I fixed it with Perl in about 30 seconds. I’m curious if that’s the sort of problem you never encounter, or if it is, what you reach for.

scottlocklin 7y ago

People (me anyway) used perl back in the dark ages of the 90s for the same sorts of things people use python for now. It was really the only option for open source general purpose interpreters for quite a while. People still use it in biotech I think, or were 5 years ago; the text processing capabilities are useful for genomic data. Awk/sed and the ETL stack that comes with every Unix-like OS (aka od, tr, cut, sort) and all that are superb data wrangling tools. Log file parsing, data cleaning, even actual data science at scale can be done with these tools. They're extremely efficient, as they're designed from an era when pretty much all interesting data was comparable to or much larger than memory size. As such, you can do a lot of stuff with them that most people don't imagine is even possible. FWIIW for high end data scientists; I don't consider knowledge of these tools to be optional at all. Anyone who hasn't used them in their career hasn't worked on serious problems, and is probably the kind of educated idiot who will suggest you do a job in a giant Hadoop cluster that you could easily do on one machine.

kshacker 7y ago

So I just wrote one for personal use. I was looking for a duplicate file/photo finder and read some reviews about losing data so I said why would I trust a third party for my data and wrote a dupe finder. Use a hash (md5) to make a match (it is not perfect but it works) and then sort and print to show me the dupes. The first version was 45 lines of code although I could have reduced lines if I tried. You don't need awk strictly but I used it to prettify my output and add some filters. Current code with lots more bells and whistles is less than 150 lines (including white space) of bash and awk. And it works for me and I trust it. I could set up a database, write multiple layers of code ... but really?

andrewl 7y ago

The author offers one answer to this in his opening text: I say it's useful on servers because log files, dump files, or whatever text format servers end up dumping to disk will tend to grow large, and you'll have many of them per server. If you ever get into the situation where you have to analyze gigabytes of files from 50 different servers without tools like Splunk or its equivalents, it would feel fairly bad to have to download all these files locally to then drive some forensics on them. This personally happens to me when some Erlang nodes tend to die and leave a crash dump of 700MB to 4GB behind, or on smaller individual servers (say a VPS) where I need to quickly go through logs, looking for a common pattern.

DLA 7y ago

You ever deal with very large sets of large text files? You really CTRL-f through multi-GB files? Have you never needed to transform data between formats or yank out subsets of data for processing? You may have just working on business systems that did not deal with significant data volumes or data you did not control (coming from some other organization, for example).