5 ms·

I don't think it's much better to use SQL, it's literally killing flies with nuclear bombs. Awk does the job perfectly well and you don't have to spin up a SQL

I don't think it's much better to use SQL, it's literally killing flies with nuclear bombs. Awk does the job perfectly well and you don't have to spin up a SQL database, import the data, deal with any possible problem when importing the data. I don't know why a lot of people have this aversion to the console when it can do the job fairly well, specially in these cases where you just want a quick check.

mercer 8y ago
How would SQLite fare in regards to your concerns? You won't have to 'spin up' a database and importing from various sources, AFAIK, is pretty easy these days (and OP even goes out of their way to deal with VARCHAR fields). It strikes me as much easier than having to figure out Awk, but that's assuming most people are more likely to know SQL and/or use it in the future. I mean, I'd be happy to try and solve the problem with Awk because it's been high on my list of things to learn, but SQL does seem like an easier solution in most cases.
- gjulianm 8y ago
  You still have to create the SQLite database and have the client handy. Awk is there by default in all *nix systems, and for these basic tasks it's incredibly easy (see http://www.hcs.harvard.edu/~dholland/computers/awk.html http://www.hcs.harvard.edu/~dholland/computers/awk.html for a quick intro). Once you know it you can use it to process a lot of text files in the console without any kind of worry. Maybe it's a preprocess step before importing data to a database, maybe you're just exploring a dataset, maybe you want to modify some command's output in the middle of a pipe. I learned awk a few years ago and it does not have equivalent when the only thing you want to do is to consume column-based text and do some simple processing: it is simple and incredibly fast.
  - mercer 8y ago
    Thanks for the link and the motivation to keep reading the AWK book I started a while ago!
onychomys 8y ago
Identifying the problem is only half the battle, though. The end goal would be to go through and fix each record. Surely that's easier done in a database tool than on the command line. Additionally, these data presumably live in a database at the museums they're from. They put them on the web as CSV (or similar) so that the public can see them, but the museum IT guy would end up making the eventual changes in their tracking system anyway.
kbenson 8y ago
> Awk does the job perfectly well I've always been of the opinion that you're better served by learning Perl and using it. The syntax for quick things is very similar to Awk, but you also get all of CPAN to back you up in case you need to do something more complex. For example, real CSV parsing, HTML parsing with queryselector access, complex record destructuring and/or inner-record decompression, and sprintf (among many others) for for output. I still do a lot of cat / grep / cut / sort / uniq type commands, but whenever I have need of a little complexity, Perl is definitely my go to tool. I think it's already installed in most common installation choices for most distros.
- gjulianm 8y ago
  I don't think I've used awk scripts more than in one or two occasions, I use it mostly for one liners and there I like awk's simplicity. When I need something more complex I resort to Python which is slower but usually manageable, you're right that if I knew Perl I'd have both tools in one language.

mercer 8y ago

How would SQLite fare in regards to your concerns? You won't have to 'spin up' a database and importing from various sources, AFAIK, is pretty easy these days (and OP even goes out of their way to deal with VARCHAR fields). It strikes me as much easier than having to figure out Awk, but that's assuming most people are more likely to know SQL and/or use it in the future. I mean, I'd be happy to try and solve the problem with Awk because it's been high on my list of things to learn, but SQL does seem like an easier solution in most cases.

gjulianm 8y ago
You still have to create the SQLite database and have the client handy. Awk is there by default in all *nix systems, and for these basic tasks it's incredibly easy (see http://www.hcs.harvard.edu/~dholland/computers/awk.html http://www.hcs.harvard.edu/~dholland/computers/awk.html for a quick intro). Once you know it you can use it to process a lot of text files in the console without any kind of worry. Maybe it's a preprocess step before importing data to a database, maybe you're just exploring a dataset, maybe you want to modify some command's output in the middle of a pipe. I learned awk a few years ago and it does not have equivalent when the only thing you want to do is to consume column-based text and do some simple processing: it is simple and incredibly fast.
- mercer 8y ago
  Thanks for the link and the motivation to keep reading the AWK book I started a while ago!

onychomys 8y ago

Identifying the problem is only half the battle, though. The end goal would be to go through and fix each record. Surely that's easier done in a database tool than on the command line. Additionally, these data presumably live in a database at the museums they're from. They put them on the web as CSV (or similar) so that the public can see them, but the museum IT guy would end up making the eventual changes in their tracking system anyway.

kbenson 8y ago

> Awk does the job perfectly well I've always been of the opinion that you're better served by learning Perl and using it. The syntax for quick things is very similar to Awk, but you also get all of CPAN to back you up in case you need to do something more complex. For example, real CSV parsing, HTML parsing with queryselector access, complex record destructuring and/or inner-record decompression, and sprintf (among many others) for for output. I still do a lot of cat / grep / cut / sort / uniq type commands, but whenever I have need of a little complexity, Perl is definitely my go to tool. I think it's already installed in most common installation choices for most distros.

gjulianm 8y ago
I don't think I've used awk scripts more than in one or two occasions, I use it mostly for one liners and there I like awk's simplicity. When I need something more complex I resort to Python which is slower but usually manageable, you're right that if I knew Perl I'd have both tools in one language.