4 ms·

I use awk to write csv-parsers for my personal finance setup. They take transaction logs and convert them into a ledger format (https://www.ledger-cli.org/ http

by gen220 7y ago

I use awk to write csv-parsers for my personal finance setup. They take transaction logs and convert them into a ledger format (https://www.ledger-cli.org/ https://www.ledger-cli.org/). There's some pretty trivial if/else branches based on regexp's, which awk is pretty good at expressing. I'd normally write this sort of thing in python. The awk program is usually smaller than a comparable python program, but for me the main selling point is that awk programs a more "UNIX-pipe-native" than python programs are. At some point, I'll probably rewrite these programs in a more "serious" programming language, once the file sizes get too big or something, but for now they're working great and are easy to extend.

mauvehaus 7y ago
I've used it for this too, but things tend to go south pretty quick if you have escaped or quoted commas. That, unfortunately, is where I usually break down and pull out python.
- gen220 7y ago
  hehe, I hit that bump in the road too, but maybe I was too stubborn. FWIW, you can get around it. I just took a peek because I couldn't remember how to do it off the top of my head. It looks like I had to use gawk -F ',' -v FPAT='([^,]+)|("[^"/]+")' to get the behavior you're looking for. Seems like I nabbed it from https://www.gnu.org/software/gawk/manual/html_node/Splitting-By-Content.html https://www.gnu.org/software/gawk/manual/html_node/Splitting... Agreed that this is the sane point at which to pull out python.
  - mauvehaus 7y ago
    I hadn't run across that one yet. Thank you! The risk of that seems to be if CSV allows embedded escaped quotes in a quoted string. Does it? I don't know. And CSV is pretty loosely defined. For most people it's probably "whatever Excel emits or ingests". And I think that's why we're on the same page about pulling out python and using a module where somebody has explored what the corner cases are and dealt with them for us already.
    - gen220 7y ago
      Yea CSV is famously bespoke, I don’t know if it’s ever been standardized? In my experience, each implementation is more or less unique. Even if you allow escaping, different formats permit different methods of escaping and you might have to account for each! shudders It’s much easier to support that in python than it is in awk with regexp. The awk route will eventually make you a regexp wizard, though, which confers it’s own benefits. :)

by gen220 7y ago

mauvehaus 7y ago

I've used it for this too, but things tend to go south pretty quick if you have escaped or quoted commas. That, unfortunately, is where I usually break down and pull out python.

gen220 7y ago
hehe, I hit that bump in the road too, but maybe I was too stubborn. FWIW, you can get around it. I just took a peek because I couldn't remember how to do it off the top of my head. It looks like I had to use gawk -F ',' -v FPAT='([^,]+)|("[^"/]+")' to get the behavior you're looking for. Seems like I nabbed it from https://www.gnu.org/software/gawk/manual/html_node/Splitting-By-Content.html https://www.gnu.org/software/gawk/manual/html_node/Splitting... Agreed that this is the sane point at which to pull out python.
- mauvehaus 7y ago
  I hadn't run across that one yet. Thank you! The risk of that seems to be if CSV allows embedded escaped quotes in a quoted string. Does it? I don't know. And CSV is pretty loosely defined. For most people it's probably "whatever Excel emits or ingests". And I think that's why we're on the same page about pulling out python and using a module where somebody has explored what the corner cases are and dealt with them for us already.
  - gen220 7y ago
    Yea CSV is famously bespoke, I don’t know if it’s ever been standardized? In my experience, each implementation is more or less unique. Even if you allow escaping, different formats permit different methods of escaping and you might have to account for each! shudders It’s much easier to support that in python than it is in awk with regexp. The awk route will eventually make you a regexp wizard, though, which confers it’s own benefits. :)