4 ms·

I've found this seriously cool! While not my primary role at my job, I often find myself dealing with lots of disparate data sets, usually needing to do some s

I've found this seriously cool! While not my primary role at my job, I often find myself dealing with lots of disparate data sets, usually needing to do some sort of manipulation, cleaning, searching, etc. Every now and then encounter something like this and it seems to me that there are potentially a nice set of command line tools/utilities that I should be adding to my belt. Anywhere I should particularly start taking a look? Like, if my goal is to because much better at wrangling CSV/text-delimited files, searching across folders of docs for numbers, etc., where is my first entry point into trying to become much more proficient at it?

pdimitar 4y ago
Here's a recent gateway post: https://jvns.ca/blog/2022/04/12/a-list-of-new-ish--command-line-tools/ https://jvns.ca/blog/2022/04/12/a-list-of-new-ish--command-l... There are even more. :)
niccl 4y ago
I do a lot of that sort of thing, and my go-to tools are grep (with full regular expressions), sort, uniq, head, tail, sed, and, the 200 pound gorilla (not as heavy as it could be) awk. Then if all else fails, Python
NegativeLatency 4y ago
fd, bat, fzf are good starting points also mdfind if you're on macos I'm hesitant to go fully into "new" stuff so I can maintain skills on random *nix boxes I need to ssh into.
benhoyt 4y ago
I highly recommend you learn the basics of AWK, which is a little language that lets you match on and run commands on every line of input (so AWK "scripts" are usually one-liners). The very basics can be learn in 5-10 minutes, the whole language in a couple of hours (read the first three chapters of "The AWK Programming Language" by A, W, K.) For searching recursively through folders, I recommend ripgrep (rg), though it won't do number conversions and the like. Self-promotion (but I think it's useful here) is my GoAWK project, which is a POSIX-compatible version of AWK, with the recent addition of proper CSV support: https://benhoyt.com/writings/goawk-csv/ https://benhoyt.com/writings/goawk-csv/
- jnordwick 4y ago
  Why would you do a posix compat awk? I loved awk, but it needs to be updated adn refined into a better language instead of just recreating the old one.
  - benhoyt 4y ago
    Compatibility: so it can run all the existing AWK scripts without modification, and people can refer to existing AWK documentation when writing scripts, and they'll just work under GoAWK. For a 45-year old language, it's actually surprisingly good. There are a few annoyances and clumsy things (getting substrings is a function call without nice str[start:end] syntax, regex matches are harder than they need to be, string concatenation syntax is weird, arrays aren't nestable (except in Gawk), and so on). I've actually thought it'd be great to "fix" these things and make a nicer AWK language, but the problem is then no one would use it. :-)
btingle 4y ago
Try jq for parsing/selecting JSON data.

pdimitar 4y ago

Here's a recent gateway post: https://jvns.ca/blog/2022/04/12/a-list-of-new-ish--command-line-tools/ https://jvns.ca/blog/2022/04/12/a-list-of-new-ish--command-l... There are even more. :)

niccl 4y ago

I do a lot of that sort of thing, and my go-to tools are grep (with full regular expressions), sort, uniq, head, tail, sed, and, the 200 pound gorilla (not as heavy as it could be) awk. Then if all else fails, Python

NegativeLatency 4y ago

fd, bat, fzf are good starting points also mdfind if you're on macos I'm hesitant to go fully into "new" stuff so I can maintain skills on random *nix boxes I need to ssh into.

benhoyt 4y ago

I highly recommend you learn the basics of AWK, which is a little language that lets you match on and run commands on every line of input (so AWK "scripts" are usually one-liners). The very basics can be learn in 5-10 minutes, the whole language in a couple of hours (read the first three chapters of "The AWK Programming Language" by A, W, K.) For searching recursively through folders, I recommend ripgrep (rg), though it won't do number conversions and the like. Self-promotion (but I think it's useful here) is my GoAWK project, which is a POSIX-compatible version of AWK, with the recent addition of proper CSV support: https://benhoyt.com/writings/goawk-csv/ https://benhoyt.com/writings/goawk-csv/

jnordwick 4y ago
Why would you do a posix compat awk? I loved awk, but it needs to be updated adn refined into a better language instead of just recreating the old one.
- benhoyt 4y ago
  Compatibility: so it can run all the existing AWK scripts without modification, and people can refer to existing AWK documentation when writing scripts, and they'll just work under GoAWK. For a 45-year old language, it's actually surprisingly good. There are a few annoyances and clumsy things (getting substrings is a function call without nice str[start:end] syntax, regex matches are harder than they need to be, string concatenation syntax is weird, arrays aren't nestable (except in Gawk), and so on). I've actually thought it'd be great to "fix" these things and make a nicer AWK language, but the problem is then no one would use it. :-)

btingle 4y ago

Try jq for parsing/selecting JSON data.