3 ms·
Unless your data contains spaces, tabs, or, god forbid, newlines. Unix pipeline tools lack any sort of useful data structuring capabilities, making them appropr
by ambrop7 8y ago
Unless your data contains spaces, tabs, or, god forbid, newlines. Unix pipeline tools lack any sort of useful data structuring capabilities, making them appropriate for one-off tasks at most.
- johncolanduoni 8y agoYeah I feel like there’s a pretty broad medium between Hadoop and awk. I generally find a python script to be a much clearer solution than an dense line of awk; the latter tends to turn into clever code golf pretty quickly.
- devnonymous 8y agoWhat are you on about? I can't think of one Unix command that manipulates structured data and doesn't offer a delimiter option. Furthermore the entire point (if you take a moment to think about it) of pipelines is, instead of all commands supporting your preferred delimiter, you need one command to translate (I don't know perhaps something named tr? ) in the pipeline to deal with such limitations. Do you seriously think non-whitespace separated structured records is a novel idea which the simpler times of Unix didn't have to deal with? Have you looked at the passwd file? /rant
- mrighele 8y agoSpaces, tab and newlines are not a problem. The fact that there are some standard tools available doesn't mean you are limited to that. If you have a CSV file with spaces and newlines use cvskit or a small python script importing the relevant library. If you have to parse JSON file you can use jq to pick the relevant fields regardless of how the document is formatted. You can even process binary data as long as the file format is understood by the tool.
- zimpenfish 8y agoTranscoding CSV to TSV using csvkit has been a life saver for me the last few months due to an old industry's insistence on CSV as an information delivery method. jq has previously been a lifesaver too.
- deleted 8y ago[deleted]