3 ms·

This is a common theme with the classic unixy text processing based workflows, they work really nicely when the incoming data is nice and simple but fall apart

by zokier 3y ago

This is a common theme with the classic unixy text processing based workflows, they work really nicely when the incoming data is nice and simple but fall apart on edge cases. And because often also the error handling is incomplete it might be difficult to know that the pipeline is producing garbage output. Related example is the recently discussed problems with filenames, and the many pitfalls that scripts can stumble upon when handling them.

hfkwer 3y ago
> And because often also the error handling is incomplete it might be difficult to know that the pipeline is producing garbage output. The thing is that from the point of view of each individual tool, there isn't even an error! You ask awk to split on commas and give you the second field, it splits on commas and gives you the second field. It opened the file no problem, it didn't run out of memory writing its output to stdout... There were no errors to speak of from what it could see. The "error" is an error in the overall logic of the code.
keybored 3y ago
> but fall apart on edge cases. The problem is that edge cases are not really edge cases… in certain locales. ;) Someone could be using commas as the decimal separator since that’s how they do it ’round these parts.[1][2] So it’s not like they’re storing “weird” stuff in tabular format, like HTML blog articles; they’re just storing numbers. Of course, Excel and other software has tried to be way too clever about it with different separators depending on locale, which makes the whole pipeline complex and context-dependent. [1] A surprising amount of countries have this convention, including my own… doesn’t make sense to me since it becomes harder to list numbers. [2] And the number one selling point of CSV is that it is “human editable”, so of course someone is going to do that at some point in the pipeline.
dbro 3y ago
That is correct, the data needs to be simple where the delimiter characters are never embedded inside a quoted field. I wrote a simple (and fast) utility to ensure that CSV files are handled properly by all the standard UNIX command line data tools. If you like using awk, sed, cut, tr, etc. then it may be useful to you. <https://github.com/dbro/csvquote https://github.com/dbro/csvquote> Using it with the first example command from this article would be csvquote file.csv | awk -F, '{print $1}' | csvquote -u By using the "-u" flag in the last step of the pipeline, all of the problematic quoted delimiters get restored.

by zokier 3y ago

hfkwer 3y ago

> And because often also the error handling is incomplete it might be difficult to know that the pipeline is producing garbage output. The thing is that from the point of view of each individual tool, there isn't even an error! You ask awk to split on commas and give you the second field, it splits on commas and gives you the second field. It opened the file no problem, it didn't run out of memory writing its output to stdout... There were no errors to speak of from what it could see. The "error" is an error in the overall logic of the code.

keybored 3y ago

> but fall apart on edge cases. The problem is that edge cases are not really edge cases… in certain locales. ;) Someone could be using commas as the decimal separator since that’s how they do it ’round these parts.[1][2] So it’s not like they’re storing “weird” stuff in tabular format, like HTML blog articles; they’re just storing numbers. Of course, Excel and other software has tried to be way too clever about it with different separators depending on locale, which makes the whole pipeline complex and context-dependent. [1] A surprising amount of countries have this convention, including my own… doesn’t make sense to me since it becomes harder to list numbers. [2] And the number one selling point of CSV is that it is “human editable”, so of course someone is going to do that at some point in the pipeline.

dbro 3y ago

That is correct, the data needs to be simple where the delimiter characters are never embedded inside a quoted field. I wrote a simple (and fast) utility to ensure that CSV files are handled properly by all the standard UNIX command line data tools. If you like using awk, sed, cut, tr, etc. then it may be useful to you. <https://github.com/dbro/csvquote https://github.com/dbro/csvquote> Using it with the first example command from this article would be csvquote file.csv | awk -F, '{print $1}' | csvquote -u By using the "-u" flag in the last step of the pipeline, all of the problematic quoted delimiters get restored.