3 ms·

It's even worse than that. Most non-trivial, and some trivial scripts and one liners rely on naive parsing (regex/cut etc) because that's the only tool in the t

by hggigg 2y ago

It's even worse than that. Most non-trivial, and some trivial scripts and one liners rely on naive parsing (regex/cut etc) because that's the only tool in the toolbox. This resulted in some horrific problems over the years. I take a somewhat hard line that scripts and terminals are for executing sequential commands naively only. Call it "glue". If you're writing a program, use a higher level programming language and parse things properly. This problem of course does tend to turn up in higher level languages but at least you can pull a proper parser in off the shelf there if you need to. Notably if I see anyone parsing CSVs with cut again I'm going to die inside. Try unpicking a problem where someone put in the name field "Smith, Bob"...

chasil 2y ago
You might have enjoyed DCL under VMS. It did not immediately succumb to envy of the Korn shell.
Bluecobra 2y ago
> Notably if I see anyone parsing CSVs with cut again I'm going to die inside. Try unpicking a problem where someone put in the name field "Smith, Bob"... How do you tackle this? Would you count the numbers of commas in each line then manually fix the lines that contain more fields?
- hggigg 2y ago
  You parse them and reject CSVs that do not conform. There is absolutely no way to reason about a malformed CSV.
- harry8 2y ago
  http://www.catb.org/~esr/writings/taoup/html/ch05s02.html http://www.catb.org/~esr/writings/taoup/html/ch05s02.html paragraph titled DSV style. (Yeah esr, not a fan, whatever...) Csv sucks no matter what, there is no one csv spec. Then even if you assume the file is "MS Excel style csv" you can't validate it conforms. There's a bunch of things the libraries do that cope with at least some of it that you will not replicate with cut or an awk one liner.
- ReleaseCandidat 2y ago
  Yes, as in "check the number of parsed fields for each line" and don't forget about empty fields. Throw an error and stop the program if the number of columns isn't consistent. Which doesn't mean that you can't parse the whole file and output all errors at once (which is the preferred way, we don't live in the 90s any more ;), just don't process the wrong result. And with usable error messages, not just "invalid line N".
aadhavans 2y ago
What if you had constraints on the CSV files? Suppose you knew that they don't contain spaces, for example. In that case, I don't see the problem with using UNIX tools.
- throw10920 2y ago
  Then you're not actually processing the CSV format, you're processing a subset of it. You'll also likely bake that assumption into your system and forget about it, and then potentially violate it later. Well-defined structured data formats, formal grammars, and parsers exist for a reason. Unix explicitly eschews that in favor of the fiction of "plain text", which is not a format for structured data by definition.

by hggigg 2y ago

chasil 2y ago

You might have enjoyed DCL under VMS. It did not immediately succumb to envy of the Korn shell.

Bluecobra 2y ago

> Notably if I see anyone parsing CSVs with cut again I'm going to die inside. Try unpicking a problem where someone put in the name field "Smith, Bob"... How do you tackle this? Would you count the numbers of commas in each line then manually fix the lines that contain more fields?

hggigg 2y ago
You parse them and reject CSVs that do not conform. There is absolutely no way to reason about a malformed CSV.
harry8 2y ago
http://www.catb.org/~esr/writings/taoup/html/ch05s02.html http://www.catb.org/~esr/writings/taoup/html/ch05s02.html paragraph titled DSV style. (Yeah esr, not a fan, whatever...) Csv sucks no matter what, there is no one csv spec. Then even if you assume the file is "MS Excel style csv" you can't validate it conforms. There's a bunch of things the libraries do that cope with at least some of it that you will not replicate with cut or an awk one liner.
ReleaseCandidat 2y ago
Yes, as in "check the number of parsed fields for each line" and don't forget about empty fields. Throw an error and stop the program if the number of columns isn't consistent. Which doesn't mean that you can't parse the whole file and output all errors at once (which is the preferred way, we don't live in the 90s any more ;), just don't process the wrong result. And with usable error messages, not just "invalid line N".

aadhavans 2y ago

What if you had constraints on the CSV files? Suppose you knew that they don't contain spaces, for example. In that case, I don't see the problem with using UNIX tools.

throw10920 2y ago
Then you're not actually processing the CSV format, you're processing a subset of it. You'll also likely bake that assumption into your system and forget about it, and then potentially violate it later. Well-defined structured data formats, formal grammars, and parsers exist for a reason. Unix explicitly eschews that in favor of the fiction of "plain text", which is not a format for structured data by definition.