5 ms·

I wrote my own CSV code. That's only about 10 issues, and I understand them all, and they're easy to handle, and to test (unlike, say, threading issues, or OS

by ken 6y ago

I wrote my own CSV code. That's only about 10 issues, and I understand them all, and they're easy to handle, and to test (unlike, say, threading issues, or OS bugs that don't happen on my computer). That wasn't even close to being one of the more difficult parts of my program to write. I've had to do zero work to maintain it. I don't seem to have ever had any bugs filed against it. Are these supposed to be difficult? Sure, don't write your own encryption algorithm, and avoid touching threads if at all possible, but CSV? Sure, I wish I could have used a library. Unfortunately, once you start adding requirements, the options start to vanish. It needs to be a C (or C-callable) library. It needs to accept input as Unicode strings (lines), not just files on disk, or bytes. It needs to return results line-at-a-time. It needs to have a compatible license. It needs to support the same package manager as I'm using (or be a trivial to include, like a single file). It needs to be reliable (essentially bug-free, or at the very least currently maintained). It needs to be fairly efficient. What library supports all these? > Writing CSV code that works with files out there in the real world is a difficult task. The rabbit hole goes deep. Ruby CSV library is 2321 lines. I love Ruby, but that file looks a little nuts to me right now. The block comment at the top of csv.rb is over 500 lines long. The Ruby module tries to do everything. Just "def open(filename, mode="r", options)" and its docstring are 100 lines long! This class includes methods for auto-detecting the encoding, and auto-detecting common date formats, and some kind of system "for building Unix-like filters for CSV data". I don't put all of that into my CSV parser, because I want my code to be modular. I have features like data detectors and filters and open-file-by-filename in my program, too, but they're fully independent, not tied to any one file format. When I come up with a better way to detect dates in text, I don't want to tweak a 6-line regex in every file format I've written.

greggman3 6y ago
> I don't put all of that into my CSV parser, The art of not cluttering a library with too much stuff is a hard lesson to acquire. Most library authors try to throw in the kitchen sink. It's insane. I'm sure I was guilty of it in the past. For parser take a string. let me do the I/O outside. If you want to offer streams then just offer an abstract interface stream and then a few examples of implementing a stream but don't include them in your library. It's just clutter. This is one if largest problems with NPM. So many libraries try to do too much. Some come with a command line utility "just because"!??!?! It then adds more surface area, more dependencies, more churn as we have to update all those un-needed dependencies.
- mikepurvis 6y ago
  Isn't NPM also notorious for ultra minimalist libraries like leftpad as well, though?
  - tehbeard 6y ago
    Leftpad is more a symptom of JS lacking a real standard library than anything else.
  - jsf01 6y ago
    Yes. One of the costs of having a low barrier to entry for publishing is that the quality is generally lower. NPM still hosts tons of very well written code, but you have to sift through quite a lot of noise to find it.
mey 6y ago
I wish the JVM was less of a pariah. I love being able to just reach for https://commons.apache.org/proper/commons-csv/user-guide.html https://commons.apache.org/proper/commons-csv/user-guide.htm... Here is the entire logic of the CSV parser (test code is in a separate path) https://gitbox.apache.org/repos/asf?p=commons-csv.git;a=tree;f=src/main/java/org/apache/commons/csv;h=83af61a0d67b70f708125b0ca9d41b09a9396178;hb=refs/heads/master https://gitbox.apache.org/repos/asf?p=commons-csv.git;a=tree...
- the_af 6y ago
  In what sense is the JVM a pariah? It's one of the most successful software platforms.
  - mey 6y ago
    - Oracle's licensing confusion (where does Java and Java(TM) begin/end). - The JVM historically has included the kitchen sink (Java 9 started addressing this). - People's impressions that it is slow or a memory hog. Edit: (Looking at you Eclipse/Intellij/Glassfish/JBoss/Atlassian) - Experiences with bad code bases due to it's ubiquity and age. Old code bases tend to evolve in interesting ways. - It's not the new shiny thing. I have meet plenty of developers who dismiss Java out of hand. I've also seen DBA's dismiss FKs. I don't think how good/bad it is has anything to do with people's dislike of it, more that there is a certain vocal group that dislike it for being the "enterprise" solution.
    - the_af 6y ago
      But this does not correlate to how used the JVM actually is in the outside world. The JVM is huge. It's not a pariah. The perceptions of JVM slowness have been debunked again and again, and anyone who cares knows they are mostly a red herring and have been so for years. Java and the JVM are mature, battle-tested, widely used technologies in the industry, easy to hire for and with multitude of libraries for almost whatever you want to do. It may not be the latest shiny new thing, but that's different. Pariah means an outcast. Neither Java nor the JVM are outcasts.
- 6y ago
rmrfrmrf 6y ago
having been bit by both reinventing the wheel and overly-relying on third party code, i find that my position on this varies from week to week.
- chipotle_coyote 6y ago
  As I find myself writing my own PHP quasi-framework from scratch in 2020, I am definitely feeling this. On the one hand, reinventing a wheel like this is utterly insane. On the other hand, for the simple project I'm doing this for, nearly all the other wheels I've found turn out to be monster truck tires, milspec tank treads, or waterskis.
rovolo 6y ago
1) You wrote a general CSV input parser, but what about someone writing CSV output? The obvious way (write a ',' between each cell and a '\n' between each row) works most of the time, but it doesn't always work. The output needs to be escaped correctly, and data escaping should be well tested. I think the issue is that CSV sounds very easy to implement, but most naive implementations will be wrong. 2) You took on exactly the responsibilities the article author said you'd have to take on. Based on the screenshots for your "Strukt" application, it looks like the user controls configuration/validation for parsing CSV and they will fix any issues with parsing the CSV file themselves. > If a supplied CSV is arbitrary, the only real way to make sure the data is correct is for an user to check it and eventually specify the delimiter, quoting rule,… Barring that you may end up with a error or worse silently corrupted data. 3) I think the better question is, how do you handle malformed CSV? Is your error correction as robust as Ruby's? Do you handle malformed CSV the same way as other libraries? If it's handled differently by different parsers, you can run into issues like Apple recently did [0] [0] https://siguza.github.io/psychicpaper/ https://siguza.github.io/psychicpaper/
- ken 6y ago
  > You took on exactly the responsibilities the article author said you'd have to take on. Yep, and having been there, I’m reporting: it’s not nearly as bad as this post makes it sound. I’ve written web apps, too, and (for example) CSS is 100 times worse. Nobody is suggesting every website should use only browser default styles, though. > I think the better question is, how do you handle malformed CSV? What does “malformed” mean? The good/bad thing about CSV is that virtually every text file is valid! The only malformation I can think of is an open quote with no matching close quote (so the entire rest of the document is one value). My implementation is streaming, so there’s no great way to flag it: any future data could have the other quote! We show the data as it’s parsed, so it should be obvious to the user what is going on, and where. There are areas of computing that are too complex. I’m usually the first to complain about such things. I really don’t think CSV parsing is one of them.
  - btilly 6y ago
    "What does “malformed” mean? The good/bad thing about CSV is that virtually every text file is valid! The only malformation I can think of is an open quote with no matching close quote (so the entire rest of the document is one value). My implementation is streaming, so there’s no great way to flag it: any future data could have the other quote!* An example of improper csv code is this: hello,world,"And this, contains "bad quotes"...",1,2,3 Lest you think this is made up, I ran across this when someone cut and paste Excel into a text field. I also have seen batch processing of user files break hard when a quote issue like this caused a hand-rolled CSV parser to conclude that half the file was a single very long field.
geraldbauer 6y ago
> I love Ruby, but that file looks a little nuts to me right now. The block comment at the top of csv.rb is over 500 lines long. The Ruby module tries to do everything... FYI: CsvReader [1] is a more modern alternative to the old standard csv.rb library in ruby. [1]: https://github.com/csvreader/csvreader https://github.com/csvreader/csvreader
platz 6y ago
There is a big difference between writing an application and being a library author.
burntsushi 6y ago
> Sure, I wish I could have used a library. Unfortunately, once you start adding requirements, the options start to vanish. It needs to be a C (or C-callable) library. It needs to accept input as Unicode strings (lines), not just files on disk, or bytes. It needs to return results line-at-a-time. It needs to have a compatible license. It needs to support the same package manager as I'm using (or be a trivial to include, like a single file). It needs to be reliable (essentially bug-free, or at the very least currently maintained). It needs to be fairly efficient. What library supports all these? I always thought people just used libcsv in cases like this. If that's missing something though, then I believe my Rust csv library[1] would be good enough if I added a C API, which would be pretty easy and I would be happy to do it if it really filled a niche. I think the only hurdle at that point would be whether you could tolerate a Rust compiler in your build dependencies. [1] - https://docs.rs/csv-core/0.1.10/csv_core/ https://docs.rs/csv-core/0.1.10/csv_core/
heavenlyblue 6y ago
I am betting that Ruby library still can’t handle a good 50% of the files that people throw at it because there’s not such thing as CSV standard. On the other hand writing a custom parser using some off-the-shelf parser combinator library is easy and straightforward (ie I know how much time it will take to implement one and what exactly it does). Sometimes having multiple implementations of the same is better than a single generalised implementation struggling from combinatorial explosion.
goto11 6y ago
True, is not rocket science to write a CSV parser. The problem with CSV is it looks so deceptively simple so people write simplistic ad-hoc parsers/exporters all the time, so we end up with a bunch of incompatibilities.

by ken 6y ago

greggman3 6y ago

> I don't put all of that into my CSV parser, The art of not cluttering a library with too much stuff is a hard lesson to acquire. Most library authors try to throw in the kitchen sink. It's insane. I'm sure I was guilty of it in the past. For parser take a string. let me do the I/O outside. If you want to offer streams then just offer an abstract interface stream and then a few examples of implementing a stream but don't include them in your library. It's just clutter. This is one if largest problems with NPM. So many libraries try to do too much. Some come with a command line utility "just because"!??!?! It then adds more surface area, more dependencies, more churn as we have to update all those un-needed dependencies.

mikepurvis 6y ago
Isn't NPM also notorious for ultra minimalist libraries like leftpad as well, though?
- tehbeard 6y ago
  Leftpad is more a symptom of JS lacking a real standard library than anything else.
- jsf01 6y ago
  Yes. One of the costs of having a low barrier to entry for publishing is that the quality is generally lower. NPM still hosts tons of very well written code, but you have to sift through quite a lot of noise to find it.

mey 6y ago

I wish the JVM was less of a pariah. I love being able to just reach for https://commons.apache.org/proper/commons-csv/user-guide.html https://commons.apache.org/proper/commons-csv/user-guide.htm... Here is the entire logic of the CSV parser (test code is in a separate path) https://gitbox.apache.org/repos/asf?p=commons-csv.git;a=tree;f=src/main/java/org/apache/commons/csv;h=83af61a0d67b70f708125b0ca9d41b09a9396178;hb=refs/heads/master https://gitbox.apache.org/repos/asf?p=commons-csv.git;a=tree...

the_af 6y ago
In what sense is the JVM a pariah? It's one of the most successful software platforms.
- mey 6y ago
  - Oracle's licensing confusion (where does Java and Java(TM) begin/end). - The JVM historically has included the kitchen sink (Java 9 started addressing this). - People's impressions that it is slow or a memory hog. Edit: (Looking at you Eclipse/Intellij/Glassfish/JBoss/Atlassian) - Experiences with bad code bases due to it's ubiquity and age. Old code bases tend to evolve in interesting ways. - It's not the new shiny thing. I have meet plenty of developers who dismiss Java out of hand. I've also seen DBA's dismiss FKs. I don't think how good/bad it is has anything to do with people's dislike of it, more that there is a certain vocal group that dislike it for being the "enterprise" solution.
  - the_af 6y ago
    But this does not correlate to how used the JVM actually is in the outside world. The JVM is huge. It's not a pariah. The perceptions of JVM slowness have been debunked again and again, and anyone who cares knows they are mostly a red herring and have been so for years. Java and the JVM are mature, battle-tested, widely used technologies in the industry, easy to hire for and with multitude of libraries for almost whatever you want to do. It may not be the latest shiny new thing, but that's different. Pariah means an outcast. Neither Java nor the JVM are outcasts.
6y ago

rmrfrmrf 6y ago

having been bit by both reinventing the wheel and overly-relying on third party code, i find that my position on this varies from week to week.

chipotle_coyote 6y ago
As I find myself writing my own PHP quasi-framework from scratch in 2020, I am definitely feeling this. On the one hand, reinventing a wheel like this is utterly insane. On the other hand, for the simple project I'm doing this for, nearly all the other wheels I've found turn out to be monster truck tires, milspec tank treads, or waterskis.

rovolo 6y ago

1) You wrote a general CSV input parser, but what about someone writing CSV output? The obvious way (write a ',' between each cell and a '\n' between each row) works most of the time, but it doesn't always work. The output needs to be escaped correctly, and data escaping should be well tested. I think the issue is that CSV sounds very easy to implement, but most naive implementations will be wrong. 2) You took on exactly the responsibilities the article author said you'd have to take on. Based on the screenshots for your "Strukt" application, it looks like the user controls configuration/validation for parsing CSV and they will fix any issues with parsing the CSV file themselves. > If a supplied CSV is arbitrary, the only real way to make sure the data is correct is for an user to check it and eventually specify the delimiter, quoting rule,… Barring that you may end up with a error or worse silently corrupted data. 3) I think the better question is, how do you handle malformed CSV? Is your error correction as robust as Ruby's? Do you handle malformed CSV the same way as other libraries? If it's handled differently by different parsers, you can run into issues like Apple recently did [0] [0] https://siguza.github.io/psychicpaper/ https://siguza.github.io/psychicpaper/

ken 6y ago
> You took on exactly the responsibilities the article author said you'd have to take on. Yep, and having been there, I’m reporting: it’s not nearly as bad as this post makes it sound. I’ve written web apps, too, and (for example) CSS is 100 times worse. Nobody is suggesting every website should use only browser default styles, though. > I think the better question is, how do you handle malformed CSV? What does “malformed” mean? The good/bad thing about CSV is that virtually every text file is valid! The only malformation I can think of is an open quote with no matching close quote (so the entire rest of the document is one value). My implementation is streaming, so there’s no great way to flag it: any future data could have the other quote! We show the data as it’s parsed, so it should be obvious to the user what is going on, and where. There are areas of computing that are too complex. I’m usually the first to complain about such things. I really don’t think CSV parsing is one of them.
- btilly 6y ago
  "What does “malformed” mean? The good/bad thing about CSV is that virtually every text file is valid! The only malformation I can think of is an open quote with no matching close quote (so the entire rest of the document is one value). My implementation is streaming, so there’s no great way to flag it: any future data could have the other quote!* An example of improper csv code is this: hello,world,"And this, contains "bad quotes"...",1,2,3 Lest you think this is made up, I ran across this when someone cut and paste Excel into a text field. I also have seen batch processing of user files break hard when a quote issue like this caused a hand-rolled CSV parser to conclude that half the file was a single very long field.

geraldbauer 6y ago

> I love Ruby, but that file looks a little nuts to me right now. The block comment at the top of csv.rb is over 500 lines long. The Ruby module tries to do everything... FYI: CsvReader [1] is a more modern alternative to the old standard csv.rb library in ruby. [1]: https://github.com/csvreader/csvreader https://github.com/csvreader/csvreader

platz 6y ago

There is a big difference between writing an application and being a library author.

burntsushi 6y ago

> Sure, I wish I could have used a library. Unfortunately, once you start adding requirements, the options start to vanish. It needs to be a C (or C-callable) library. It needs to accept input as Unicode strings (lines), not just files on disk, or bytes. It needs to return results line-at-a-time. It needs to have a compatible license. It needs to support the same package manager as I'm using (or be a trivial to include, like a single file). It needs to be reliable (essentially bug-free, or at the very least currently maintained). It needs to be fairly efficient. What library supports all these? I always thought people just used libcsv in cases like this. If that's missing something though, then I believe my Rust csv library[1] would be good enough if I added a C API, which would be pretty easy and I would be happy to do it if it really filled a niche. I think the only hurdle at that point would be whether you could tolerate a Rust compiler in your build dependencies. [1] - https://docs.rs/csv-core/0.1.10/csv_core/ https://docs.rs/csv-core/0.1.10/csv_core/

heavenlyblue 6y ago

I am betting that Ruby library still can’t handle a good 50% of the files that people throw at it because there’s not such thing as CSV standard. On the other hand writing a custom parser using some off-the-shelf parser combinator library is easy and straightforward (ie I know how much time it will take to implement one and what exactly it does). Sometimes having multiple implementations of the same is better than a single generalised implementation struggling from combinatorial explosion.

goto11 6y ago

True, is not rocket science to write a CSV parser. The problem with CSV is it looks so deceptively simple so people write simplistic ad-hoc parsers/exporters all the time, so we end up with a bunch of incompatibilities.