4 ms·

This scary stuff only applies if you need to read garbage CSVs from a million different sources. Even then, like ken says here, that's only like 10 issues to fi

This scary stuff only applies if you need to read garbage CSVs from a million different sources. Even then, like ken says here, that's only like 10 issues to figure out. Exporting is likely the more common case and is pretty simple. Quote fields if they contain \r\n", I'd say relax, focus on your particular problem - if a simple 2-3 line export loop fits your use case and is simpler than a csv library (e.g. in C/C++ perhaps where dependencies are a pain) then why not. This fear mongering about "you can't possibly do this, the library writers are much smarter than you" can lead to it's own ridiculousness of hundreds of dependencies and ungodly waste of processor time, memory, build time, containerization hassle, etc. to solve what should be a simple problem.

crispyambulance 6y ago
> This scary stuff only applies if you need to read garbage CSVs from a million different sources. Even then, like ken says here, that's only like 10 issues to figure out. Well, garbage CSV's are garbage CSV's. They can be inconsistent, and if you're reading millions of them (OK, even hundreds) it can be a real pain-in-the-ass data janitorial task-- the equivalent of cleaning a public bathroom after homeless people have sprayed diarrhea all over the floors, walls, and ceiling. The hard part of dealing with the "10 issues" is you have to figure out which ones apply to which files (or even lines). A CSV linter is often helpful to diagnose, that would be the janitorial equivalent of rubber gloves and a bucket of soap, but it's still a lot of work. If you truly have millions of inconsistent files to deal with, the only way to keep sane is to categorize them by some parameters that make sense... size, date, keywords, even word-stats. Then, you can programmatically tackle each category. It's a messy problem no matter what and gets more ugly as the scale increases. And all of this is often before you even start what you actually need to do.
zelly 6y ago
Learned helplessness. You will never git gud if you're just copying and pasting libraries. I blame npm.
kristopolous 6y ago
There's a balance. Massaging wacky csv feeds to actually be parsable because someone wrote their own incorrectly is a real problem. Seen it dozens of times. I think the real problem is programming has become too easy. Almost nobody really has to worry about data structures, memory management, any of the hard problems that filter for competency. Some wacky insane stuff runs just fine on these 8-core multi-gigabyte ram computers we literally have in our pockets. A lot of it is on a VM inside of yet another VM, even that's fine. Just about every bad idea seems to work. These kinds of issues are to be expected.
7thaccount 6y ago
I agree. Sometimes the business case is needing to read .CSV files that have some pretty bizarre properties and you need a massive library that takes care of all of that. In my case (I do a LOT of .CSV work) though none of these problems exist. I haven't reached for Python's .CSV library in years and neither have my co-workers. We simply loop through the file, split the strings on commas, and have a few if statements to parse on a situation by situation basis. Extremely old school, but it works very well and is easier than dealing with the objects that are generated from using the library. I realize this probably doesn't work with all use cases on HN though, or even most. My ideal language has direct support for making file I/O, dictionaries, sets, whatever all in core and easy to use. This keeps me from having to write my own helper classes and modules or bring in 3rd party libraries. Other languages like to have a community package for everything to where you have 9 options for everything. This seems to be common in JS and Perl and is certainly valid, but I'm not a big fan. Python hits a sweet spot for me.
- bubba1236 6y ago
  I hope you're joking or this a special kind of stupid
bob1029 6y ago
This is my take as well. Most of our CSV concerns center around generating them, not consuming them. Generating a valid CSV string that Excel wont complain about is approximately 2 lines of LINQ if you have a clean starting point (i.e. a list of class instances with strongly-typed members). For cases where I have to read garbage CSV, I first go to the source and determine if we can't get a better export or interface into the data. Most shit CSV I see is a consequence of someone hacking together a SQL query to produce the output. I'd be inclined to just request a database export at that point. Obviously not always an option, but you should always insist on having the best representation of the business data before you start banging your head against the wall on parsing it.
jastr 6y ago
I have a saas for non-devs to open big CSVs, so code I've written has parsed thousands of garbage CSVs (and failed to parse thousands others). Lots of pitfalls are easy to avoid if you already know them, which it sounds like you do. The author is saying that most engineers aren't aware of these issues, and the author is correct. Writing a for loop is easy, but so is using a language's built in CSV writer.

crispyambulance 6y ago

> This scary stuff only applies if you need to read garbage CSVs from a million different sources. Even then, like ken says here, that's only like 10 issues to figure out. Well, garbage CSV's are garbage CSV's. They can be inconsistent, and if you're reading millions of them (OK, even hundreds) it can be a real pain-in-the-ass data janitorial task-- the equivalent of cleaning a public bathroom after homeless people have sprayed diarrhea all over the floors, walls, and ceiling. The hard part of dealing with the "10 issues" is you have to figure out which ones apply to which files (or even lines). A CSV linter is often helpful to diagnose, that would be the janitorial equivalent of rubber gloves and a bucket of soap, but it's still a lot of work. If you truly have millions of inconsistent files to deal with, the only way to keep sane is to categorize them by some parameters that make sense... size, date, keywords, even word-stats. Then, you can programmatically tackle each category. It's a messy problem no matter what and gets more ugly as the scale increases. And all of this is often before you even start what you actually need to do.

zelly 6y ago

Learned helplessness. You will never git gud if you're just copying and pasting libraries. I blame npm.

kristopolous 6y ago

There's a balance. Massaging wacky csv feeds to actually be parsable because someone wrote their own incorrectly is a real problem. Seen it dozens of times. I think the real problem is programming has become too easy. Almost nobody really has to worry about data structures, memory management, any of the hard problems that filter for competency. Some wacky insane stuff runs just fine on these 8-core multi-gigabyte ram computers we literally have in our pockets. A lot of it is on a VM inside of yet another VM, even that's fine. Just about every bad idea seems to work. These kinds of issues are to be expected.

7thaccount 6y ago

I agree. Sometimes the business case is needing to read .CSV files that have some pretty bizarre properties and you need a massive library that takes care of all of that. In my case (I do a LOT of .CSV work) though none of these problems exist. I haven't reached for Python's .CSV library in years and neither have my co-workers. We simply loop through the file, split the strings on commas, and have a few if statements to parse on a situation by situation basis. Extremely old school, but it works very well and is easier than dealing with the objects that are generated from using the library. I realize this probably doesn't work with all use cases on HN though, or even most. My ideal language has direct support for making file I/O, dictionaries, sets, whatever all in core and easy to use. This keeps me from having to write my own helper classes and modules or bring in 3rd party libraries. Other languages like to have a community package for everything to where you have 9 options for everything. This seems to be common in JS and Perl and is certainly valid, but I'm not a big fan. Python hits a sweet spot for me.

bubba1236 6y ago
I hope you're joking or this a special kind of stupid

bob1029 6y ago

This is my take as well. Most of our CSV concerns center around generating them, not consuming them. Generating a valid CSV string that Excel wont complain about is approximately 2 lines of LINQ if you have a clean starting point (i.e. a list of class instances with strongly-typed members). For cases where I have to read garbage CSV, I first go to the source and determine if we can't get a better export or interface into the data. Most shit CSV I see is a consequence of someone hacking together a SQL query to produce the output. I'd be inclined to just request a database export at that point. Obviously not always an option, but you should always insist on having the best representation of the business data before you start banging your head against the wall on parsing it.

jastr 6y ago

I have a saas for non-devs to open big CSVs, so code I've written has parsed thousands of garbage CSVs (and failed to parse thousands others). Lots of pitfalls are easy to avoid if you already know them, which it sounds like you do. The author is saying that most engineers aren't aware of these issues, and the author is correct. Writing a for loop is easy, but so is using a language's built in CSV writer.