> This scary stuff only applies if you need to read garbage CSVs from a million different sources. Even then, like ken says here, that's only like 10 issues to

> This scary stuff only applies if you need to read garbage CSVs from a million different sources. Even then, like ken says here, that's only like 10 issues to figure out. Well, garbage CSV's are garbage CSV's. They can be inconsistent, and if you're reading millions of them (OK, even hundreds) it can be a real pain-in-the-ass data janitorial task-- the equivalent of cleaning a public bathroom after homeless people have sprayed diarrhea all over the floors, walls, and ceiling. The hard part of dealing with the "10 issues" is you have to figure out which ones apply to which files (or even lines). A CSV linter is often helpful to diagnose, that would be the janitorial equivalent of rubber gloves and a bucket of soap, but it's still a lot of work. If you truly have millions of inconsistent files to deal with, the only way to keep sane is to categorize them by some parameters that make sense... size, date, keywords, even word-stats. Then, you can programmatically tackle each category. It's a messy problem no matter what and gets more ugly as the scale increases. And all of this is often before you even start what you actually need to do.