3 ms·
My one trick there is to just not ask for CSV, and import Excel files directly. Most languages have libraries to read basic (non-formula) excel files, they work
by masklinn 9y ago
My one trick there is to just not ask for CSV, and import Excel files directly. Most languages have libraries to read basic (non-formula) excel files, they work fine, and provide richer data models than CSV (though somewhat risky as you have to deal with formattings and finding e.g. numbers where you expected strings and the other way around).
- purple-again 9y agoThis is unacceptable in large business settings today. A list of assets alone can be tens of millions of lines. A list of transactions? Oh boy time to break out Rust if you want your database to be finished this month.
- masklinn 9y ago> This is unacceptable in large business settings today. A list of assets alone can be tens of millions of lines. A list of transactions? The original comment specifically mentioned excel. Excel can not work on "tens of millions of lines" and your objection is thus irrelevant. If your data does not come from excel, there is obviously no reason to make it go through excel. There is also no reason to make it go through CSV, make it go through an actual serialisation format which is not mostly unspecified garbage since you're programmatically serialising it for interchange and hopefully not expecting human edition of the intermediate megabytes of data dump.
- maxerickson 9y agoDo those things start out as an Excel file? Because that was the suggestion, to avoid Excel->CSV headaches by importing Excel files directly. I don't think they meant to always use Excel for interchange.
- ryanbrunner 9y agoGenerally speaking though, the areas where the sorts of issues that the article talks about come up is from user-specified CSV mostly loaded from Excel. When you're dealing with enormous amounts of data coming from another (automated) system, you can often make assumptions that wouldn't be practical with user-supplied CSV, like assuming a particular encoding, delimiter, etc.