3 ms·
This is unacceptable in large business settings today. A list of assets alone can be tens of millions of lines. A list of transactions? Oh boy time to break
by purple-again 9y ago
This is unacceptable in large business settings today. A list of assets alone can be tens of millions of lines. A list of transactions? Oh boy time to break out Rust if you want your database to be finished this month.
- masklinn 9y ago> This is unacceptable in large business settings today. A list of assets alone can be tens of millions of lines. A list of transactions? The original comment specifically mentioned excel. Excel can not work on "tens of millions of lines" and your objection is thus irrelevant. If your data does not come from excel, there is obviously no reason to make it go through excel. There is also no reason to make it go through CSV, make it go through an actual serialisation format which is not mostly unspecified garbage since you're programmatically serialising it for interchange and hopefully not expecting human edition of the intermediate megabytes of data dump.
- maxerickson 9y agoDo those things start out as an Excel file? Because that was the suggestion, to avoid Excel->CSV headaches by importing Excel files directly. I don't think they meant to always use Excel for interchange.
- ryanbrunner 9y agoGenerally speaking though, the areas where the sorts of issues that the article talks about come up is from user-specified CSV mostly loaded from Excel. When you're dealing with enormous amounts of data coming from another (automated) system, you can often make assumptions that wouldn't be practical with user-supplied CSV, like assuming a particular encoding, delimiter, etc.