3 ms·

>> […] Excel […] > Don't use broken tools. Tell that to your accounting and finance department and let us know how the message is received. > If you're force

>> […] Excel […] > Don't use broken tools. Tell that to your accounting and finance department and let us know how the message is received. > If you're forced to use Excel, then it's not really you in control, is it? In which case the up-thread's advice to "just produce strict RFC 4180-compliant CSV data" is worthless. "Just." We're stuck with whatever CSVs we get, so 'just' doing X is not an option.

SammyStacks 2y ago
From a pragmatic viewpoint, the CSVs that I get from finance (usually saved as .xlsx) have the same issues for parsing the data as a CSV. But since the issues are consistent, I can automate conversion from .xlsx to CSV, then process the CSV using awk to eliminate errors in further parsing the CSV (for import, analysis, etc.). Sure, I'm essentially parsing the CSV twice but, because the parsing issues are consistent, I can automate to make the process efficient. Obviously that wouldn't work for CSVs with different structures, but can be effective in the workplace in certain scenarios.
- disgruntledphd2 2y ago
  As long as a human didn't generate the file, all things can be automated. However, if you ever have the misfortune of dealing with human generated files (particularly Excels) then you will suffer much pain and loss. I once had to deal with a "CSV" which had not one, not two but 6(!) distinct date formats in the same file. Life as a data scientist kinda sucks sometimes :shrug:.
  - hyperman1 2y ago
    Before 2010 and UTF-8 everywhere , I regularly had the misfortune of dealing with multi encoding CSVs. Someone got CSVs from multiple sources and catted them together. One source uses ISO 8859-1, another -15, another UTF-8, sometimes a greek or russian or even ebcdic was in there. Fun trying to guess where one stopped and the other begun . Of course, none of them were consistent crlf or escape wise.
bathtub365 2y ago
The up-thread’s comment (emphasis mine): > And *if you are in control of producing data*, just produce strict RFC 4180-compliant CSV data The point of the comment was that you likely aren’t in control of producing the data, so the article’s recommendation of using an entirely different format is likely also invalid. I’m not sure what you are arguing against as you seem to actually agree with them.
- deleted 2y ago
  [deleted]

SammyStacks 2y ago

From a pragmatic viewpoint, the CSVs that I get from finance (usually saved as .xlsx) have the same issues for parsing the data as a CSV. But since the issues are consistent, I can automate conversion from .xlsx to CSV, then process the CSV using awk to eliminate errors in further parsing the CSV (for import, analysis, etc.). Sure, I'm essentially parsing the CSV twice but, because the parsing issues are consistent, I can automate to make the process efficient. Obviously that wouldn't work for CSVs with different structures, but can be effective in the workplace in certain scenarios.

disgruntledphd2 2y ago
As long as a human didn't generate the file, all things can be automated. However, if you ever have the misfortune of dealing with human generated files (particularly Excels) then you will suffer much pain and loss. I once had to deal with a "CSV" which had not one, not two but 6(!) distinct date formats in the same file. Life as a data scientist kinda sucks sometimes :shrug:.
- hyperman1 2y ago
  Before 2010 and UTF-8 everywhere , I regularly had the misfortune of dealing with multi encoding CSVs. Someone got CSVs from multiple sources and catted them together. One source uses ISO 8859-1, another -15, another UTF-8, sometimes a greek or russian or even ebcdic was in there. Fun trying to guess where one stopped and the other begun . Of course, none of them were consistent crlf or escape wise.

bathtub365 2y ago

The up-thread’s comment (emphasis mine): > And *if you are in control of producing data*, just produce strict RFC 4180-compliant CSV data The point of the comment was that you likely aren’t in control of producing the data, so the article’s recommendation of using an entirely different format is likely also invalid. I’m not sure what you are arguing against as you seem to actually agree with them.

deleted 2y ago
[deleted]