3 ms·

Don't use broken tools. The key phrase is "in control of producing data". If you're forced to use Excel, then it's not really you in control, is it?

by atrus 2y ago

throw0101d 2y ago
>> […] Excel […] > Don't use broken tools. Tell that to your accounting and finance department and let us know how the message is received. > If you're forced to use Excel, then it's not really you in control, is it? In which case the up-thread's advice to "just produce strict RFC 4180-compliant CSV data" is worthless. "Just." We're stuck with whatever CSVs we get, so 'just' doing X is not an option.
- SammyStacks 2y ago
  From a pragmatic viewpoint, the CSVs that I get from finance (usually saved as .xlsx) have the same issues for parsing the data as a CSV. But since the issues are consistent, I can automate conversion from .xlsx to CSV, then process the CSV using awk to eliminate errors in further parsing the CSV (for import, analysis, etc.). Sure, I'm essentially parsing the CSV twice but, because the parsing issues are consistent, I can automate to make the process efficient. Obviously that wouldn't work for CSVs with different structures, but can be effective in the workplace in certain scenarios.
  - disgruntledphd2 2y ago
    As long as a human didn't generate the file, all things can be automated. However, if you ever have the misfortune of dealing with human generated files (particularly Excels) then you will suffer much pain and loss. I once had to deal with a "CSV" which had not one, not two but 6(!) distinct date formats in the same file. Life as a data scientist kinda sucks sometimes :shrug:.
    - hyperman1 2y ago
      Before 2010 and UTF-8 everywhere , I regularly had the misfortune of dealing with multi encoding CSVs. Someone got CSVs from multiple sources and catted them together. One source uses ISO 8859-1, another -15, another UTF-8, sometimes a greek or russian or even ebcdic was in there. Fun trying to guess where one stopped and the other begun . Of course, none of them were consistent crlf or escape wise.
- bathtub365 2y ago
  The up-thread’s comment (emphasis mine): > And *if you are in control of producing data*, just produce strict RFC 4180-compliant CSV data The point of the comment was that you likely aren’t in control of producing the data, so the article’s recommendation of using an entirely different format is likely also invalid. I’m not sure what you are arguing against as you seem to actually agree with them.
  - deleted 2y ago
    [deleted]
solarkraft 2y ago
“Just be in control” is unfortunately bad advice when you’re … not in control.
- TheCleric 2y ago
  I don't think the advice is "Just be in control" as much as "Acknowledge you're not in control most of the time."
Aniket-N 2y ago
This is some next level response where Excel is called a “broken tool”. You may not agree with their choices or design or anything else. But calling the entirety of the product isn’t making a strong case towards the prior point.

Don't use broken tools. The key phrase is "in control of producing data". If you're forced to use Excel, then it's not really you in control, is it?

by atrus 2y ago

Don't use broken tools. The key phrase is "in control of producing data". If you're forced to use Excel, then it's not really you in control, is it?

throw0101d 2y ago

>> […] Excel […] > Don't use broken tools. Tell that to your accounting and finance department and let us know how the message is received. > If you're forced to use Excel, then it's not really you in control, is it? In which case the up-thread's advice to "just produce strict RFC 4180-compliant CSV data" is worthless. "Just." We're stuck with whatever CSVs we get, so 'just' doing X is not an option.

SammyStacks 2y ago
From a pragmatic viewpoint, the CSVs that I get from finance (usually saved as .xlsx) have the same issues for parsing the data as a CSV. But since the issues are consistent, I can automate conversion from .xlsx to CSV, then process the CSV using awk to eliminate errors in further parsing the CSV (for import, analysis, etc.). Sure, I'm essentially parsing the CSV twice but, because the parsing issues are consistent, I can automate to make the process efficient. Obviously that wouldn't work for CSVs with different structures, but can be effective in the workplace in certain scenarios.
- disgruntledphd2 2y ago
  As long as a human didn't generate the file, all things can be automated. However, if you ever have the misfortune of dealing with human generated files (particularly Excels) then you will suffer much pain and loss. I once had to deal with a "CSV" which had not one, not two but 6(!) distinct date formats in the same file. Life as a data scientist kinda sucks sometimes :shrug:.
  - hyperman1 2y ago
    Before 2010 and UTF-8 everywhere , I regularly had the misfortune of dealing with multi encoding CSVs. Someone got CSVs from multiple sources and catted them together. One source uses ISO 8859-1, another -15, another UTF-8, sometimes a greek or russian or even ebcdic was in there. Fun trying to guess where one stopped and the other begun . Of course, none of them were consistent crlf or escape wise.
bathtub365 2y ago
The up-thread’s comment (emphasis mine): > And *if you are in control of producing data*, just produce strict RFC 4180-compliant CSV data The point of the comment was that you likely aren’t in control of producing the data, so the article’s recommendation of using an entirely different format is likely also invalid. I’m not sure what you are arguing against as you seem to actually agree with them.
- deleted 2y ago
  [deleted]

solarkraft 2y ago

“Just be in control” is unfortunately bad advice when you’re … not in control.

TheCleric 2y ago
I don't think the advice is "Just be in control" as much as "Acknowledge you're not in control most of the time."

Aniket-N 2y ago

This is some next level response where Excel is called a “broken tool”. You may not agree with their choices or design or anything else. But calling the entirety of the product isn’t making a strong case towards the prior point.