3 ms·

Thank you for taking the time to reply. FWIW my comment was added a few minutes before your edits so it only contained the [1] footnote and possibly other edits

Thank you for taking the time to reply. FWIW my comment was added a few minutes before your edits so it only contained the [1] footnote and possibly other edits. I guess I was missing the "why" where your post focused on the "what". Of course when you are working with sensitive data like that it makes a lot of sense to be really careful about security. I am well aware about the problems with the csv format. But a lot of times you can not really expect clients to be able to download and decode a parquet file, when all they want is to submit a password, download the data and look at it in excel.

dspillett 2y ago
> download the data and look at it in excel Oh, that is a whole extra rant! As much as I actually like Excel as a tool for numerous tasks, it gets things annoyingly wrong too. We've had clients who manually merge or otherwise tinker with data in Excel to feed to us¹ which can cause numerous problems if leading zeros get cut off identifiers that aren't actually numbers but get interpreted as such, and the age-old joke about Excel being a bit of an inv-cel that mistakes things for dates is still very relevant. Also it requires (as do a number of MS tools) UTF8 files to have the initial BOM, which is actually not recommended as per the relevant standards², or it will assume Win1252 with all the text corruption that implies if your data contains accented characters, currency symbols, or anything else not in the 7-bit ASCII character set. Sometimes we see data where this has happened at some point to some of it, but not all, so the file has a mix of current UTF8 and corrupted UTF8 then encoded as UTF8. Funfunfun. ---- [1] in one case because their requirements changed, and they didn't want to pay anyone (us, the company responsible for the system the data was output from, or their own internal IT/processing/other teams) to automate the new data manipulation… [2] https://en.wikipedia.org/wiki/Byte_order_mark#cite_note-5:~:text=%22The%20Unicode%20Standard%205.0%2C%20Chapter%202%3AGeneral%20Structure%22%20(PDF).%20p.%C2%A036.%20Retrieved%2030%20November%202008.%20Use%20of%20a%20BOM%20is%20neither%20required%20nor%20recommended%20for%20UTF%2D8 https://en.wikipedia.org/wiki/Byte_order_mark#cite_note-5:~:...
- fifilura 2y ago
  Yeah. I rarely work with Excel. When we moved to google sheets, most of the problems you listed mysteriously disappeared :)

dspillett 2y ago

> download the data and look at it in excel Oh, that is a whole extra rant! As much as I actually like Excel as a tool for numerous tasks, it gets things annoyingly wrong too. We've had clients who manually merge or otherwise tinker with data in Excel to feed to us¹ which can cause numerous problems if leading zeros get cut off identifiers that aren't actually numbers but get interpreted as such, and the age-old joke about Excel being a bit of an inv-cel that mistakes things for dates is still very relevant. Also it requires (as do a number of MS tools) UTF8 files to have the initial BOM, which is actually not recommended as per the relevant standards², or it will assume Win1252 with all the text corruption that implies if your data contains accented characters, currency symbols, or anything else not in the 7-bit ASCII character set. Sometimes we see data where this has happened at some point to some of it, but not all, so the file has a mix of current UTF8 and corrupted UTF8 then encoded as UTF8. Funfunfun. ---- [1] in one case because their requirements changed, and they didn't want to pay anyone (us, the company responsible for the system the data was output from, or their own internal IT/processing/other teams) to automate the new data manipulation… [2] https://en.wikipedia.org/wiki/Byte_order_mark#cite_note-5:~:text=%22The%20Unicode%20Standard%205.0%2C%20Chapter%202%3AGeneral%20Structure%22%20(PDF).%20p.%C2%A036.%20Retrieved%2030%20November%202008.%20Use%20of%20a%20BOM%20is%20neither%20required%20nor%20recommended%20for%20UTF%2D8 https://en.wikipedia.org/wiki/Byte_order_mark#cite_note-5:~:...

fifilura 2y ago
Yeah. I rarely work with Excel. When we moved to google sheets, most of the problems you listed mysteriously disappeared :)