4 ms·

Would it be possible for you to take a step back and describe the problem you are infuriated by again?

dspillett 2y ago
Sitting on the phone with people who's job it is to know these things for their organisation, having to explain to them that no, they shouldn't send me the private key, not even (especially not even) if they send the passphrase too, and many other problems. There seems to be a dearth of people for the ops jobs that need doing, hence OP's “ops are an almost extinct species”, so the tasks are thrown around and have to be done by the first person who doesn't duck fast enough. Those of us on the other side of the company interaction boundary end up having to help them with what should be the basics. I'm not their infrastructure team, I don't have access to be able to answer some of the questions they ask about their own infrastructure. I shouldn't even be client facing never mind teaching them how to do their jobs. The infuriating part (perhaps “frightening” is a better word here) is that there are people managing the secure transmission of data (sometimes sensitive personal data) who do not understand what is going on.
- Corrado 2y ago
  Tell me about it. I'm in the healthcare industry and hospital systems are way too complex and/or they don't have the proper people to manage them. Getting them to understand SFTP is hard enough, let alone public key encryption. Additionally, their systems are often so old and out of date that they don't support anything newer than SFTP. On top of that the regulations they have to follow are just overwhelming and completely out of date. Honestly, I kinda feel sorry for the poor DevOps people in hospitals. It's got to be torture.
- fifilura 2y ago
  Thank you for taking the time to reply. FWIW my comment was added a few minutes before your edits so it only contained the [1] footnote and possibly other edits. I guess I was missing the "why" where your post focused on the "what". Of course when you are working with sensitive data like that it makes a lot of sense to be really careful about security. I am well aware about the problems with the csv format. But a lot of times you can not really expect clients to be able to download and decode a parquet file, when all they want is to submit a password, download the data and look at it in excel.
  - dspillett 2y ago
    > download the data and look at it in excel Oh, that is a whole extra rant! As much as I actually like Excel as a tool for numerous tasks, it gets things annoyingly wrong too. We've had clients who manually merge or otherwise tinker with data in Excel to feed to us¹ which can cause numerous problems if leading zeros get cut off identifiers that aren't actually numbers but get interpreted as such, and the age-old joke about Excel being a bit of an inv-cel that mistakes things for dates is still very relevant. Also it requires (as do a number of MS tools) UTF8 files to have the initial BOM, which is actually not recommended as per the relevant standards², or it will assume Win1252 with all the text corruption that implies if your data contains accented characters, currency symbols, or anything else not in the 7-bit ASCII character set. Sometimes we see data where this has happened at some point to some of it, but not all, so the file has a mix of current UTF8 and corrupted UTF8 then encoded as UTF8. Funfunfun. ---- [1] in one case because their requirements changed, and they didn't want to pay anyone (us, the company responsible for the system the data was output from, or their own internal IT/processing/other teams) to automate the new data manipulation… [2] https://en.wikipedia.org/wiki/Byte_order_mark#cite_note-5:~:text=%22The%20Unicode%20Standard%205.0%2C%20Chapter%202%3AGeneral%20Structure%22%20(PDF).%20p.%C2%A036.%20Retrieved%2030%20November%202008.%20Use%20of%20a%20BOM%20is%20neither%20required%20nor%20recommended%20for%20UTF%2D8 https://en.wikipedia.org/wiki/Byte_order_mark#cite_note-5:~:...
    - fifilura 2y ago
      Yeah. I rarely work with Excel. When we moved to google sheets, most of the problems you listed mysteriously disappeared :)

Sitting on the phone with people who's job it is to know these things for their organisation, having to explain to them that no, they shouldn't send me the private key, not even (especially not even) if they send the passphrase too, and many other problems. There seems to be a dearth of people for the ops jobs that need doing, hence OP's “ops are an almost extinct species”, so the tasks are thrown around and have to be done by the first person who doesn't duck fast enough. Those of us on the other side of the company interaction boundary end up having to help them with what should be the basics. I'm not their infrastructure team, I don't have access to be able to answer some of the questions they ask about their own infrastructure. I shouldn't even be client facing never mind teaching them how to do their jobs. The infuriating part (perhaps “frightening” is a better word here) is that there are people managing the secure transmission of data (sometimes sensitive personal data) who do not understand what is going on.

Corrado 2y ago
Tell me about it. I'm in the healthcare industry and hospital systems are way too complex and/or they don't have the proper people to manage them. Getting them to understand SFTP is hard enough, let alone public key encryption. Additionally, their systems are often so old and out of date that they don't support anything newer than SFTP. On top of that the regulations they have to follow are just overwhelming and completely out of date. Honestly, I kinda feel sorry for the poor DevOps people in hospitals. It's got to be torture.
fifilura 2y ago
Thank you for taking the time to reply. FWIW my comment was added a few minutes before your edits so it only contained the [1] footnote and possibly other edits. I guess I was missing the "why" where your post focused on the "what". Of course when you are working with sensitive data like that it makes a lot of sense to be really careful about security. I am well aware about the problems with the csv format. But a lot of times you can not really expect clients to be able to download and decode a parquet file, when all they want is to submit a password, download the data and look at it in excel.
- dspillett 2y ago
  > download the data and look at it in excel Oh, that is a whole extra rant! As much as I actually like Excel as a tool for numerous tasks, it gets things annoyingly wrong too. We've had clients who manually merge or otherwise tinker with data in Excel to feed to us¹ which can cause numerous problems if leading zeros get cut off identifiers that aren't actually numbers but get interpreted as such, and the age-old joke about Excel being a bit of an inv-cel that mistakes things for dates is still very relevant. Also it requires (as do a number of MS tools) UTF8 files to have the initial BOM, which is actually not recommended as per the relevant standards², or it will assume Win1252 with all the text corruption that implies if your data contains accented characters, currency symbols, or anything else not in the 7-bit ASCII character set. Sometimes we see data where this has happened at some point to some of it, but not all, so the file has a mix of current UTF8 and corrupted UTF8 then encoded as UTF8. Funfunfun. ---- [1] in one case because their requirements changed, and they didn't want to pay anyone (us, the company responsible for the system the data was output from, or their own internal IT/processing/other teams) to automate the new data manipulation… [2] https://en.wikipedia.org/wiki/Byte_order_mark#cite_note-5:~:text=%22The%20Unicode%20Standard%205.0%2C%20Chapter%202%3AGeneral%20Structure%22%20(PDF).%20p.%C2%A036.%20Retrieved%2030%20November%202008.%20Use%20of%20a%20BOM%20is%20neither%20required%20nor%20recommended%20for%20UTF%2D8 https://en.wikipedia.org/wiki/Byte_order_mark#cite_note-5:~:...
  - fifilura 2y ago
    Yeah. I rarely work with Excel. When we moved to google sheets, most of the problems you listed mysteriously disappeared :)