4 ms·

> ops are an almost extinct species. This is infuriating at times. We ingest a lot of data from 3rd party systems of our clients for things like KPI analysis,

> ops are an almost extinct species. This is infuriating at times. We ingest a lot of data from 3rd party systems of our clients for things like KPI analysis, the devs¹ provided various APIs for submitting data and we can read from various others, but good old reliable dumping-CSV-files-to-us-via-SFTP is still what most clients want to do. I have no problem with this², BUT the amount of times I end up talking to clients (sometimes large banking organisations) that don't have anyone available who actually understands SFTP/SSH amazes me. We often resort to password based auth because they simply don't get private/public keys (or claim their other infrastructure just doesn't work with them). ---- [1] used to be a full-stacker myself, but I've let large parts of my skillset rot so except for when working on legacy issues I'm a more specialised database/data-processing/infrastructure fellow these days [2] other than, of course, dealing with bad CSV output implementations⁴ or just all the combinations of how to encode problems like quotes, eols in long stings, and so forth. JSON³ is starting to get some traction as an alternative which is handy because being more modern it is more rigidly designed compared to the mess that CSV is due to decades of implementations being slapped together in an it'll-do-until-it-doesn't manner. [3] Don't expect structured data though: the clients & 3rd parties I deal with are sending the same simple table format data, with multiple files for us to join together later when there is more complex structure, that they would send as CSV, just using the more bloated format⁵ for the gain in reduced encoding confusions. [4] not quoting values containing the separator character or otherwise escaping it, not dealing with existing quotes when values do get quoted, etc. - if you'd be surprised to find CSV is far from a solved problem today, then prepare to be very surprised! [5] Don't get me wrong when I mention the format bloat: this _is_ a good trade for easing the other issues. Bandwidth & storage (or compute resource for compression) are more than cheap & fast enough to make it so.

fifilura 2y ago
Would it be possible for you to take a step back and describe the problem you are infuriated by again?
- dspillett 2y ago
  Sitting on the phone with people who's job it is to know these things for their organisation, having to explain to them that no, they shouldn't send me the private key, not even (especially not even) if they send the passphrase too, and many other problems. There seems to be a dearth of people for the ops jobs that need doing, hence OP's “ops are an almost extinct species”, so the tasks are thrown around and have to be done by the first person who doesn't duck fast enough. Those of us on the other side of the company interaction boundary end up having to help them with what should be the basics. I'm not their infrastructure team, I don't have access to be able to answer some of the questions they ask about their own infrastructure. I shouldn't even be client facing never mind teaching them how to do their jobs. The infuriating part (perhaps “frightening” is a better word here) is that there are people managing the secure transmission of data (sometimes sensitive personal data) who do not understand what is going on.
  - Corrado 2y ago
    Tell me about it. I'm in the healthcare industry and hospital systems are way too complex and/or they don't have the proper people to manage them. Getting them to understand SFTP is hard enough, let alone public key encryption. Additionally, their systems are often so old and out of date that they don't support anything newer than SFTP. On top of that the regulations they have to follow are just overwhelming and completely out of date. Honestly, I kinda feel sorry for the poor DevOps people in hospitals. It's got to be torture.
  - fifilura 2y ago
    Thank you for taking the time to reply. FWIW my comment was added a few minutes before your edits so it only contained the [1] footnote and possibly other edits. I guess I was missing the "why" where your post focused on the "what". Of course when you are working with sensitive data like that it makes a lot of sense to be really careful about security. I am well aware about the problems with the csv format. But a lot of times you can not really expect clients to be able to download and decode a parquet file, when all they want is to submit a password, download the data and look at it in excel.
kuekacang 2y ago
sftp? That's too modern. We had 3 or 4 vendors that only do ftp via vpn. Also ops and and networking/vpn is on different departments. > [1] am currently heading to a similar path
vladvasiliu 2y ago
> clients (sometimes large banking organisations) that don't have anyone available who actually understands SFTP/SSH Oh man, I've lost track of the number of times I was infuriated by this. I especially love the guys pretending to take security seriously only to show up with some obsolete ssh config that doesn't work out of the box on modern systems.
jiggawatts 2y ago
Why even use SFTP? Just have them upload blobs to S3 or Azure Storage. The security is better and both are serverless solutions that cost basically nothing at normal scale. I seriously don’t understand why people in 2024 insist in reproducing these core services but badly and at great expense. PS: Azure Storage is multi-protocol and supports SFTP! https://learn.microsoft.com/en-us/azure/storage/blobs/secure-file-transfer-protocol-support https://learn.microsoft.com/en-us/azure/storage/blobs/secure...
- KronisLV 2y ago
  > Just have them upload blobs to S3 or Azure Storage. Not everyone can store their data in cloud services, most likely. That said, self-hosted S3 compatible solutions like MinIO might be a good choice: https://github.com/minio/minio https://github.com/minio/minio or maybe SeaweedFS for something with a more permissive license: https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs
  - marcus0x62 2y ago
    > That said, self-hosted S3 compatible solutions like MinIO might be a good choice But then you would have people who can't figure out SSH trying to figure out private S3-compatible infrastructure.
    - more_corn 2y ago
      Having walked people through both I’m going to say setting up a client for an S3 clone is easier.
  - jiggawatts 2y ago
    > Not everyone can store their data in cloud This is a nearly verbatim conversation I’ve had three or four times in the last twelve months: “Why not just use blob storage for the data ingest?” “We have security standards for sensitive data; we have to use private links and validated protocols and systems.” “Okay, what’s step two? Where does the data end up? Where do end users consume it from?” “Power BI!” “The local app?” “No, the online version.” “You just said you can’t use cloud services like blob storage!” “We can’t!” “You’re using Power BI!” “That’s different!” “How?” “Err…”

fifilura 2y ago

Would it be possible for you to take a step back and describe the problem you are infuriated by again?

dspillett 2y ago
Sitting on the phone with people who's job it is to know these things for their organisation, having to explain to them that no, they shouldn't send me the private key, not even (especially not even) if they send the passphrase too, and many other problems. There seems to be a dearth of people for the ops jobs that need doing, hence OP's “ops are an almost extinct species”, so the tasks are thrown around and have to be done by the first person who doesn't duck fast enough. Those of us on the other side of the company interaction boundary end up having to help them with what should be the basics. I'm not their infrastructure team, I don't have access to be able to answer some of the questions they ask about their own infrastructure. I shouldn't even be client facing never mind teaching them how to do their jobs. The infuriating part (perhaps “frightening” is a better word here) is that there are people managing the secure transmission of data (sometimes sensitive personal data) who do not understand what is going on.
- Corrado 2y ago
  Tell me about it. I'm in the healthcare industry and hospital systems are way too complex and/or they don't have the proper people to manage them. Getting them to understand SFTP is hard enough, let alone public key encryption. Additionally, their systems are often so old and out of date that they don't support anything newer than SFTP. On top of that the regulations they have to follow are just overwhelming and completely out of date. Honestly, I kinda feel sorry for the poor DevOps people in hospitals. It's got to be torture.
- fifilura 2y ago
  Thank you for taking the time to reply. FWIW my comment was added a few minutes before your edits so it only contained the [1] footnote and possibly other edits. I guess I was missing the "why" where your post focused on the "what". Of course when you are working with sensitive data like that it makes a lot of sense to be really careful about security. I am well aware about the problems with the csv format. But a lot of times you can not really expect clients to be able to download and decode a parquet file, when all they want is to submit a password, download the data and look at it in excel.

kuekacang 2y ago

sftp? That's too modern. We had 3 or 4 vendors that only do ftp via vpn. Also ops and and networking/vpn is on different departments. > [1] am currently heading to a similar path

vladvasiliu 2y ago

> clients (sometimes large banking organisations) that don't have anyone available who actually understands SFTP/SSH Oh man, I've lost track of the number of times I was infuriated by this. I especially love the guys pretending to take security seriously only to show up with some obsolete ssh config that doesn't work out of the box on modern systems.

jiggawatts 2y ago

Why even use SFTP? Just have them upload blobs to S3 or Azure Storage. The security is better and both are serverless solutions that cost basically nothing at normal scale. I seriously don’t understand why people in 2024 insist in reproducing these core services but badly and at great expense. PS: Azure Storage is multi-protocol and supports SFTP! https://learn.microsoft.com/en-us/azure/storage/blobs/secure-file-transfer-protocol-support https://learn.microsoft.com/en-us/azure/storage/blobs/secure...

KronisLV 2y ago
> Just have them upload blobs to S3 or Azure Storage. Not everyone can store their data in cloud services, most likely. That said, self-hosted S3 compatible solutions like MinIO might be a good choice: https://github.com/minio/minio https://github.com/minio/minio or maybe SeaweedFS for something with a more permissive license: https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs
- marcus0x62 2y ago
  > That said, self-hosted S3 compatible solutions like MinIO might be a good choice But then you would have people who can't figure out SSH trying to figure out private S3-compatible infrastructure.
  - more_corn 2y ago
    Having walked people through both I’m going to say setting up a client for an S3 clone is easier.
- jiggawatts 2y ago
  > Not everyone can store their data in cloud This is a nearly verbatim conversation I’ve had three or four times in the last twelve months: “Why not just use blob storage for the data ingest?” “We have security standards for sensitive data; we have to use private links and validated protocols and systems.” “Okay, what’s step two? Where does the data end up? Where do end users consume it from?” “Power BI!” “The local app?” “No, the online version.” “You just said you can’t use cloud services like blob storage!” “We can’t!” “You’re using Power BI!” “That’s different!” “How?” “Err…”