5 ms·

Why even use SFTP? Just have them upload blobs to S3 or Azure Storage. The security is better and both are serverless solutions that cost basically nothing at

Why even use SFTP? Just have them upload blobs to S3 or Azure Storage. The security is better and both are serverless solutions that cost basically nothing at normal scale. I seriously don’t understand why people in 2024 insist in reproducing these core services but badly and at great expense. PS: Azure Storage is multi-protocol and supports SFTP! https://learn.microsoft.com/en-us/azure/storage/blobs/secure-file-transfer-protocol-support https://learn.microsoft.com/en-us/azure/storage/blobs/secure...

KronisLV 2y ago
> Just have them upload blobs to S3 or Azure Storage. Not everyone can store their data in cloud services, most likely. That said, self-hosted S3 compatible solutions like MinIO might be a good choice: https://github.com/minio/minio https://github.com/minio/minio or maybe SeaweedFS for something with a more permissive license: https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs
- marcus0x62 2y ago
  > That said, self-hosted S3 compatible solutions like MinIO might be a good choice But then you would have people who can't figure out SSH trying to figure out private S3-compatible infrastructure.
  - more_corn 2y ago
    Having walked people through both I’m going to say setting up a client for an S3 clone is easier.
- jiggawatts 2y ago
  > Not everyone can store their data in cloud This is a nearly verbatim conversation I’ve had three or four times in the last twelve months: “Why not just use blob storage for the data ingest?” “We have security standards for sensitive data; we have to use private links and validated protocols and systems.” “Okay, what’s step two? Where does the data end up? Where do end users consume it from?” “Power BI!” “The local app?” “No, the online version.” “You just said you can’t use cloud services like blob storage!” “We can’t!” “You’re using Power BI!” “That’s different!” “How?” “Err…”
  - dspillett 2y ago
    Similar, though only for one client ATM, and our clients are fine with cloud storage these days as long as everything is in an account properly locked down (bring your own key, etc.) so only their application instances can access it. At least the data doesn't rest (as far as we know…) in PowerBI.
eddythompson80 2y ago
You're simply missing the point. > I seriously don’t understand why people in 2024 insist in reproducing these core services but badly and at great expense. Because those users have a flow that they like and don't want to change it regardless of all the pitfalls in that flow. "It works, there is just this one problem. Why don't you invent a solution for this one problem for us. I don't want to use S3, or Azure or whatever. Just give me an sftp endpoint I can upload files to". Oh CSV doesn't have a standardized way of parsing escapes? why don't you handle a `# escape=\` or `# doublequote` or `# singlequote` or `# ignore-new-lines-in-the-middle-of-a-string` and we will use that. Heck, it's 2024, and we have a fully automated, source control (github, gitlab, bitbucket, etc) integrated, S3/Azure Storage/GCP Buckets integrated, Dockerfile and Makefile aware, application deployment service and we still have a non-trivial number of users asking "Why can't I just drag and drop my .php or .py files over fileZilla to deploy my code? I don't want to use git or docker or S3". Oh, you can't version our code updates when we just drag 100 files over sftp? why don't we update a `version.txt` file when we're done updating then you count that as a version?
- jiggawatts 2y ago
  > Because users have a flow that they like https://www.bbc.com/news/articles/cx82407j1v3o https://www.bbc.com/news/articles/cx82407j1v3o Some of these solutions were never good, but build a momentum of “we’ve always done it this way”.
dspillett 2y ago
> Why even use SFTP? Because that is the infrastructure they have, what their current arrangements support. We can offer a number of other methods, but if their other systems don't support than and they don't want to pay for upgrades, SFTP it is. Not that I have anything against gobs of data arriving via SFTP for cleaning and importing¹ – it is tried & tested, reliable, and if both sides have a clue² it is easy & pain free. Much the same set of reasons they still want to send us CSV formatted data, often Win1252 encoded rather than UTF8, not JSON or something else better defined. > Just have them upload blobs to S3 or Azure Storage. The data is ending up in Azure storage in our case. But any transfer method still has identification and authentication matters to deal with, and they need to understand and their other software needs to support, the method chosen. This is why SFTP often wins by default: it is a relatively proven technique that people see as secure, and it is commonly supported. > The security is better and both are serverless solutions that cost basically nothing at normal scale. The cost to the client is only zero if their other systems already support it. They are very much against anything that implies extra development costs or the cost & effort of assessing and switching to newer systems. > PS: Azure Storage is multi-protocol and supports SFTP! SFTP support in Azure is a pretty recent addition, some time in this last year IIRC. We (well, I) had to put together our own relay arrangement some years ago: OpenSSH, appropriately locked down, with blob containers in storage accounts mounted via blobfuse). It works well, in nice cheap VMs/containers. SFTP support in Azure works out rather expensive for our use case, or any reasonable use case IMO. It needs the hierarchical namespace enabled though that is “only” an additional $27/month per storage account, but then you need to pay for SFTP support by the hour!³⁴ Unless we constrain the clients to sending data at specific times that comes to >$200 per month, per account. We can't get away with charging our smaller clients, or the bigger ones for that matter, an extra $2,400/yr nor can we afford to eat that ourselves. See https://azure.microsoft.com/en-gb/pricing/calculator/ https://azure.microsoft.com/en-gb/pricing/calculator/ for reference. We could work around the per-account part by stuffing all the data into multiple containers in one account – but that would not fly with some of the requirements financial sector companies understandably have for data at rest, so we would never pass an audit if we did that. Many even want their live and UAT/training/other environments to have completely separate storage accounts so that is potentially more than one lot of up to $2,400 per year per client needing SFTP. And even on one account, $2,400/year to enable 24/7 availability of SFTP transfers is absolutely ridiculous. ---- [1] I'd rather it be cleaned first, but that is a separate issue [2] This part is, of course, the problem… [3] $0.30/hour: https://azure.microsoft.com/en-gb/pricing/details/storage/blobs/#:~:text=SFTP%20enabled%20for%20storage%20account%20(per%20hour) https://azure.microsoft.com/en-gb/pricing/details/storage/bl... [4] No shit… https://learn.microsoft.com/en-gb/azure/storage/blobs/secure-file-transfer-protocol-support#:~:text=To%20avoid%20passive%20charges%2C%20consider%20enabling%20SFTP%20only%20when%20you%20are%20actively%20using%20it%20to%20transfer%20data https://learn.microsoft.com/en-gb/azure/storage/blobs/secure....

KronisLV 2y ago

> Just have them upload blobs to S3 or Azure Storage. Not everyone can store their data in cloud services, most likely. That said, self-hosted S3 compatible solutions like MinIO might be a good choice: https://github.com/minio/minio https://github.com/minio/minio or maybe SeaweedFS for something with a more permissive license: https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs

marcus0x62 2y ago
> That said, self-hosted S3 compatible solutions like MinIO might be a good choice But then you would have people who can't figure out SSH trying to figure out private S3-compatible infrastructure.
- more_corn 2y ago
  Having walked people through both I’m going to say setting up a client for an S3 clone is easier.
jiggawatts 2y ago
> Not everyone can store their data in cloud This is a nearly verbatim conversation I’ve had three or four times in the last twelve months: “Why not just use blob storage for the data ingest?” “We have security standards for sensitive data; we have to use private links and validated protocols and systems.” “Okay, what’s step two? Where does the data end up? Where do end users consume it from?” “Power BI!” “The local app?” “No, the online version.” “You just said you can’t use cloud services like blob storage!” “We can’t!” “You’re using Power BI!” “That’s different!” “How?” “Err…”
- dspillett 2y ago
  Similar, though only for one client ATM, and our clients are fine with cloud storage these days as long as everything is in an account properly locked down (bring your own key, etc.) so only their application instances can access it. At least the data doesn't rest (as far as we know…) in PowerBI.

eddythompson80 2y ago

You're simply missing the point. > I seriously don’t understand why people in 2024 insist in reproducing these core services but badly and at great expense. Because those users have a flow that they like and don't want to change it regardless of all the pitfalls in that flow. "It works, there is just this one problem. Why don't you invent a solution for this one problem for us. I don't want to use S3, or Azure or whatever. Just give me an sftp endpoint I can upload files to". Oh CSV doesn't have a standardized way of parsing escapes? why don't you handle a `# escape=\` or `# doublequote` or `# singlequote` or `# ignore-new-lines-in-the-middle-of-a-string` and we will use that. Heck, it's 2024, and we have a fully automated, source control (github, gitlab, bitbucket, etc) integrated, S3/Azure Storage/GCP Buckets integrated, Dockerfile and Makefile aware, application deployment service and we still have a non-trivial number of users asking "Why can't I just drag and drop my .php or .py files over fileZilla to deploy my code? I don't want to use git or docker or S3". Oh, you can't version our code updates when we just drag 100 files over sftp? why don't we update a `version.txt` file when we're done updating then you count that as a version?

jiggawatts 2y ago
> Because users have a flow that they like https://www.bbc.com/news/articles/cx82407j1v3o https://www.bbc.com/news/articles/cx82407j1v3o Some of these solutions were never good, but build a momentum of “we’ve always done it this way”.

dspillett 2y ago

> Why even use SFTP? Because that is the infrastructure they have, what their current arrangements support. We can offer a number of other methods, but if their other systems don't support than and they don't want to pay for upgrades, SFTP it is. Not that I have anything against gobs of data arriving via SFTP for cleaning and importing¹ – it is tried & tested, reliable, and if both sides have a clue² it is easy & pain free. Much the same set of reasons they still want to send us CSV formatted data, often Win1252 encoded rather than UTF8, not JSON or something else better defined. > Just have them upload blobs to S3 or Azure Storage. The data is ending up in Azure storage in our case. But any transfer method still has identification and authentication matters to deal with, and they need to understand and their other software needs to support, the method chosen. This is why SFTP often wins by default: it is a relatively proven technique that people see as secure, and it is commonly supported. > The security is better and both are serverless solutions that cost basically nothing at normal scale. The cost to the client is only zero if their other systems already support it. They are very much against anything that implies extra development costs or the cost & effort of assessing and switching to newer systems. > PS: Azure Storage is multi-protocol and supports SFTP! SFTP support in Azure is a pretty recent addition, some time in this last year IIRC. We (well, I) had to put together our own relay arrangement some years ago: OpenSSH, appropriately locked down, with blob containers in storage accounts mounted via blobfuse). It works well, in nice cheap VMs/containers. SFTP support in Azure works out rather expensive for our use case, or any reasonable use case IMO. It needs the hierarchical namespace enabled though that is “only” an additional $27/month per storage account, but then you need to pay for SFTP support by the hour!³⁴ Unless we constrain the clients to sending data at specific times that comes to >$200 per month, per account. We can't get away with charging our smaller clients, or the bigger ones for that matter, an extra $2,400/yr nor can we afford to eat that ourselves. See https://azure.microsoft.com/en-gb/pricing/calculator/ https://azure.microsoft.com/en-gb/pricing/calculator/ for reference. We could work around the per-account part by stuffing all the data into multiple containers in one account – but that would not fly with some of the requirements financial sector companies understandably have for data at rest, so we would never pass an audit if we did that. Many even want their live and UAT/training/other environments to have completely separate storage accounts so that is potentially more than one lot of up to $2,400 per year per client needing SFTP. And even on one account, $2,400/year to enable 24/7 availability of SFTP transfers is absolutely ridiculous. ---- [1] I'd rather it be cleaned first, but that is a separate issue [2] This part is, of course, the problem… [3] $0.30/hour: https://azure.microsoft.com/en-gb/pricing/details/storage/blobs/#:~:text=SFTP%20enabled%20for%20storage%20account%20(per%20hour) https://azure.microsoft.com/en-gb/pricing/details/storage/bl... [4] No shit… https://learn.microsoft.com/en-gb/azure/storage/blobs/secure-file-transfer-protocol-support#:~:text=To%20avoid%20passive%20charges%2C%20consider%20enabling%20SFTP%20only%20when%20you%20are%20actively%20using%20it%20to%20transfer%20data https://learn.microsoft.com/en-gb/azure/storage/blobs/secure....