14 ms·
AWS Transfer for SFTP – Fully Managed SFTP Service for Amazon S3
- koolba 8y agoEnterprises will love this. There are so many legacy app flows kicked off via sftp/scp file drops. Being able to hook into those via lambda events on the associated S3 bucket will create a whole ecosystem of enterprise spaghetti for years to come.
- mattdeboard 8y agoSpaghetti as a service, delicious
- notyourwork 8y agoWelcome to the real world.
- mattdeboard 8y agoplop
- mattdeboard 8y ago(this was me plopping into the real world)
- brazzledazzle 8y agoThe only downside is whitelisting but not on the SFTP server-side. Many enterprises restrict egress SFTP (usually for security reasons) so you need to provide IPs and they can’t frequently change because it can take enterprise network admins quite some time to deal with all of the bureaucracy and change control. That said, I wouldn’t be surprised if modern networking gear can handle CNAMEs but there’s no guarantee that they’re using modern gear or if they are that the questionable outsourced team even knows how to deal with the modern capabilities. This will certainly help a lot of use cases though.
- blincoln 8y agoIt's less whether the gear is modern and more about the layer that it operates at. A network firewall doesn't see the DNS name that an internal system looked up in order to make an outbound connection. It just sees the source/destination IP/port. Processing a rule based on source/destination IP or CIDR and port is very fast, and all happens locally. Trying to make that device handle rules by IP address is pretty tricky. Does it do a reverse lookup on the destination IP? That may not give a result that's even remotely like what the client used, especially for cloud-hosted destinations. For a lot of applications (probably including this one), a proxy is a good approach, because DNS resolution can be delegated to the proxy, and therefore the proxy can easily apply DNS-based rules as well as IP/CIDR-based rules. However, proxies tend to make people unhappy because they generally require at least some configuration on the client side. Microsoft used to sell a product[1] that made this transparent for Windows clients[2], but obviously that doesn't help for most modern shops where a lot of the systems are Linux, MacOS, etc. [1] Internet Security and Acceleration Server ("ISA"), later renamed to Threat Management Gateway ("TMG"), now deprecated and approaching EOL. [2] It hooked into the network stack and rerouted requests based on a proxy routing rule table. Imagine a centrally-managed proxychains, but with the system configured to default to check the proxychains config file for every outbound TCP connection.
- icebraining 8y agoI wonder if you could use the DNS resolution cache itself to do the reverse lookup. As long as the DNS cache lasted at least as long as the TTL, it should work.
- brazzledazzle 8y agoYeah I assumed something like Palo Alto’s or maybe ASA’s could do more since (I believe) they’re doing actual inspection but I’m only familiar with them in passing.
- LinuxBender 8y agoYou can get a dedicated S3 gateway which has a static IP.
- mrweasel 8y ago>There are so many legacy app flows kicked off via sftp/scp file drops Yes,.... Legacy apps... because no one would choose SFTP for system that designed in 2017. Seriously this is great, so many solution rely on SFTP, but so many companies fail at managing the service. Having an SFTP service that just works and is secure (hopefully) will help a ton of compnies.
- dested 8y agoThere are a handful of services out there that do this, I know because I've needed it on multiple occasions. It's nice that Amazon is providing it in house now, but it just reminds of me the last time I went to re:invent and walked through the vendor area and thought about how many of these companies are four dev cycles away from Amazon producing a baked in competitor. Hard to make a B To B Amazon tool these days.
- scarface74 8y agoDave Winer: http://scripting.com/stories/2007/04/04/coexistingWithPlatformVend.html http://scripting.com/stories/2007/04/04/coexistingWithPlatfo... Sometimes developers choose a niche that's either directly in the path of the vendor, or even worse, on the roadmap of the vendor. In those cases, they don't really deserve our sympathy. It's almost like a game of PR, there's no way you're not going to have a fight on your hands. Further discussed by Joel Spolsky https://www.joelonsoftware.com/2009/06/10/platform-vendors/ https://www.joelonsoftware.com/2009/06/10/platform-vendors/
- kristianp 8y agoThanks. At the end of Spolsky's article he mentions a way to survive: "A good platform always has opportunities for applications that aren’t just gap-fillers. These are the kind of application that the vendor is unlikely ever to consider a core feature, usually because it’s vertical — it’s not something everyone is going to want. There is exactly zero chance that Apple is ever going to add a feature to the iPhone for dentists. Zero."
- scarface74 8y agoWatch next year at WWDC they integrate a feature into HealthKit that uses the FaceID sensors to tell if you have cavities.....
- ryanSrich 8y agoThat’s quite literally AWS’s playbook. Fill the vendor hall with tech, learn it, see what sticks and then crush it. This is why I’m bullish on cloud agnostic tech. These practices don’t typically fair well in the enterprise space. This is why companies like MSFT are interesting to me. They partner and rarely kill. Amazon is the complete opposite.
- viggity 8y agoWe currently pay $250/month through some small vendor for hipaa compliant sftp hosting (that we transfer a whopping 50kb on a weekly basis). I always felt like it was a rip off, but azure/aws didn't have their own version. And I'm loathe to manage a VM. PaaS is my sugar bear. My eyes lit up when I saw this. We're an azure shop, but I'm not afraid to use AWS for limited cases. Then I saw - $.30/hr (so, $214/mo). Really? REALLY? Wouldn't it be comically easy to just the add SFTP as a protocol option for S3? Why does this need a dedicated VM to run it? (Yes, I know this is PaaS and you don't manage the VM, but they're essentially pricing it that way)
- jimktrains2 8y agoI don't know about comically easy, but should be doable to have some sort of shared frontend. $214 (even $100) is a really beefy vm, though. I wonder whats provisioned and why.
- tecleandor 8y agoNothing beefy, probably, but as for HIPAA compliance, AFAIK, you need to sign several specific contracts with your provider and blahblahblah, probably they're just billing you for the incovenience and for having the HIPAA seal. "I'm HIPAA, the rest aren't, just pay for it".
- pg_bot 8y agoYou have to sign a single business associate agreement (BAA) depending on the nature of the business you are working with. These are usually boilerplate contracts around 2-3 pages long and full of legalese. It is uncommon for someone to charge you for signing a BAA. It is very common to tie these plans into enterprise only pricing. This is terrible because it adds unnecessary costs to the medical system (which get passed onto consumers) and it completely shuts out smaller players from entering the marketplace. (Cue me side eyeing every single error tracking software SaaS currently on the market who wants to start at 5K a year for their 'small' plan - get real guys)
- sigi45 8y ago1tb for 40,- dollar? What? Did i just became stupid?
- blaisio 8y agoOMG this will save me so much time if it works. I wish they had this feature a few years ago!
- hemancuso 8y agoI'd be curious how this handles all the posix cases not well suited to object storage. Renaming a folder than has a million files/folders inside is a single operation in SFTP, but 2 million operations on S3. Does it handle writing at arbitrary offsets within a file? Does it download the file first then let you start writing? What about just writing a few bytes at the beginning of a large existing file and then closing your SFTP handle? How about 2 users accessing same file via SFTP at the same time?
- NathanKP 8y agoI don't work on the S3 team so I can't answer all these questions but relative to the "2 million operations on S3" question I can point out that S3 now has batch operation support: https://aws.amazon.com/about-aws/whats-new/2018/11/s3-batch-operations/ https://aws.amazon.com/about-aws/whats-new/2018/11/s3-batch-...
- burtonator 8y agoI think the point being that when a filesystem the mv is atomic and just updates an inode but on S3 those operations can take place on thousand of different machines.
- icebraining 8y agoSFTP is not a filesystem, though; to get a rename to be atomic, you must pass SSH_FXF_RENAME_ATOMIC, and the server can return SSH_FX_OP_UNSUPPORTED.
- hemancuso 8y agoNearly any SFTP client would assume a rename would be a near-instant operation on the server, and would probably fall-over if it took an hour.
- consultutah 8y agoThis is something that AWS needs to change. I work with millions of large files and have to keep a very large (PBs) local storage array just to make sure that things are right before uploading to S3 so that I don’t have to pay and wait for arch changes like this.
- pnutjam 8y agoSftp is natively supported by Linux. I'm surprised this is a thing.
- AnaniasAnanas 8y agoit isn't to my knowledge. You need an ssh client that supports sftp.
- pnutjam 8y agoUnless you disable it in the sshd_config, it's supported by most Linux distributions. Yes, you'll need a client, but any modern client supports sftp. The only tricky part is chrooting the users.
- burtonator 8y agoThis would mean that rsync now works with S3 as it has an sftp target... correct?
- dpedu 8y agoNo, rsync over ssh requires the rsync binary on the remote side. Possible for Amazon to add but I don't see it mentioned.
- LinuxBender 8y agoYou should try the mirror sub-system of lftp [1]. It can replicate rsync behavior on a chroot sftp server. No idea if that works on Amazon, but I use it all the time on my own chroot sftp servers. [1] - https://tinyvpn.org/sftp/#lftp https://tinyvpn.org/sftp/#lftp
- lathiat 8y agolftp is fantastic. The mirror function has a “reverse mode” too For regular tasks you could also look at “rclone” which is like rsync in many ways but can upload to s3, backblaze b2, sftp and any more directly. Without remote support.
- ac29 8y agoYou might just want to look into rclone, which supports S3 and many other cloud services natively.
- atonse 8y agoAh this would've been so useful 18 months ago. I had to spend MONTHS to convince a vendor (government) to use S3 to upload (keybase-encrypted) files instead of SFTP. And they finally budged. This would've been so much easier.
- dagss 8y agoOne can also implement SFTP on top of anyting just like HTTP apis if you make your own backend, e.g. https://github.com/pkg/sftp https://github.com/pkg/sftp
- crazysim 8y agoHere's an example: https://github.com/moriyoshi/s3-sftp-proxy https://github.com/moriyoshi/s3-sftp-proxy No FUSE. Pure Go so it's low on resource usage and high in platform compatibility. No OpenSSH. No screwing around with Linux users or whatever. Just a single declarative configuration file. You can run this baby in a Docker container with some adjustments to the host if you want this on port 22. I had to sourcegraph GitHub a bit to find this thing. SEO is so bad on this implementation. I don't know why.
- rob 8y agoIt's 2018 though, that's unnecessary. Amazon should of had this standard years ago. Glad it's here!
- emilfihlman 8y agoAnyone became extremely annoyed by the constant "breathing" of Amazon Polly?
- dagss 8y agoAn alternative to doing file based SFTP is to just treat SFTP like an API. A company I work for implemented an SFTP service where every operation simply translates to some SQL DB lookup. And a file download kicks off a larger SQL query and generates the report on the fly, streaming the result straight through to the SFTP client. Works great! SFTP can be an API just like HTTP. Under the hood the protocol is reasonably contained and doesn't require a filesystem backend at all. Depends a lot on the usecase of course. See https://github.com/pkg/sftp https://github.com/pkg/sftp
- LoSboccacc 8y agoI did something similar using the apache ftp client, as long as the resources are identified by a path it's very convenient and extremely easy, just implement a FileSystemFactory with apache mina and off you go.
- hemancuso 8y agoWhat if you open an SFTP handle, and then write 5 bytes halfway through a 20 GB file and close the handle? How do you translate that?
- ddorian43 8y agohint: he doesn't.
- LinuxBender 8y agoYou could. I mean, at least with OpenSSH you can specify a byte range. That is how lftp is able to chop up files into many streams on SFTP. I can't imagine anyone doing this with a database however, at least, not for writes.
- e12e 8y agoSimilar to how you might handle writing 5 bytes 20gb offset into /dev/null?
- icebraining 8y agoSeems similar to if someone sends an HTTP method to an URL that doesn't support it. Just return an error.
- matchagaucho 8y agoWe use a number of AWS LightSail servers for SFTP today, which mostly sit idle. Will definitely adopt this!
- secabeen 8y agoReally? Lightsail costs you $0.0047 USD/hour. This costs $0.30/hour. It's ~80x more.
- matchagaucho 8y agoDidn't notice SFTP costs. Guess I'm sticking with LightSail.
- pnutjam 8y agoThe price is significantly more.
- markstos 8y agoThe AWS pricing page for this service says it costs about $225/month for a lightly used instance. I implemented the same kind of thing on AWS using a nano-sized instance for about $10/month. The instance is managed with an Ansible Role for automated SFTP server management. I connected it with an off-the-shelf AWS Lamda function which listens for S3 PUT events and copies files to the SFTP server as needed. My solution took a little more human-time to setup than the AWS service might, but once setup, it saves about $200/month.
- scarface74 8y ago$200 a month is nothing for a business. Anything that we don't have to manage ourselves or worry about reliability, scalability, and we can just use our AWS business support plan is a win.
- secabeen 8y agoIn large business, sure. In small business and education, a $200/mo commitment could easily require approval by the owner or the department chair.
- martyhu 8y agoI'd love to see something for plain vanilla FTP. Many enterprises still use it, would love to see AWS support that as well.
- merb 8y agoplease no. ftp is a security nightmare.
- jessaustin 8y agoYes that sounds kind of like "loaded guns for young children as a service"...
- DenisM 8y agoIt seems there are no web-hooks / callbacks, so you don't get notified when a new file is uploaded (or someone downloads a file). Another issue is that if your have to support a partner with SFTP data transfer requirements you may have to support one with FTP/FTPS requirements as well. At this point you will have to go to a dedicated FTP server (or outsource it to another company) anyway, and AWS SFTP service will be redundant in this scheme.
- unfunco 8y agoIt's S3, so you can use Lambda, it says so in the article. > You can write AWS Lambda functions to to build an “intelligent” FTP site that processes incoming files as soon as they are uploaded, query the files in situ using Amazon Athena, and easily connect to your existing data ingestion process.
- vorticalbox 8y agoYou can set a lambda to be triggered on a file upload, at work we do this for creating reports. Lambda dumps mongo data to an s3 bucket which triggers another lambda to create a csv.
- DenisM 8y agoI see. Lambdas are more fussy though, I can’t just trigger an action in my server, I have to write code that does that.
- merlincorey 8y agoCallbacks/webhooks are handled by Lambda, according to the article.
- jopsen 8y agoCharged by the hour... pay for an instance.. Tsk tsk.. The thing I love about S3 and cloud services in general is when I pay per request and can scale through the roof. Whenever a services is meter by number of instances my interest fades, and I look for other solutions.. S3 has this very handsoff feeling to it :)
- wmf 8y agoAWS should investigate a new concept called serverless.
- pm90 8y agoWell, its likely because the sftp server itself is always running. If they had an sftp lambda function... sure.
- jopsen 8y agoSure, it's because an sftp server is always running. But I immediately extrapolate that this also means it has bandwidth limits and limits on concurrency, etc.
- retromario 8y agoAnyone know if this can be used directly from CloudFormation?
- scarface74 8y agoHow are you envision using it.
- retromario 8y agoIdeally the same way we define an EC2 instance, perhaps bound directly to an S3 bucket resource defined in the same script. Ideally reading the config definition from an S3 file that we can update at will.
- scarface74 8y agoIf the file is PUT in a S3 bucket via SFTP, you can trigger a lambda that run CF template. But that seems kind of Rube Goldbergish. Why not make the change to the file, push it to Git and use CodePipeline with lambda?
- yjftsjthsd-h 8y agoNot as useful if you're invested in CF, but Terraform can be better about getting new features. Just FYI.
- syntheticcdo 8y agoNew services rarely launch with CF support, if you want to programmatically create SFTP servers TODAY you could write a Lambda that uses the SDK and reference that Lambda with a CF Custom Resource.
- retromario 8y ago> New services rarely launch with CF support Yeah I'm slowly learning that. Thanks for the tip, at least it's a step in the right direction.
- manigandham 8y agoThis is why AWS is so far ahead: survey the landscape, find the things they don't already cover, and come up with a managed service for it. It's usually not perfect, but it almost always just works.
- qwerty456127 8y ago> SFTP (Secure File Transfer Protocol) It's SSH File Transfer Protocol. When you say Secure File Transfer Protocol many people think about FTP over SSL if you don't emphasize it's about SSH.
- floatboth 8y ago> many people think about FTP over SSL if you don't emphasize it's about SSH Huh? Sure there's always potential for confusion but every time I heard anything about FTP over SSL (which no one seems to actually use) it's been called "FTPS"
- qwerty456127 8y agoI agree FTPS is the right acronym for this but I had to correct people about this all the time. So many people actually have no idea SSH does more than just letting you execute command line programs on a remote server and FTP is not the only/best protocol to access remote file systems over the Internet.
- rob 8y agohttps://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol In computing, the SSH File Transfer Protocol (also Secure File Transfer Protocol, or SFTP) is a network protocol that provides file access, file transfer, and file management over any reliable data stream.
- Spivak 8y agoFTPS is FTP over TLS and is pretty much dead. Any time someone says SFTP it's SSH.
- Crontab 8y agoI've never even heard of FTP over SSL. Is that something that never became popular due to the existence of OpenSSH?
- mr_toad 8y ago
- bmilleare 8y agoA tonne of our (enterprise-ey) customers had such trouble trying to integrate into our S3 flow that we started launching VPS for each that abstracted it away into simple SFTP upload/download, which they were used to. Although this is much more expensive than Lightsail, the man hours saved will make it worthwhile.
- Spivak 8y agoCan you elaborate? I mean a plain CentOS server running SFTP, S3FS seems about as set and forget as it gets. And each? Surely chrooting users would let you consolidate all of those servers into one (or one cluster for HA I suppose).
- acdha 8y ago> Can you elaborate? I mean a plain CentOS server running SFTP, S3FS seems about as set and forget as it gets. Think about the operational costs: someone needs to manage keys, logging, security updates, when S3FS coughs a lung and hangs you need to catch that problem and remount it to restore service, etc. This service reuses the existing authentication systems so you don't need to spend time configuring and managing integration with your customers’ LDAP/AD infrastructure, etc. If you deal with anything which hits PCI, HIPAA, etc. you need to be able to certify that your custom design meets those requirements as well. That's not to say you can't do it yourself but for many places there's a fairly significant amount of work where the cost of doing it yourself is greater than 5+ years of managed service costs.
- lordfoo 8y agoInstead of s3fs, what's wrong with using the aws s3 cli? That's what we do for our clients' SFTP servers
- acdha 8y agoS3fs means you can use most existing apps without managing local storage. It doesn’t work quite as well in practice but the concept is appealing if you need to support software which wasn’t designed for AWS and uses non-trivial data volumes.
- yawz 8y agoUnless I'm missing something, this functionality has been on the AWS Marketplace for a while. We've already used an SFTP Gateway straight out of the marketplace. This is a tough news for these folks, and generally speaking, if you're making good enough money off the marketplace, then you're possibly on the collision course with Amazon's "new" roadmap.
- jonstewart 8y agoPresumably this will handle large file uploads with aplomb? Multipart upload with s3 can be a pain (when you want someone else to be doing the uploading).
- CloudBuddy 8y agoIf your into plain vanilla SFTP and don't have large storage needs, https://cloudbuddy.cloud https://cloudbuddy.cloud takes less than 1 minute to setup.
- narsil 8y agoWe ended up implementing a REST API endpoint for SFTP to provide an easy way for web apps to transfer content without having to speak the FTP protocol: https://kloudless.com/products/file-storage/ https://kloudless.com/products/file-storage/ I can see this being valuable for apps to get user content into S3 more efficiently from the server-side rather than funneling it through hosted servers. The one caveat is programmatic user management, which I'm sure is possible.
- lsh 8y agoWe use Cloudgates for our FTP/SFTP to S3 interface: https://cloudgates.net/pricing https://cloudgates.net/pricing They're cheap, stable and dead simple to set up. This offering from AWS looks attractive, but at $.30/hour for the server makes it $219/mo vs $25/mo. edit: just a satisfied customer
- hrez 8y agoSo, how do I make sure my connection isn't MITM-ed? There is no server host key anywhere to compare. No CA certificate support. Doesn't look like ed25519 is supported either. Somehow people don't use self-signed certificates all over the web but for sftp it's "fine" apparently.
- sk5t 8y agoFor SSH (+SFTP) you are expected/obligated/etc. to have some way to verify the correct host key. There is no relationship to the clusterfudge of public CAs. Nor are there x509 certs.
- hrez 8y agoBut you can't verify it since there is no host key published for sftp service (at least in AWS console).
- jiveturkey 8y agothank you for the dose of reality.
- sneak 8y agoThis incurs per-hour charges to run the VM that runs sshd, same as running a micro instance with FUSE S3 would, although with slightly less admin attention required.
- xaduha 8y agoNot a single mention of WebDAV here, sad.
- agopaul 8y agoVery interesting considering that it enables to make legacy applications work with S3 basically. The price is quite high for small projects though: $0.30/hour > $216/m > $2592/y
- tonylemesmer 8y agosomewhat late to the commenting party but apparently WinSCP can communicate with S3: https://winscp.net/eng/docs/guide_amazon_s3 https://winscp.net/eng/docs/guide_amazon_s3
- twodave 8y agoSo, I spent a couple days a few months ago building exactly this on an ec-2 instance. I have an SFTP service running on an Ubuntu box, it has jailed homes for users, it's ssh-key-only, uses s3fs to persist things to the correct buckets, etc. My only problem with the managed service (which I'd LOVE to switch to tbh) is I can't for the life of me get it to actually connect and upload a file. I suspect I'm doing something wrong in IAM, but the tutorials suck and it looks like IAM isn't even ready for this service yet. I can get a user authenticated, but it's like it's trying to figure out where "home" is and crapping out, connection closed. Nothing helpful in the verbose output, either. Bummer.
- twodave 8y agoAnd to emphasize, the process of simply adding a user to this thing SUCKS. In my homebrew instance, it's just a matter of generating the key pair and dropping the public key into a folder on S3. Cron job reads the bucket, creates new users/homes/etc for anything new, all pasted together using bash scripts basically. But at the end of the day it's ridiculously simple. I'd hoped a fully managed solution would actually be simpler (instead of simply more stable because it's managed, after all).