17 ms·
SFTPGo: A Full Featured SFTP Server in Go
- todd3834 7y agoGo seems like a really good fit for this kind of project
- alpb 7y agoYeah, I'm hoping to see use of Go routines in useful rewrites of projects like rsync with parallel file uploads. It can lead to clean, extensible, and maintainable code with still low overhead and high throughput.
- meruru 7y agoIt's in C again, but since you mentioned rewrites and rsync: https://github.com/kristapsdz/openrsync https://github.com/kristapsdz/openrsync
- andrewchambers 7y agoI did a project a while ago for fun using go to implement an sftp server for a dropbox account. (Though I used openssh for the ssh server). https://github.com/andrewchambers/sftpplease https://github.com/andrewchambers/sftpplease
- linuxdude314 7y agoPart of the utility of SFTP to clients and admins is the fact there are no additional requirements other than SSHD to function. This seems to fly in the face of that. If you are willing to install more software, why not use a more feature filled file server?
- pilif 7y agoOpenSSH is very much tied to system users, but sometimes you might want to give access to external users that don’t need to exist as actual unix users. because SSH and SFTP are so closely tied together, a configuration via PAM is pretty hard and inconvenient because creating fake users via PAM for SFTP will also create them for SSH and because there’s no easy way to map all such virtual users to the same user-id. Also, because OpenSSH has zero support for virtual users, aside of a PAM configuration, you also need an NSS configuration and now all your virtual users in some database have suddenly become system users on your box. SFTP as a protocol on the other hand is very convenient over, say FTP over TLS because it’s using a single TCP port and it has been created this century. So having this self-contained project is useful when you need to allow third parties access to files but you also don’t want to create system users for them or risk f’ing something up with PAM
- jasonjayr 7y agoI'd second this, I've longed struggled to come up with a easy-to-deploy sshd/sftp/chroot configuration that permitted easy database-driven configuration w/o extra shell access. You have to fight a lot of defaults to get this just right. Would the OpenSSH upstream accept patches for an unprivileged sshd/sftp-subsystem to make this easier to use their battle tested code?
- Nux 7y agoProftpd can do sftp-only out of the box.
- jasonjayr 7y agoI am not sure why I didn't realize that sooner, but that's awesome, thank you for pointing that out.
- pilif 7y agoon the other hand, its SQL backend modules insist on fetching a password and comparing them using their own functionality and there's no way to get to the the user's typed password into a custom query. That means that there's no way to authenticate users if you use a password encryption scheme not supported by the bundled modules (like bcrypt for example)
- sandreas 7y agoIf you just want to share files and not giving the option to upload files, you could try graft... https://github.com/sandreas/graft https://github.com/sandreas/graft graft serve ./*.txt will serve every txt file in the current dir...
- yjftsjthsd-h 7y agoFor that use case, I just use `python3 -m http.server` (or the Python 2 equivalent), especially since Python is almost always already installed.
- paulddraper 7y agoWhat do you recommend? Proftpd? A lot of more advanced servers are either (1) proprietary (2) unmaintained (3) hard to use (4) require a Linux VM or all of the above.
- Mayzie 7y agoTo be fair, it is written in Go - a language where a lot of projects built can just be single, portable binaries with no dependencies (as this project appears to be).
- dvfjsdhgfv 7y agoWith this solution you can easily create fake account just for sftp, and the installation is very simple. In other words, it solves the biggest security problem with sftp accounts - what you usually want is only to give people access to a few files, and not to give them full system accounts.
- lkschubert8 7y agoI actually had to implement something very similar. The use case was an SFTP upload interface for clients that they would use the same login credentials they use for the web portal to upload to unique subfolders in Azure Storage.
- NERDiT 7y agoI agree, unless some robust solution is needed - I prefer just the daemon and using the CLI to push/pull content
- colek42 7y agoA lot of the world runs on sftp. The flexible user management in this code would have saved my team about a month of time implimenting our own solution.
- nickcw 7y agoThis looks like a very interesting project to make a lightweight sftp server, though the dependency on an SQL db makes it a bit harder to install. Go has a really good set of crypto primitives, an excellent ssh implementation maintained by the core team, and a great sftp server and client library which makes this project possible. I recently added an sftp server to rclone: https://rclone.org/commands/rclone_serve_sftp/ https://rclone.org/commands/rclone_serve_sftp/ - this can serve any of the cloud providers rclone support as sftp (or local disk). This runs on windows/macOS/linux too. It was a joy to add this as the Go libraries are very well thought out and easy to use.
- edwinyzh 7y agoNot sure the case in this context, but usually the SQLite dependency can almost be seen as no-dependency at all - it's just a file on the disk without any external dependency.
- rakoo 7y agoOn the other hand it requires a C compiler, so it's definitely heavier than a pure go application
- jlubawy 7y agoSerious question, why hasn't anyone rewritten it in Go? Been awhile since I've looked at it but C to Go doesn't seem that difficult to translate. Edit: just looked at the amalgamation again, it's huge... Still surprised no one has tried it yet that I know of. Pre-processor directives would probably make it even more difficult though. Edit2: and it already works so what's the point, other than an extremely challenging problem
- makapuf 7y agoSqlite databases are often not shared between applications. Do you know a similar sql, in process database in pure go (even if not completely similar feature wise ?)
- heinrichhartman 7y agoCan this project be used as a library and integrated into an existing program? I am asking, b/c I am currently working on a little document management system for home-use. It looks like the only sane way to integrate with document scanners is (S)FTP upload or Email (SMPT). I have tried to use off-the-shelf FTP servers for this and inotify to get notified of new uploads. The the solutions work OK, but are hard to setup and rather brittle (and limited to Linux). Go seems to be the ideal language for this project, because its concurrency model and the ease of distribution (as single binary files). - Has anyone here experience with building systems like this (integrating via FTP/SMTP)? - Any recommendations for languages and libraries?
- quickquestion42 7y agoThe linked project seems to be a wrapper around https://github.com/pkg/sftp https://github.com/pkg/sftp
- heinrichhartman 7y agoAh Thx! Should have checked the code myself. The library looks good. Not too much documentation (https://github.com/pkg/sftp/issues/267 https://github.com/pkg/sftp/issues/267, https://github.com/pkg/sftp/issues/247 https://github.com/pkg/sftp/issues/247), but straight forward and battle tested.
- mook 7y agoYou might need to double check your requirements — SFTP is a very different protocol from FTP/FTPS. Using SFTP for something like document scanners without strong Unix roots would be surprising to me, at least.
- drondin 7y agoI was looking for something like this when I was asked to get an FTP working on kubernetes... Don't ask why...
- jhoechtl 7y agoUsing JSON for configuration: Fail
- microcolonel 7y agoGo is a very convenient option of you have an API agreement based on dropping files on an SFTP.
- maxekman 7y agoI’m currently architecting a transport administration system written fully in Go which includes XMLEDI over SFTP; this project is perfect as we need to integrate with the current IAM.
- deedubaya 7y agoI've been avoiding starting a project myself which will require SFTP. Passing text files around via SFTP is how a lot of the world works. It's still painful to implement this, and makes the transfer of data slow (polling for new files to process etc). This introduces a big delay in any data feedback cycles. Managing users with system accounts is painful too. This project looks to alleviate a lot of these problems. It'd be great to be able to register webhooks so that it could send events to external systems. Ideally I'd like to know when a file is created/deleted etc without having to walk directories on the sftp server on a regular basis.
- drakkan 7y agoEvents are now supported, you can configure custom system commands and/or HTTP notifications on SFTP upload, download, delete or rename
- paulddraper 7y agoIf you use AWS, there's a managed S3 SFTP service. Solves the availability and notification problems. They way overcharge though. ($300/month)
- drinane 7y agojust drop files with guids into stage dirs and send all staged files over to another stage dir on the other side. When successfullly archive on both ends of the transfer.
- diroussel 7y agoYes, if you can avoid polling by getting events it can be a good system. But really you want to be notified when a file is closed successfully, not closed after a timeout of other error. For me, in a previous system I uploaded as a .tmp file, then the uploaded renames to .zip at the end of the upload. The rename is atomic so no chance of the consumer reading a partial file. I was dealing in CSV, TSV and XML files. I chose to upload them all in a zip wrapper not only because they compress well, but also because each file has a CRC and the file won’t be extracted if the file is damaged or truncated. It’s hard to tell if a raw CSV file has been truncated.