3 ms·

A Portable Cloud Experiment: SFTP Cloud Storage Sync

23 points by luu 8y ago

iampims 8y ago
Relevant: The Law of Leaky Abstractions [1] [1] https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...
stevenringo 8y ago
Have you looked at https://rclone.org/ https://rclone.org/? Also written in Go, and handles more cloud providers (and SFTP) than you can shake a stick at.
- lathiat 8y ago
  +1 for rclone, it is fantastic and I am using it to sync both local filesystems to backblaze b2, as well as directly from a remote SFTP filesystem also to backblaze b2 or local. Have also used it to back up my Google Drive, it exports all of the google documents as like docx or xlsx (or whatever you request). Critical for a small non profit where we use Google Drive to keep all our docs, theres no good easy way to back it up otherwise. It's really a fantastic bit of software. Having said that it seems OPs use case was really to explore "Go Cloud" so it's probably not that helpful input in the context of the actual article, but great context for anyone reading the comments that might actually want such a working solution. One thing I keep getting errors from rclone about that is sortof related to this, is that it can't sync symlinks from SFTP as it doesn't really understand them - something that can be fixed but it's an example of the fact you always have side cases to cover- even though rclone seems to have done a good job at handling many of them.
- d0ugie 8y ago
  I just discovered the joys of sshfs which lets you mount whatever you can ssh/sftp into. What'll they think of next.. https://github.com/libfuse/sshfs https://github.com/libfuse/sshfs
khc 8y ago
It's true that you usually get the lowest common denominator when you build abstractions, but cloud storage has a pretty broad set of useful common denominators. The fact that the author runs into so much trouble says more about the library than the approach.

3 ms·

A Portable Cloud Experiment: SFTP Cloud Storage Sync

23 points by luu 8y ago

iampims 8y ago
Relevant: The Law of Leaky Abstractions [1] [1] https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...
stevenringo 8y ago
Have you looked at https://rclone.org/ https://rclone.org/? Also written in Go, and handles more cloud providers (and SFTP) than you can shake a stick at.
- lathiat 8y ago
  +1 for rclone, it is fantastic and I am using it to sync both local filesystems to backblaze b2, as well as directly from a remote SFTP filesystem also to backblaze b2 or local. Have also used it to back up my Google Drive, it exports all of the google documents as like docx or xlsx (or whatever you request). Critical for a small non profit where we use Google Drive to keep all our docs, theres no good easy way to back it up otherwise. It's really a fantastic bit of software. Having said that it seems OPs use case was really to explore "Go Cloud" so it's probably not that helpful input in the context of the actual article, but great context for anyone reading the comments that might actually want such a working solution. One thing I keep getting errors from rclone about that is sortof related to this, is that it can't sync symlinks from SFTP as it doesn't really understand them - something that can be fixed but it's an example of the fact you always have side cases to cover- even though rclone seems to have done a good job at handling many of them.
- d0ugie 8y ago
  I just discovered the joys of sshfs which lets you mount whatever you can ssh/sftp into. What'll they think of next.. https://github.com/libfuse/sshfs https://github.com/libfuse/sshfs
khc 8y ago
It's true that you usually get the lowest common denominator when you build abstractions, but cloud storage has a pretty broad set of useful common denominators. The fact that the author runs into so much trouble says more about the library than the approach.