6 ms·

I work in a bioinformatics lab that develops data visualizations for genetics. A typical file for a whole human genome is in the 50-100GB range. These files are

I work in a bioinformatics lab that develops data visualizations for genetics. A typical file for a whole human genome is in the 50-100GB range. These files are heavily indexed and streaming-friendly, but still rather unwieldy to work with. Our most common flow involves files stored on S3. The app will download the index, then use sampling and streaming techniques to pull from the full file. This works well because S3 supports range requests. Where S3 falls down for us is sharing. I'm not aware of any easy way to share S3 files with email. You either have to create a signed URL (which has a max expiration), make the file public, or use IAM, which I'm pretty sure requires all users to have AWS accounts. That's a non-starter. Google Drive is the opposite. It's better for sharing. Just drop in an email address, or share a permanent link. But it no longer supports range requests, so it's essentially walled off from the rest of the web. What's crazy to me is that neither of these services provide a non-authenticated way to use a CLI tool to download/sync a directory tree, even if it's public. rclone is probably the best tool in this space, and it requires you to configure "remotes" ahead of time. You can't just point it at a public bucket and have it download. I think there's still room for improvement and competition in the cloud storage space.

rsync 7y ago
"What's crazy to me is that neither of these services provide a non-authenticated way to use a CLI tool to download/sync a directory tree, even if it's public. rclone is probably the best tool in this space, and it requires you to configure "remotes" ahead of time." It is actually possible to run 'rclone' without configuring a remote ahead of time. It's relatively recent, but if you look at this discussion thread: https://forum.rclone.org/t/can-rclone-be-run-solely-with-command-line-options-no-config-no-env-vars/6314 https://forum.rclone.org/t/can-rclone-be-run-solely-with-com... ... and the later development that was done on this ... you will see that you can run a single 'rclone' command (albeit, a very long and complicated one) that will specify everything that was in your remote configuration. I hope this is helpful ... "I think there's still room for improvement and competition in the cloud storage space." We're trying[1] ... and specifically, with rclone[2] ... [1] https://www.rsync.net/products/universal.html https://www.rsync.net/products/universal.html [2] https://rclone.org/ https://rclone.org/
- anderspitman 7y ago
  That's definitely a step in the right direction for rclone, but I still think the protocols are the problem. Or more precisely, the lack of a protocol designed for sharing data between services. HTTP is good for retrieving, but lacks directories and a standard for uploads. WebDAV is good in theory, but too complicated and implementations aren't compatible enough. I've been playing around with my own take[0], but it's still really early. Also, I think OAuth is too complicated. That's tricky though because most of the features have been added for good reasons. I've been keeping an eye on rsync.net for while. I really like a lot of the things you're doing. I think SSH is pretty much the best way of doing remote file management that we have today. The main problems are it's hard to share stuff and it doesn't work in browsers. [0] https://github.com/remfs/remfs-http-spec https://github.com/remfs/remfs-http-spec
  - rsync 7y ago
    "I've been keeping an eye on rsync.net for while. I really like a lot of the things you're doing." Thank you - appreciated! " think SSH is pretty much the best way of doing remote file management that we have today." I will alert you to the issue of deep inefficiencies in SSH/SFTP over long WAN links. This problem has been solved by HPN-SSH[1] and is likely already built into your OpenSSH distribution, but I mention it here because it is not very well known ... we run into customers doing long, distant (hops) transfers over SSH that we sometimes need to step through the process of updating their ssh so they can negotiate HPN-SSH with us. It's also worth noting that you can configure ssh do use a "null" cipher which might be applicable with borg uploads, bit I digress ... "The main problems are it's hard to share stuff and it doesn't work in browsers." This is true - and by design. There will never be anonymous / non-credentialed access to rsync.net accounts and no rsync.net storage array will ever run an httpd. Our nmap results will always be one line long: TCP22/sshd.
    - anderspitman 7y ago
      > I will alert you to the issue of deep inefficiencies in SSH/SFTP over long WAN links Good to know, thanks! > This is true - and by design. There will never be anonymous / non-credentialed access to rsync.net accounts and no rsync.net storage array will ever run an httpd. Our nmap results will always be one line long: TCP22/sshd. Makes sense from a security standpoint. But I think there's a business opportunity for having cloud storage that allows you to host your files over HTTP. Imagine if google drive was your CMS. Just drag your static site over and it's live. Edit HTML files in-place. Set permissions based on path.
lukeschlather 7y ago
> Google Drive is the opposite. It's better for sharing. Just drop in an email address, or share a permanent link. But it no longer supports range requests, so it's essentially walled off from the rest of the web. What's the difference between a permanent link and simply making the file public? (The email address, that's essentially requiring the person to have a Google account, so that's the same as IAM.)
- anderspitman 7y ago
  Not a huge difference if you add your own obfuscation layer, but the UI and tooling isn't built around that use case. Does Drive require a google account? I haven't tested that. If so that's really sad. I know you can share with non-google email addresses but maybe they have to already be logged in with a google account for it to work.
sirsar 7y ago
Two ideas come to mind: - You can create public pre-signed URLs with an unlimited expiration date using SigV2. I doubt SigV2 will stop working anytime soon. - Or, if you want the link to provide access to anyone, why not make the object public, but put a UUID in the key name to prevent enumeration?
- anderspitman 7y ago
  Never heard of sigv2, but looks like it's currently on its way out? I like the UUID idea. Especially since you can always move the file under a new UUID to revoke previous access. Not sure if there's a way to generate a new link with GDrive.
- yongjik 7y ago
  You have to be careful. By default, the bucket owner pays for outbound traffic, which can get really expensive: $0.09/GB for the first 10TB. (And there's no caching!)
deleted 7y ago
[deleted]
khc 7y ago
https://github.com/kahing/goofys/ https://github.com/kahing/goofys/ can detect public bucket automatically and not require prior setup. https://cloud.google.com/storage/docs/gsutil https://cloud.google.com/storage/docs/gsutil works as well (despite it's from google it works fine with s3) iirc
- anderspitman 7y ago
  Thanks, I'll check those out

rsync 7y ago

"What's crazy to me is that neither of these services provide a non-authenticated way to use a CLI tool to download/sync a directory tree, even if it's public. rclone is probably the best tool in this space, and it requires you to configure "remotes" ahead of time." It is actually possible to run 'rclone' without configuring a remote ahead of time. It's relatively recent, but if you look at this discussion thread: https://forum.rclone.org/t/can-rclone-be-run-solely-with-command-line-options-no-config-no-env-vars/6314 https://forum.rclone.org/t/can-rclone-be-run-solely-with-com... ... and the later development that was done on this ... you will see that you can run a single 'rclone' command (albeit, a very long and complicated one) that will specify everything that was in your remote configuration. I hope this is helpful ... "I think there's still room for improvement and competition in the cloud storage space." We're trying[1] ... and specifically, with rclone[2] ... [1] https://www.rsync.net/products/universal.html https://www.rsync.net/products/universal.html [2] https://rclone.org/ https://rclone.org/

anderspitman 7y ago
That's definitely a step in the right direction for rclone, but I still think the protocols are the problem. Or more precisely, the lack of a protocol designed for sharing data between services. HTTP is good for retrieving, but lacks directories and a standard for uploads. WebDAV is good in theory, but too complicated and implementations aren't compatible enough. I've been playing around with my own take[0], but it's still really early. Also, I think OAuth is too complicated. That's tricky though because most of the features have been added for good reasons. I've been keeping an eye on rsync.net for while. I really like a lot of the things you're doing. I think SSH is pretty much the best way of doing remote file management that we have today. The main problems are it's hard to share stuff and it doesn't work in browsers. [0] https://github.com/remfs/remfs-http-spec https://github.com/remfs/remfs-http-spec
- rsync 7y ago
  "I've been keeping an eye on rsync.net for while. I really like a lot of the things you're doing." Thank you - appreciated! " think SSH is pretty much the best way of doing remote file management that we have today." I will alert you to the issue of deep inefficiencies in SSH/SFTP over long WAN links. This problem has been solved by HPN-SSH[1] and is likely already built into your OpenSSH distribution, but I mention it here because it is not very well known ... we run into customers doing long, distant (hops) transfers over SSH that we sometimes need to step through the process of updating their ssh so they can negotiate HPN-SSH with us. It's also worth noting that you can configure ssh do use a "null" cipher which might be applicable with borg uploads, bit I digress ... "The main problems are it's hard to share stuff and it doesn't work in browsers." This is true - and by design. There will never be anonymous / non-credentialed access to rsync.net accounts and no rsync.net storage array will ever run an httpd. Our nmap results will always be one line long: TCP22/sshd.
  - anderspitman 7y ago
    > I will alert you to the issue of deep inefficiencies in SSH/SFTP over long WAN links Good to know, thanks! > This is true - and by design. There will never be anonymous / non-credentialed access to rsync.net accounts and no rsync.net storage array will ever run an httpd. Our nmap results will always be one line long: TCP22/sshd. Makes sense from a security standpoint. But I think there's a business opportunity for having cloud storage that allows you to host your files over HTTP. Imagine if google drive was your CMS. Just drag your static site over and it's live. Edit HTML files in-place. Set permissions based on path.

lukeschlather 7y ago

> Google Drive is the opposite. It's better for sharing. Just drop in an email address, or share a permanent link. But it no longer supports range requests, so it's essentially walled off from the rest of the web. What's the difference between a permanent link and simply making the file public? (The email address, that's essentially requiring the person to have a Google account, so that's the same as IAM.)

anderspitman 7y ago
Not a huge difference if you add your own obfuscation layer, but the UI and tooling isn't built around that use case. Does Drive require a google account? I haven't tested that. If so that's really sad. I know you can share with non-google email addresses but maybe they have to already be logged in with a google account for it to work.

sirsar 7y ago

Two ideas come to mind: - You can create public pre-signed URLs with an unlimited expiration date using SigV2. I doubt SigV2 will stop working anytime soon. - Or, if you want the link to provide access to anyone, why not make the object public, but put a UUID in the key name to prevent enumeration?

anderspitman 7y ago
Never heard of sigv2, but looks like it's currently on its way out? I like the UUID idea. Especially since you can always move the file under a new UUID to revoke previous access. Not sure if there's a way to generate a new link with GDrive.
yongjik 7y ago
You have to be careful. By default, the bucket owner pays for outbound traffic, which can get really expensive: $0.09/GB for the first 10TB. (And there's no caching!)

deleted 7y ago

[deleted]

khc 7y ago

https://github.com/kahing/goofys/ https://github.com/kahing/goofys/ can detect public bucket automatically and not require prior setup. https://cloud.google.com/storage/docs/gsutil https://cloud.google.com/storage/docs/gsutil works as well (despite it's from google it works fine with s3) iirc

anderspitman 7y ago
Thanks, I'll check those out