3 ms·

I've often felt we need an abstraction for just this. "Bring your own storage" so that you can sign up and provide a "bucket", then the service will read/write

I've often felt we need an abstraction for just this. "Bring your own storage" so that you can sign up and provide a "bucket", then the service will read/write to that. I think the difficulty lies in all the discrepancies between different storage mechanisms and normalizing the SLA there. If Dropbox is down, and your user is on Dropbox, you'll need a way to communicate that their particular "storage" is offline. The trade off is likely worth it - that is being in full control of your data and having a data export "for free" is worth it to a lot of folks. OpenDAL was on HN recently and would be a pretty decent abstraction to use for this: https://github.com/apache/incubator-opendal https://github.com/apache/incubator-opendal

pradn 3y ago
There's a bunch of S3-compatible bucket stores. I suppose one could ask their user for their S3 bucket key. It should be possible for an S3-like service to provide one-bucket-only keys w/ rate limits. The users knows where their data is and exactly when its being read/written. And when done using the app, the user can even revoke access. Of course, having users manage a storage bucket and its key is a bit tedious. Browsers could generate a per-app bucket and provide the bucket key to the app, all with user permission (like when a site asks for location permissions). These buckets can be shared between apps too. As long as apps can read the same data formats, data portability is stellar. It would be great to have an ecosystem of apps that all use compatible buckets/formats. But this is unlikely to happen bc this sort of transparency and data portability isn't currently a big demand by users / governments. Privacy activists have gotten a great number of wins over the past decade. We need to make progress in many more computing issues.
mickael-kerjean 3y ago
> I've often felt we need an abstraction for just this. "Bring your own storage" I made exactly this: https://github.com/mickael-kerjean/filestash https://github.com/mickael-kerjean/filestash The public API enables programmatical access to S3, SFTP, FTP, GIT, WebDav, Samba, Local FS, NFS, Backblaze, Storj, Artifactory, .... and more. Under the hood, each storage implements this interface: type IBackend interface { Ls(path string) ([]os.FileInfo, error) Cat(path string) (io.ReadCloser, error) Mkdir(path string) error Rm(path string) error Mv(from string, to string) error Save(path string, file io.Reader) error Touch(path string) error } There's some hook so you can define your own and do even more funky stuff like virtual FS wich compose onto anything you want/need. There's a bunch of weird implementation like the mysql storage, which represent first level folders as databases, second level folders as tables and files as the actual rows. fun fact: it was born out of a reflection from the infamous top comment of the Dropbox launch on HN
- freedomben 3y ago
  Whoa, this is a solution I've looked for several times and can never find. Thanks!
spencerflem 3y ago
RemoteStorage https://remotestorage.io/ https://remotestorage.io/ seems to be trying to do this too I also really like the https://sandstorm.io https://sandstorm.io approach which goes a little farther beyond
daveguy 3y ago
https://rclone.org https://rclone.org is a very effective storage shim. You'd still want to limit supported backends to provide a smooth UX. There are binaries for the major OSs (including BSDs, Plan9, and Solaris). They have Android builds but they are not first class (yet). Unfortunately I do not see any support for iOS.
- whalesalad 3y ago
  Good for sync but not at all the ask here
networked 3y ago
> I've often felt we need an abstraction for just this. "Bring your own storage" so that you can sign up and provide a "bucket", then the service will read/write to that. The remoteStorage protocol is one: https://remotestorage.io/ https://remotestorage.io/. Sadly, it seems remoteStorage never took off. The first version of the spec was published in 2011, and the reference implementation reached version 1.0 in 2017, but if you look at the list of known apps that support it (https://remotestorage.io/apps/ https://remotestorage.io/apps/), there are few active projects.

pradn 3y ago

There's a bunch of S3-compatible bucket stores. I suppose one could ask their user for their S3 bucket key. It should be possible for an S3-like service to provide one-bucket-only keys w/ rate limits. The users knows where their data is and exactly when its being read/written. And when done using the app, the user can even revoke access. Of course, having users manage a storage bucket and its key is a bit tedious. Browsers could generate a per-app bucket and provide the bucket key to the app, all with user permission (like when a site asks for location permissions). These buckets can be shared between apps too. As long as apps can read the same data formats, data portability is stellar. It would be great to have an ecosystem of apps that all use compatible buckets/formats. But this is unlikely to happen bc this sort of transparency and data portability isn't currently a big demand by users / governments. Privacy activists have gotten a great number of wins over the past decade. We need to make progress in many more computing issues.

mickael-kerjean 3y ago

> I've often felt we need an abstraction for just this. "Bring your own storage" I made exactly this: https://github.com/mickael-kerjean/filestash https://github.com/mickael-kerjean/filestash The public API enables programmatical access to S3, SFTP, FTP, GIT, WebDav, Samba, Local FS, NFS, Backblaze, Storj, Artifactory, .... and more. Under the hood, each storage implements this interface: type IBackend interface { Ls(path string) ([]os.FileInfo, error) Cat(path string) (io.ReadCloser, error) Mkdir(path string) error Rm(path string) error Mv(from string, to string) error Save(path string, file io.Reader) error Touch(path string) error } There's some hook so you can define your own and do even more funky stuff like virtual FS wich compose onto anything you want/need. There's a bunch of weird implementation like the mysql storage, which represent first level folders as databases, second level folders as tables and files as the actual rows. fun fact: it was born out of a reflection from the infamous top comment of the Dropbox launch on HN

freedomben 3y ago
Whoa, this is a solution I've looked for several times and can never find. Thanks!

spencerflem 3y ago

RemoteStorage https://remotestorage.io/ https://remotestorage.io/ seems to be trying to do this too I also really like the https://sandstorm.io https://sandstorm.io approach which goes a little farther beyond

daveguy 3y ago

https://rclone.org https://rclone.org is a very effective storage shim. You'd still want to limit supported backends to provide a smooth UX. There are binaries for the major OSs (including BSDs, Plan9, and Solaris). They have Android builds but they are not first class (yet). Unfortunately I do not see any support for iOS.

whalesalad 3y ago
Good for sync but not at all the ask here

networked 3y ago

> I've often felt we need an abstraction for just this. "Bring your own storage" so that you can sign up and provide a "bucket", then the service will read/write to that. The remoteStorage protocol is one: https://remotestorage.io/ https://remotestorage.io/. Sadly, it seems remoteStorage never took off. The first version of the spec was published in 2011, and the reference implementation reached version 1.0 in 2017, but if you look at the list of known apps that support it (https://remotestorage.io/apps/ https://remotestorage.io/apps/), there are few active projects.