4 ms·

The things we could build if S3 specified a simple OAuth2-based protocol for delegating read/write access. The world needs an HTTP-based protocol for apps to ac

The things we could build if S3 specified a simple OAuth2-based protocol for delegating read/write access. The world needs an HTTP-based protocol for apps to access data on the user's behalf. Google Drive is the closest to this but it only has a single provider and other issues[0]. I'm sad remoteStorage never caught on. I really hope Solid does well but it feels too complex to me. My own take on the problem is https://gemdrive.io/ https://gemdrive.io/, but it's mostly on hold while I'm focused on other parts of the self-hosting stack. [0]: https://gdrivemusic.com/help https://gdrivemusic.com/help

Spivak 3y ago
Such a system would be amazing. It would really force companies whose products are UIs on top of S3 to compete hard because adversarial interoperability would be an ever present threat from your competitors. It really is such a shame that all the projects that tried/are trying to create data sovereignty for users became weird crypto.
- ttymck 3y ago
  I agree with both halves of your comment, but I realized I can't identify the connection between S3 oauth and data sovereignty. Could you elaborate?
  - Spivak 3y ago
    So the idea would be that you have an account with AWS (or realistically a more consumer friendly service that's Amazon branded) where all your data lives. Then when you use say Dropbox you can pick "Use my own storage" and grant Dropbox via OAuth the ability to write to /dropbox in your bucket and all your files would live there instead of Dropbox's servers. Lots of the data sovereignty solutions also include a database like interface you can grant apps the ability to use but I can't imagine that catching on initially. Apple actually already does this with iCloud storage but hides it really well so it feels seamless.
    - BHSPitMonkey 3y ago
      Isn't this essentially how the Dropbox API already works (for apps that support using it)? I've used many apps over the years that offer this option alongside some alternatives.
simonw 3y ago
Absolutely this. I would LOVE to be able to build apps that store people's data in their own S3 bucket, billed to their own account. Doing that right now is monumentally difficult. I built an entire CLI app just for solving the "issue AWS credentials that can only access this specific bucket" problem, but I really don't want to have to talk my users through installing and running something like that: https://s3-credentials.readthedocs.io/en/stable/ https://s3-credentials.readthedocs.io/en/stable/
gusmd 3y ago
You can get close with a Cognito Identity Pool that exchanges your user's keys for AWS credentials associated with an IAM role that has access to the resources you want to read/write on their behalf. Pretty standard pattern. https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-identity.html https://docs.aws.amazon.com/cognito/latest/developerguide/co... edit: I think I misread your comment. I understood it as your app wanting to delegate access to a user's data to the client, but it seems like you want the user to delegate access to their own data to your app? Different use-cases.
ent101 3y ago
We're building this at https://puter.com https://puter.com
- anderspitman 3y ago
  You mean you're implementing something like this to be used by puter.com?
jamesblonde 3y ago
Most apps, however, assume POSIX-like data access. I would love to see a client-side minimally dependent library that mounts a local directory that is actually the user's S3 bucket.
- afr0ck 3y ago
  Linux has FUSE, which is a framework to develop user-level filesystems. Mounting S3 buckets is a very good use case. Sshfs and httpfs are more or less similar in this regard.
  - anderspitman 3y ago
    Yep, and WinFSP and dokany are two options for FUSE on Windows. I'd recommend using rclone or maybe check this list: https://winfsp.dev/doc/Known-File-Systems/ https://winfsp.dev/doc/Known-File-Systems/
wizwit999 3y ago
Apache Iceberg is kind of this, but more oriented around large data lake datasets.

Spivak 3y ago

Such a system would be amazing. It would really force companies whose products are UIs on top of S3 to compete hard because adversarial interoperability would be an ever present threat from your competitors. It really is such a shame that all the projects that tried/are trying to create data sovereignty for users became weird crypto.

ttymck 3y ago
I agree with both halves of your comment, but I realized I can't identify the connection between S3 oauth and data sovereignty. Could you elaborate?
- Spivak 3y ago
  So the idea would be that you have an account with AWS (or realistically a more consumer friendly service that's Amazon branded) where all your data lives. Then when you use say Dropbox you can pick "Use my own storage" and grant Dropbox via OAuth the ability to write to /dropbox in your bucket and all your files would live there instead of Dropbox's servers. Lots of the data sovereignty solutions also include a database like interface you can grant apps the ability to use but I can't imagine that catching on initially. Apple actually already does this with iCloud storage but hides it really well so it feels seamless.
  - BHSPitMonkey 3y ago
    Isn't this essentially how the Dropbox API already works (for apps that support using it)? I've used many apps over the years that offer this option alongside some alternatives.

simonw 3y ago

Absolutely this. I would LOVE to be able to build apps that store people's data in their own S3 bucket, billed to their own account. Doing that right now is monumentally difficult. I built an entire CLI app just for solving the "issue AWS credentials that can only access this specific bucket" problem, but I really don't want to have to talk my users through installing and running something like that: https://s3-credentials.readthedocs.io/en/stable/ https://s3-credentials.readthedocs.io/en/stable/

gusmd 3y ago

You can get close with a Cognito Identity Pool that exchanges your user's keys for AWS credentials associated with an IAM role that has access to the resources you want to read/write on their behalf. Pretty standard pattern. https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-identity.html https://docs.aws.amazon.com/cognito/latest/developerguide/co... edit: I think I misread your comment. I understood it as your app wanting to delegate access to a user's data to the client, but it seems like you want the user to delegate access to their own data to your app? Different use-cases.

ent101 3y ago

We're building this at https://puter.com https://puter.com

anderspitman 3y ago
You mean you're implementing something like this to be used by puter.com?

jamesblonde 3y ago

Most apps, however, assume POSIX-like data access. I would love to see a client-side minimally dependent library that mounts a local directory that is actually the user's S3 bucket.

afr0ck 3y ago
Linux has FUSE, which is a framework to develop user-level filesystems. Mounting S3 buckets is a very good use case. Sshfs and httpfs are more or less similar in this regard.
- anderspitman 3y ago
  Yep, and WinFSP and dokany are two options for FUSE on Windows. I'd recommend using rclone or maybe check this list: https://winfsp.dev/doc/Known-File-Systems/ https://winfsp.dev/doc/Known-File-Systems/

wizwit999 3y ago

Apache Iceberg is kind of this, but more oriented around large data lake datasets.