7 ms·

Show HN: I built open source file sharing solution using AWS S3

63 points by rohitghumare 2y ago

I created a 100% Open source Company-wide Self-hosted File Sharing Solution for Teams Recently, I wanted to share HD images and video files with my graphic designer. She’s exceptional at her craft but isn’t familiar with AWS S3 So, I got an idea and built this. Github Repo: https://github.com/rohitg00/s3-file-share-for-free https://github.com/rohitg00/s3-file-share-for-free Detailed Guide: https://ghumare64.medium.com/i-built-a-company-wide-self-hosted-file-sharing-solution-for-teams-186dbf688de5 https://ghumare64.medium.com/i-built-a-company-wide-self-hos...

szszrk 2y ago
Hey, looks interesting and practical. Any plans on making it compatible with other s3 implementation (other cloud vendors, local minio.io etc)?
- rohitghumare 2y ago
  Yes! Will release next version which supports all cloud providers from S3 to Hetzner Cloud storage!!
  - szszrk 2y ago
    That is really neat. I had so many tiny use cases in my previous companies that could be solved by just a simple UI like this one. If you combine that with things like s3 static websites, it could be a beast that replaces some long-forgotten CMS solutions.
    - rohitghumare 2y ago
      Completely Agree. I would appreciate it if you could add your ideas as an issue here: https://github.com/rohitg00/s3-file-share-for-free/issues https://github.com/rohitg00/s3-file-share-for-free/issues
7bit 2y ago
Why should I use your solution and not Next cloud, for example?
- rohitghumare 2y ago
  Our S3 File Manager offers a lightweight, zero-configuration solution focused purely on S3 storage management, making it ideal for teams who need simple, cost-effective cloud storage without the overhead of a full collaboration suite like Nextcloud. While Nextcloud excels at comprehensive collaboration, our tool excels at simplicity and AWS integration. But we don't want to limit it to S3. We will try to include all storage by next month.
billev2k 2y ago
I'm pretty leery of making the "access key" and "secret key" so public (like typing them into a web page, or setting them in environment variables). Of course it adds significant friction to set up an IAM identity for every user, and "low friction" is one of the key requirements here.
- ryanianian 2y ago
  A "correct" implementation would give you a temporary IAM role or something (STS) based on a JWT or other authn mechanism. This is not that difficult if you're already invested in an identity ecosystem, but a right pain without something to bootstrap it. On the plus side, AWS creds can be made to be temporary and limited in scope to just the nouns/verbs required. Creating and vending those tokens is an exercise for the reader.
  - 420official 2y ago
    It really isn't that challenging to get going with JWT auth in AWS. Gitlab has pretty good documentation for how to use Gitlab ID tokens to assume roles that includes everything other than how to generate a JWT here: https://docs.gitlab.com/ee/ci/cloud_services/aws/ https://docs.gitlab.com/ee/ci/cloud_services/aws/ And of course generating OIDC PKI JWTs is pretty easy and well documented elsewhere. The harder parts in my mind are: - Updating this OSS project to serve a JWK from OIDC .well-known - Convincing people that this method of authn is safe and that those keys are securely stored
    - rohitghumare 2y ago
      I completely Agree on this point. I have this in mind for implementation. For now, I'm focusing on bringing more cloud providers.
bhawks 2y ago
Magic Wormhole gives you secure file transmission for free. Give one of the implementations a try: https://github.com/psanford/wormhole-william https://github.com/psanford/wormhole-william
- ryanianian 2y ago
  Magic wormhole is great for live, peer-to-peer transfers. But it is not great if you want to distribute a file multiple times, asynchronously, or with other functionality gained from centralized storage. This is where people typically use email, dropbox, or perhaps the tool from TFA.
  - nijave 2y ago
    I think you can seed torrents via HTTP but you'd want to add a layer of file encryption in case someone discovers your torrent Non-free Resilio Sync basically automated this
- zikduruqe 2y ago
  Why not S3 presigned URLs? It's already baked into the service anyways. https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareO...
  - gregw2 2y ago
    I agree with other posters that long lived non-autorotating IAM/S3 secret keys are not a good idea. A common alternate approach is presigned URLs. And not just on S3. Google Cloud, Backblaze, Digital Ocean, Cloudflare, Azure all have this presigned URLs functionality too (I checked for the degree of lockin before I started using S3's presigned URLs in a set of bulk-data APIs at one place I've worked.) GCP: https://cloud.google.com/storage/docs/access-control/signed-urls https://cloud.google.com/storage/docs/access-control/signed-... Digital Ocean: https://docs.digitalocean.com/products/spaces/how-to/set-file-permissions/ https://docs.digitalocean.com/products/spaces/how-to/set-fil... Hetzner: https://docs.hetzner.com/storage/object-storage/faq/buckets-objects/ https://docs.hetzner.com/storage/object-storage/faq/buckets-... Backblaze: https://backblaze-prod.us.document360.io/apidocs/b2-get-download-authorization https://backblaze-prod.us.document360.io/apidocs/b2-get-down... Cloudflare R2: https://developers.cloudflare.com/r2/api/s3/presigned-urls/ https://developers.cloudflare.com/r2/api/s3/presigned-urls/ Azure: https://learn.microsoft.com/en-us/rest/api/storageservices/delegate-access-with-shared-access-signature https://learn.microsoft.com/en-us/rest/api/storageservices/d... etc. Also, presigned URLs can be used not just for downloading files with a temporary URL but can also be created for uploading files with a temporary URL.
jatins 2y ago
might be fine for internal company use but pasting access/secret key on a third party website will get you a call from security...or worse, won't
- lizzas 2y ago
  My initial though. At least create an IAM user per file :-). Maybe that defeats the convenience. Cloning and installing is also an option.
  - rohitghumare 2y ago
    I can add this option with just a few changes in a code, but how can it be easy to use for any team?
    - lizzas 2y ago
      I am not an IAM expert but maybe the app should have an admin login that sets the IAM user with full permissions on any s3 bucket(s) needed for the app to work. There should be instructions on how to set that IAM user up (dont make it the root! It just needs full access to a single bucket ideally).
perching_aix 2y ago
I guess this is for smaller organizations with no MS365 subscription (and thus access to SharePoint)?
- justmarc 2y ago
  Does MS365 cover all potential use cases, needs and scenarios?
  - perching_aix 2y ago
    All in the world? I suppose I haven't tried it for recipes, but I guess Copilot could help with that too. As far as sharing files goes though, yes. I mean, it allows you to... share files, and do so in a controlled manner. Even edit them in-app as long as they're of a supported format.
lu1spd 2y ago
[dead]
mickael-kerjean 2y ago
Hi! I’m the author of another open-source project in the same space that I’ve been working on for a while: Filestash [1] Were you familiar with Filestash before starting this? If so, was there something specific you felt was missing that inspired you to create your own solution? Would love to hear your thoughts. github: https://github.com/mickael-kerjean/filestash https://github.com/mickael-kerjean/filestash demo on s3: https://demo.filestash.app/login?type=s3&access_key_id=Q3AM3UQ867SPQQA43P2F&secret_access_key=zuf%2BtfteSlswRu7BJ86wekitnifILbZam1KYY3TG&region=us-east-2&encryption_key=&endpoint=https%3A%2F%2Fplay.minio.io%3A9000%2F https://demo.filestash.app/login?type=s3&access_key_id=Q3AM3...
- deleted 2y ago
  [deleted]
- rohitghumare 2y ago
  Never came across this, Interesting project. Thanks for sharing
- TwoNineFive 2y ago
  Your reply comes off as arrogant and hijacking someone's announcement like this is tacky. "Were you aware I had already done this better. Why would you do something I already did? Did I mention I already did this?"
deleted 2y ago
[deleted]
scarface_74 2y ago
This seems like such a horrible, insecure idea that would never pass muster at any company I’ve ever worked at
- woranl 2y ago
  If you have a dedicated bucket just for this and the authentication is just for that bucket, then is it still considered insecure?
  - scarface_74 2y ago
    Even having long lived access keys and secret keys anywhere is insecure. There really isn’t a need for them. If you are running the application on any AWS compute - EC2, Lambda, ECS, EKS, etc, there is an IAM role attached to the VM that gives code permission to run. On the client facing side, it should be connected to your Orgs SSO solution so when a person leaves the company, you deactivate the user in one place. Besides, I can’t think of any organization of even 2 people that isn’t already using Office365/OneDrive or Google/GSuite with plenty of shared storage. The cost per seat for either is $6 - $25 per user.
lomkju 2y ago
Nice! Could you tell me why Google Drive didn’t work for you? Google Drive seems to cheaper and has better UX than S3 - $8 for 2 TB (India) - Supports file versioning - 750GB bandwidth per day
- folmar 2y ago
  If you do any less typical files Google Drive will balk at you at all times, at least for the other users who use the browser UI. What works bad: * folder with lots of medium size files * large ZIP * video bigger than a few tens of megabytes
rohitghumare 2y ago
Quick Update: I didn't expect so many people to show interest in this project. As a few suggested, I plan to add more storage solutions in the next few days. Testing Other Cloud Providers: Backblaze B2 ($5/TB/month) Wasabi ($6/TB/month) Google Cloud Storage ($20/TB/month) - I am ready to deploy to production. DigitalOcean Spaces ($5/TB/month) Cloudflare R2 ($15/TB/month) Hetzner Storage ($3/TB/month)
iJohnDoe 2y ago
I obviously wouldn’t put my key into a third-party site. However, is there any concern with the self-hosted solution? Looks pretty cool and wouldn’t mind using it. Just make a key specifically for this?
- rohitghumare 2y ago
  Yes, feel free to try open-source version available on Github. Just published a new release to support multiple cloud storage options. Thanks
up2isomorphism 2y ago
You guys waste too much time building “open” stuff on S3.
- nobodywillobsrv 2y ago
  This is what I was wondering ... why start with s3? Why not "simply" build on IPFS or something like that as a start. Even a shared syncthing key would be MVP.
rohitghumare 2y ago
New Release is out now > We support 7 cloud providers now. > Beta storage providers are still being tested. Stable: Amazon S3 ($23/TB/month) Google Cloud Storage ($20/TB/month) Cloudflare R2 ($15/TB/month) Beta/Testing: Wasabi ($6.99/TB/month) Backblaze B2 ($6/TB/month) DigitalOcean Spaces ($5/TB/month) Hetzner Storage ($4/TB/month)

63 points by rohitghumare 2y ago

szszrk 2y ago

Hey, looks interesting and practical. Any plans on making it compatible with other s3 implementation (other cloud vendors, local minio.io etc)?

rohitghumare 2y ago
Yes! Will release next version which supports all cloud providers from S3 to Hetzner Cloud storage!!
- szszrk 2y ago
  That is really neat. I had so many tiny use cases in my previous companies that could be solved by just a simple UI like this one. If you combine that with things like s3 static websites, it could be a beast that replaces some long-forgotten CMS solutions.
  - rohitghumare 2y ago
    Completely Agree. I would appreciate it if you could add your ideas as an issue here: https://github.com/rohitg00/s3-file-share-for-free/issues https://github.com/rohitg00/s3-file-share-for-free/issues

7bit 2y ago

Why should I use your solution and not Next cloud, for example?

rohitghumare 2y ago
Our S3 File Manager offers a lightweight, zero-configuration solution focused purely on S3 storage management, making it ideal for teams who need simple, cost-effective cloud storage without the overhead of a full collaboration suite like Nextcloud. While Nextcloud excels at comprehensive collaboration, our tool excels at simplicity and AWS integration. But we don't want to limit it to S3. We will try to include all storage by next month.

billev2k 2y ago

I'm pretty leery of making the "access key" and "secret key" so public (like typing them into a web page, or setting them in environment variables). Of course it adds significant friction to set up an IAM identity for every user, and "low friction" is one of the key requirements here.

ryanianian 2y ago
A "correct" implementation would give you a temporary IAM role or something (STS) based on a JWT or other authn mechanism. This is not that difficult if you're already invested in an identity ecosystem, but a right pain without something to bootstrap it. On the plus side, AWS creds can be made to be temporary and limited in scope to just the nouns/verbs required. Creating and vending those tokens is an exercise for the reader.
- 420official 2y ago
  It really isn't that challenging to get going with JWT auth in AWS. Gitlab has pretty good documentation for how to use Gitlab ID tokens to assume roles that includes everything other than how to generate a JWT here: https://docs.gitlab.com/ee/ci/cloud_services/aws/ https://docs.gitlab.com/ee/ci/cloud_services/aws/ And of course generating OIDC PKI JWTs is pretty easy and well documented elsewhere. The harder parts in my mind are: - Updating this OSS project to serve a JWK from OIDC .well-known - Convincing people that this method of authn is safe and that those keys are securely stored
  - rohitghumare 2y ago
    I completely Agree on this point. I have this in mind for implementation. For now, I'm focusing on bringing more cloud providers.

bhawks 2y ago

Magic Wormhole gives you secure file transmission for free. Give one of the implementations a try: https://github.com/psanford/wormhole-william https://github.com/psanford/wormhole-william

ryanianian 2y ago
Magic wormhole is great for live, peer-to-peer transfers. But it is not great if you want to distribute a file multiple times, asynchronously, or with other functionality gained from centralized storage. This is where people typically use email, dropbox, or perhaps the tool from TFA.
- nijave 2y ago
  I think you can seed torrents via HTTP but you'd want to add a layer of file encryption in case someone discovers your torrent Non-free Resilio Sync basically automated this
zikduruqe 2y ago
Why not S3 presigned URLs? It's already baked into the service anyways. https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareO...
- gregw2 2y ago
  I agree with other posters that long lived non-autorotating IAM/S3 secret keys are not a good idea. A common alternate approach is presigned URLs. And not just on S3. Google Cloud, Backblaze, Digital Ocean, Cloudflare, Azure all have this presigned URLs functionality too (I checked for the degree of lockin before I started using S3's presigned URLs in a set of bulk-data APIs at one place I've worked.) GCP: https://cloud.google.com/storage/docs/access-control/signed-urls https://cloud.google.com/storage/docs/access-control/signed-... Digital Ocean: https://docs.digitalocean.com/products/spaces/how-to/set-file-permissions/ https://docs.digitalocean.com/products/spaces/how-to/set-fil... Hetzner: https://docs.hetzner.com/storage/object-storage/faq/buckets-objects/ https://docs.hetzner.com/storage/object-storage/faq/buckets-... Backblaze: https://backblaze-prod.us.document360.io/apidocs/b2-get-download-authorization https://backblaze-prod.us.document360.io/apidocs/b2-get-down... Cloudflare R2: https://developers.cloudflare.com/r2/api/s3/presigned-urls/ https://developers.cloudflare.com/r2/api/s3/presigned-urls/ Azure: https://learn.microsoft.com/en-us/rest/api/storageservices/delegate-access-with-shared-access-signature https://learn.microsoft.com/en-us/rest/api/storageservices/d... etc. Also, presigned URLs can be used not just for downloading files with a temporary URL but can also be created for uploading files with a temporary URL.

jatins 2y ago

might be fine for internal company use but pasting access/secret key on a third party website will get you a call from security...or worse, won't

lizzas 2y ago
My initial though. At least create an IAM user per file :-). Maybe that defeats the convenience. Cloning and installing is also an option.
- rohitghumare 2y ago
  I can add this option with just a few changes in a code, but how can it be easy to use for any team?
  - lizzas 2y ago
    I am not an IAM expert but maybe the app should have an admin login that sets the IAM user with full permissions on any s3 bucket(s) needed for the app to work. There should be instructions on how to set that IAM user up (dont make it the root! It just needs full access to a single bucket ideally).

perching_aix 2y ago

I guess this is for smaller organizations with no MS365 subscription (and thus access to SharePoint)?

justmarc 2y ago
Does MS365 cover all potential use cases, needs and scenarios?
- perching_aix 2y ago
  All in the world? I suppose I haven't tried it for recipes, but I guess Copilot could help with that too. As far as sharing files goes though, yes. I mean, it allows you to... share files, and do so in a controlled manner. Even edit them in-app as long as they're of a supported format.

lu1spd 2y ago

[dead]

mickael-kerjean 2y ago

Hi! I’m the author of another open-source project in the same space that I’ve been working on for a while: Filestash [1] Were you familiar with Filestash before starting this? If so, was there something specific you felt was missing that inspired you to create your own solution? Would love to hear your thoughts. github: https://github.com/mickael-kerjean/filestash https://github.com/mickael-kerjean/filestash demo on s3: https://demo.filestash.app/login?type=s3&access_key_id=Q3AM3UQ867SPQQA43P2F&secret_access_key=zuf%2BtfteSlswRu7BJ86wekitnifILbZam1KYY3TG&region=us-east-2&encryption_key=&endpoint=https%3A%2F%2Fplay.minio.io%3A9000%2F https://demo.filestash.app/login?type=s3&access_key_id=Q3AM3...

deleted 2y ago
[deleted]
rohitghumare 2y ago
Never came across this, Interesting project. Thanks for sharing
TwoNineFive 2y ago
Your reply comes off as arrogant and hijacking someone's announcement like this is tacky. "Were you aware I had already done this better. Why would you do something I already did? Did I mention I already did this?"

deleted 2y ago

[deleted]

scarface_74 2y ago

This seems like such a horrible, insecure idea that would never pass muster at any company I’ve ever worked at

woranl 2y ago
If you have a dedicated bucket just for this and the authentication is just for that bucket, then is it still considered insecure?
- scarface_74 2y ago
  Even having long lived access keys and secret keys anywhere is insecure. There really isn’t a need for them. If you are running the application on any AWS compute - EC2, Lambda, ECS, EKS, etc, there is an IAM role attached to the VM that gives code permission to run. On the client facing side, it should be connected to your Orgs SSO solution so when a person leaves the company, you deactivate the user in one place. Besides, I can’t think of any organization of even 2 people that isn’t already using Office365/OneDrive or Google/GSuite with plenty of shared storage. The cost per seat for either is $6 - $25 per user.

lomkju 2y ago

Nice! Could you tell me why Google Drive didn’t work for you? Google Drive seems to cheaper and has better UX than S3 - $8 for 2 TB (India) - Supports file versioning - 750GB bandwidth per day

folmar 2y ago
If you do any less typical files Google Drive will balk at you at all times, at least for the other users who use the browser UI. What works bad: * folder with lots of medium size files * large ZIP * video bigger than a few tens of megabytes

rohitghumare 2y ago

Quick Update: I didn't expect so many people to show interest in this project. As a few suggested, I plan to add more storage solutions in the next few days. Testing Other Cloud Providers: Backblaze B2 ($5/TB/month) Wasabi ($6/TB/month) Google Cloud Storage ($20/TB/month) - I am ready to deploy to production. DigitalOcean Spaces ($5/TB/month) Cloudflare R2 ($15/TB/month) Hetzner Storage ($3/TB/month)

iJohnDoe 2y ago

I obviously wouldn’t put my key into a third-party site. However, is there any concern with the self-hosted solution? Looks pretty cool and wouldn’t mind using it. Just make a key specifically for this?

rohitghumare 2y ago
Yes, feel free to try open-source version available on Github. Just published a new release to support multiple cloud storage options. Thanks

up2isomorphism 2y ago

You guys waste too much time building “open” stuff on S3.

nobodywillobsrv 2y ago
This is what I was wondering ... why start with s3? Why not "simply" build on IPFS or something like that as a start. Even a shared syncthing key would be MVP.

rohitghumare 2y ago

New Release is out now > We support 7 cloud providers now. > Beta storage providers are still being tested. Stable: Amazon S3 ($23/TB/month) Google Cloud Storage ($20/TB/month) Cloudflare R2 ($15/TB/month) Beta/Testing: Wasabi ($6.99/TB/month) Backblaze B2 ($6/TB/month) DigitalOcean Spaces ($5/TB/month) Hetzner Storage ($4/TB/month)