4 ms·

I want to store a few dozen TB of genomics data in a publicly accessible way. Are there any better alternatives to S3 or Google Cloud Storage? I've been waiting

I want to store a few dozen TB of genomics data in a publicly accessible way. Are there any better alternatives to S3 or Google Cloud Storage? I've been waiting for Cloudflare's R2 but the beta is not yet open yet and I'm not even sure it would work well for me.

foota 5y ago
You could use requested pays with s3 or GCP, which requires the reader to use their own project and pay for requests.
chockchocschoir 5y ago
Better in terms of what? There are many variables to consider, and some might be more important than others in your case. If you "cost" is the top priority, get a dedicated instance with unmetered connection and a 1 or 10gbps port, then you'll have a static price/month that won't surprise you. If latency/bandwidth is more important, throw a CDN in front of that instance, but price will vary more then as you'll pay per data served (in most cases).
AdrienPoupa 5y ago
Wasabi [1] is admittedly up to 80% cheaper than S3. But it forces you to keep your files for 90 days at least. I saw it recommended several times as a cheaper alternative to S3-backed storage for OwnCloud. [1] https://wasabi.com/ https://wasabi.com/
- kondro 5y ago
  Wasabi's policy prevents reading more than you write. They're designed for backup, not publication.
thallium205 5y ago
Spin up a GSuite Enterprise account and drop it all in Drive. Unlimited storage in Drive for enterprise accounts.
- iskander 5y ago
  They really don't cut you off if you put 10TB of FASTQs in there and let strangers download them?
Dylan16807 5y ago
If you just want to beat the big providers, use B2. If you expect to use many terabytes of bandwidth, then it's time to rent a server or VPS and probably set up a torrent too.
- welkie 5y ago
  How about B2 combined with Cloudflare? Egress from B2 to Cloudflare is free and egress from Cloudflare to the internet is free for their free plan, and perhaps competitive on their not free plan?
  - Dylan16807 5y ago
    Maybe! You'd need to use workers for giant blobs of data, not their normal website plans, but those don't cost a lot. It's probably worth setting up a cheap torrent-seeding server too.
Quokkacalypse 5y ago
There's always ftp.

foota 5y ago

You could use requested pays with s3 or GCP, which requires the reader to use their own project and pay for requests.

chockchocschoir 5y ago

Better in terms of what? There are many variables to consider, and some might be more important than others in your case. If you "cost" is the top priority, get a dedicated instance with unmetered connection and a 1 or 10gbps port, then you'll have a static price/month that won't surprise you. If latency/bandwidth is more important, throw a CDN in front of that instance, but price will vary more then as you'll pay per data served (in most cases).

AdrienPoupa 5y ago

Wasabi [1] is admittedly up to 80% cheaper than S3. But it forces you to keep your files for 90 days at least. I saw it recommended several times as a cheaper alternative to S3-backed storage for OwnCloud. [1] https://wasabi.com/ https://wasabi.com/

kondro 5y ago
Wasabi's policy prevents reading more than you write. They're designed for backup, not publication.

thallium205 5y ago

Spin up a GSuite Enterprise account and drop it all in Drive. Unlimited storage in Drive for enterprise accounts.

iskander 5y ago
They really don't cut you off if you put 10TB of FASTQs in there and let strangers download them?

Dylan16807 5y ago

If you just want to beat the big providers, use B2. If you expect to use many terabytes of bandwidth, then it's time to rent a server or VPS and probably set up a torrent too.

welkie 5y ago
How about B2 combined with Cloudflare? Egress from B2 to Cloudflare is free and egress from Cloudflare to the internet is free for their free plan, and perhaps competitive on their not free plan?
- Dylan16807 5y ago
  Maybe! You'd need to use workers for giant blobs of data, not their normal website plans, but those don't cost a lot. It's probably worth setting up a cheap torrent-seeding server too.

Quokkacalypse 5y ago

There's always ftp.