4 ms·

If you host large, publicly available data in a cloud blob service, but you don't have a budget for it, one option is to use the "Requester Pays" feature that A

If you host large, publicly available data in a cloud blob service, but you don't have a budget for it, one option is to use the "Requester Pays" feature that Amazon and Google provide. This makes the data available to anyone to download, but they need to pay the download cost themselves. This is at the tradeoff of making your data significantly more irritating to access, as it's no longer just plugging in a URL into a program, plus everyone who wants your dataset needs to set up a billing account with Amazon or Google.

baroffoos 7y ago
Or just post a magnet link.
- CobrastanJorji 7y ago
  Sure, that's a great option for helping to reduce the cost for well-meaning general use, but the other way makes your costs 100% predictable, which is great if you're on an academic budget (but, again, way more annoying for the downloaders unless they're also using AWS).
- delfinom 7y ago
  So if there's no other seeders, you end up eating the full cost as the only seeder....
- fs111 7y ago
  torrents were supported by S3 in the past https://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.html https://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.ht...
- rtkwe 7y ago
  That probably wouldn't work here. That s3 bucket is hosting models downloaded at runtime/startup [1] and looks like under normal runs it would be cached. If this is being used in Apple's CI pipeline though the whole thing is being torn down between builds so every build and test has to fetch it again. [1] https://github.com/huggingface/pytorch-transformers/search?q=s3&unscoped_q=s3 https://github.com/huggingface/pytorch-transformers/search?q...
dx034 7y ago
Or use Cloudflare in front of it, especially with mostly static data.
- lathiat 7y ago
  Cloudflare generally doesn’t want to cache data files on non enterprise plans (the terms generally disallow it). Only websites HTML and related content.

baroffoos 7y ago

Or just post a magnet link.

CobrastanJorji 7y ago
Sure, that's a great option for helping to reduce the cost for well-meaning general use, but the other way makes your costs 100% predictable, which is great if you're on an academic budget (but, again, way more annoying for the downloaders unless they're also using AWS).
delfinom 7y ago
So if there's no other seeders, you end up eating the full cost as the only seeder....
fs111 7y ago
torrents were supported by S3 in the past https://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.html https://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.ht...
rtkwe 7y ago
That probably wouldn't work here. That s3 bucket is hosting models downloaded at runtime/startup [1] and looks like under normal runs it would be cached. If this is being used in Apple's CI pipeline though the whole thing is being torn down between builds so every build and test has to fetch it again. [1] https://github.com/huggingface/pytorch-transformers/search?q=s3&unscoped_q=s3 https://github.com/huggingface/pytorch-transformers/search?q...

dx034 7y ago

Or use Cloudflare in front of it, especially with mostly static data.

lathiat 7y ago
Cloudflare generally doesn’t want to cache data files on non enterprise plans (the terms generally disallow it). Only websites HTML and related content.