3 ms·
Ask HN: Cheapest way to host and serve large-ish files?
What options exist for serving large files on the web without getting killed on bandwidth costs?
For example, let's say I want to host a 5GB file that contains an educational ML data set on S3. Assuming 1000 people download it in a month, that would cost me $450/mo in just outbound bandwidth fees. That's almost $0.50 per download.
Obviously there must be ways to serve files cheaper than physically mailing an SD card to someone. What have you done to serve large files cheaply? Is the only option to maintain a server (either colo or virtual) that has better bandwidth pricing?
- icebraining 9y agoWell, there's Backblaze B2, it would only cost $0.10 per download: https://www.backblaze.com/b2/cloud-storage.html https://www.backblaze.com/b2/cloud-storage.html Alternatively, there's bittorrent, which was pretty much designed for this. But considered you can find decently priced dedicated servers with unmetered gigabit ports, seems like installing Linux + nginx and enabling automatic security updates might still be the best option.
- weinzierl 9y agoAmazon has Public Data Sets on AWS. I don't know the current status, but it used to be free for eligible projects [1]. BitTorrent would be the natural thing to do, but it has it's own set of issues. Jeff Atwood summarized them nicely in [2]. This was written in 2009, LegalTorrents doesn't exist anymore and you don't need a tracker anymore, but the rest of the points are still valid. His summary is: > It's ironic that BitTorrent, which is justly famous for equalizing the highly asymmetric nature of most people's internet connections, is itself so asymmetric when it comes to sharing: trivially easy to consume, but awkward and confusing to share. That's too bad, because BitTorrent is such a powerful tool for sharing. Hopefully this post demystifies the process a bit! [1] http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1232302 http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-ne... [2] https://blog.codinghorror.com/sharing-files-with-bittorrent/ https://blog.codinghorror.com/sharing-files-with-bittorrent/ EDIT: Thinking about it, there is no shortage of free public data sets. I believe many of them are provided by individuals and organizations which wouldn't be able or willing to pay the bandwidth cost out of their own pockets. Maybe you could research the solutions they are using. A starting point could be the [3]. [3] https://news.ycombinator.com/item?id=2165497 https://news.ycombinator.com/item?id=2165497
- ageitgey 9y agoI have used Internet Archive in the past for this kind of hosting. They charge nothing and provide a great (though sometimes confusing) service. But I'm all for paying my own bills as long as they are reasonable. It seems crazy that IA can afford to host huge files for free for anyone but S3 charges so much for so little. I was hoping there was a middle ground. I guess the middle ground is just running my own dedicated server. Thanks for the reply!
- mozumder 9y agoStatic business home IP address from your ISP and your own server.
- lovelearning 9y ago1 TB storage on Google Drive is $10 (100GB is $2). No bandwidth pricing.
- sixhobbits 9y agoTo add to the other comments about using bittorrent, Academic Torrents [0] is trying to solve this problem. I don't think they're that popular yet, but they aim to address some of the issues with bittorrent already highlighted in the other posts. [0] http://academictorrents.com/ http://academictorrents.com/
- ieee8023 9y agoI think averaging 2TB of data served a day is popular!