5 ms·

Blockchain storage is going to cost you a pretty penny if you were to store all of Common Crawls pentabytes, so not very feasible.

psKama 6y ago
That's not correct. When it comes to storage and transfer, blockchain alternatives are fraction of the cost of Amazon. For example Sia Skynet is offering $5/month/TB[1] storage. If you skip Skynet and run your own Sia node the price can even go lower to $2/month/TB basing on the market conditions. [1] https://blog.sia.tech/announcing-skynet-premium-plans-faster-speeds-and-longer-pinning-b5469814d2c3 https://blog.sia.tech/announcing-skynet-premium-plans-faster...
- coder543 6y ago
  Amazon is hosting the Common Crawl on S3 for free, so... yes, $2/month/TB is a lot more expensive.
- gillesjacobs 6y ago
  It seems that at least on Sia's plans, you can maximally host 20TB for 80$/month, not even a tenth of a monthly common crawl. Of course Sia's Skynet are package deals right now and I guess they're currently bootstrapping the network with users. Filecoin has no operational storage yet. Storj quotes 10$/Terabyte/month [1] so that will come out expensive. 1. https://www.storj.io/blog/2019/11/announcing-pioneer-2-and-tardigrade.io-pricing/ https://www.storj.io/blog/2019/11/announcing-pioneer-2-and-t...
gloriousternary 6y ago
Moreso than Amazon? From my (limited) experience blockchain storage solutions are often less expensive, although I've never worked with petabytes of data so maybe it's different on that scale.
kevincox 6y ago
IPFS doesn't require any blockchain. In fact someone could easily setup some IPFS nodes that fetch the data from the current host if requested over IPFS.[1] This way people could access it via IPFS and provide an alternate mirror of the data. [1] https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#ipfs-urlstore https://github.com/ipfs/go-ipfs/blob/master/docs/experimenta... The main benefits here would be - Even if the source is unavailable there may be other copies on IPFS which would be transparently used. - There may be some performance benefits in rare cases. - If you are accessing this on a bunch of machines your IPFS gateways would handle downloading the source once, then automatically using the local copy from inside your network. The maindownside is that if Amazon is donating their resources why bother with IPFS?

psKama 6y ago

That's not correct. When it comes to storage and transfer, blockchain alternatives are fraction of the cost of Amazon. For example Sia Skynet is offering $5/month/TB[1] storage. If you skip Skynet and run your own Sia node the price can even go lower to $2/month/TB basing on the market conditions. [1] https://blog.sia.tech/announcing-skynet-premium-plans-faster-speeds-and-longer-pinning-b5469814d2c3 https://blog.sia.tech/announcing-skynet-premium-plans-faster...

coder543 6y ago
Amazon is hosting the Common Crawl on S3 for free, so... yes, $2/month/TB is a lot more expensive.
gillesjacobs 6y ago
It seems that at least on Sia's plans, you can maximally host 20TB for 80$/month, not even a tenth of a monthly common crawl. Of course Sia's Skynet are package deals right now and I guess they're currently bootstrapping the network with users. Filecoin has no operational storage yet. Storj quotes 10$/Terabyte/month [1] so that will come out expensive. 1. https://www.storj.io/blog/2019/11/announcing-pioneer-2-and-tardigrade.io-pricing/ https://www.storj.io/blog/2019/11/announcing-pioneer-2-and-t...

gloriousternary 6y ago

Moreso than Amazon? From my (limited) experience blockchain storage solutions are often less expensive, although I've never worked with petabytes of data so maybe it's different on that scale.

kevincox 6y ago

IPFS doesn't require any blockchain. In fact someone could easily setup some IPFS nodes that fetch the data from the current host if requested over IPFS.[1] This way people could access it via IPFS and provide an alternate mirror of the data. [1] https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#ipfs-urlstore https://github.com/ipfs/go-ipfs/blob/master/docs/experimenta... The main benefits here would be - Even if the source is unavailable there may be other copies on IPFS which would be transparently used. - There may be some performance benefits in rare cases. - If you are accessing this on a bunch of machines your IPFS gateways would handle downloading the source once, then automatically using the local copy from inside your network. The maindownside is that if Amazon is donating their resources why bother with IPFS?