Even though Common Crawl exists, if you wanted to get the data locally to have a bit of a poke around, it's practically impossible. The next search engine will

Even though Common Crawl exists, if you wanted to get the data locally to have a bit of a poke around, it's practically impossible. The next search engine will never be made by a few kids playing around in the shed. The latest Common Crawl is roughly 120TB, enough to fit on 48 LTO-6 tapes. LTO-6 is the perfect middle ground expense-wise if you did want to play with the entire dataset at home. The tapes cost ~$30 (AUD) each, a cheap loader from eBay can be had for around ~$1000-4000 (like a Dell PowerVault or Fujitsu Eternus). Your looking at around $8k just to have your data close to your computer. Either that, or you are going to just run everything on AWS/Google Cloud/Azure. Two of these are your competitors. No one is going to copy 48 tapes for you, either :) 120TB would take around three months to download - if my ISP didn't cut me off first for probably being the heaviest consumer user of bandwidth. https://www.chatnoir.eu/ https://www.chatnoir.eu/ has a search engine that is built from the CommonCrawl data running on Elasticsearch, and it runs on 130 nodes [0]. I would love to be able to run my own search engine, but unless there are a number of breakthrough algorithms, I don't see how it can be easily achieved. I did ask some guys from the Internet Archive if they would be happy to copy some data for me onto some tapes. I wonder if there is a service for this? [0] https://webis.de/downloads/publications/papers/bevendorff_2018.pdf https://webis.de/downloads/publications/papers/bevendorff_20...