3 ms·
You might be surprised how much compute it takes. Not the same thing but, entire SO website runs on like 2 door sized racks. Search engine might be a different
by systemvoltage 5y ago
You might be surprised how much compute it takes.
Not the same thing but, entire SO website runs on like 2 door sized racks. Search engine might be a different thing but if you have funding to get started, hardware costs aren’t going to be impossibly huge. Most is labor (engineering).
I’m curious, how much compute power it takes to index the whole web? I presume queries are super fast.
- alangibson 5y agoIMHO, indexing the whole web is an antifeature for technical and research topics. But there are projects like CommonCrawl that crawl a good portion of the web and give it away free.
- estebarb 5y agoThe monthly snapshots of CommonCrawl contains around 64k files of HTML, metadata and text only. The text only and metadata are around 250MB each, so in total the corpus would be around 16TB of text and 16TB of metadata. I don't know how much it costs to them, but when I used my own crawler I was able to download 500k webpages in around 4 hours (using the $80/month basic DigitalOcean server... let's say a total of $1/1M web pages, just the crawling process).