4 ms·
Some sources update faster than others, you could index news sources hourly and low velocity sites weekly. Google does that. CommonCrawl gets 7TB/month, indexin
by arnaudsm 2y ago
Some sources update faster than others, you could index news sources hourly and low velocity sites weekly. Google does that. CommonCrawl gets 7TB/month, indexing and vectorizing that is quite manageable.
- stephantul 2y agoI think it is more complex than that. Common crawl does not index the whole web every month. So even if you use common crawl and just index it every month, which you could do pretty cheaply admittedly, I don't think that would lead to a good search index. Running an index is an extremely profitable business, from multiple points of view (you can literally earn money, but also run ads, you get information you can sell, you can buy mindshare). Everybody is looking for indexes beyond Google and Bing, but there are none. If it really is as easy as indexing common crawl, then I think we'd have more indexes.
- notyourwork 2y agoThis reads like the “I could build Dropbox in a weekend”.
- iterance 2y agoIt's just files, how hard could it be?
- rwaksmunski 2y agoNo that hard, took me 4 weekends to build a private search engine with Common Crawl, Wikipedia and HN as a link authority source. Takes about a week to crunch the data on an old Lenovo workstation with 256gb ram and some storage.
- arnaudsm 2y agoHaha yes, but my argument is not about individuals, but about "tech" companies that externalize everything and do not develop tech internally.
- marban 2y agoFor news-only there's https://littleberg.com https://littleberg.com