4 ms·
I love the shortcut Alexandria takes by indexing Common Crawl instead of crawling the web themselves. It's how I would have bootstrapped a new search engine. In
by kreeben 5y ago
I love the shortcut Alexandria takes by indexing Common Crawl instead of crawling the web themselves. It's how I would have bootstrapped a new search engine. In a future iteration they can start crawling themselves, if there is sufficient interest from the public.
Searching is screamingly fast.
The index seems stale, though. Alexandria, how old is your index?
How long did it take you to create your current index? Is that your bottleneck, perhaps, that it takes you a long time (and lots of money?) to create a Common Crawl index?
- joshuamorton 5y ago> The index seems stale, though. Alexandria, how old is your index? Common crawl indexes about once every 40 days, the current crawl's data is through January 2022, so it's 1.5 months old at best.