2 ms·
I don't think a search engine based on Commoncrawl would be able to compete with Google. They're not crawling the web frequently enough. They don't execute Jav
by fauigerzigerk 6y ago
I don't think a search engine based on Commoncrawl would be able to compete with Google.
They're not crawling the web frequently enough. They don't execute JavaScript (as far as I know) and I'm not sure they have access to everything that Google has access to (such as paywalled sites).
I could be wrong though as I'm not following the project very closely.
- anoncake 6y ago> and I'm not sure they have access to everything that Google has access to (such as paywalled sites). Which, by the way, is another hindrance for potential competitors.
- chongli 6y agoDepends how you look at it. If you want to build a search engine that avoids annoyance like paywalls and JavaScript-only sites, then this isn’t so much of a hindrance.