3 ms·
What about https://commoncrawl.org/ https://commoncrawl.org/ ?
by julienfr112 6y ago
What about https://commoncrawl.org/ https://commoncrawl.org/ ?
- fauigerzigerk 6y agoI don't think a search engine based on Commoncrawl would be able to compete with Google. They're not crawling the web frequently enough. They don't execute JavaScript (as far as I know) and I'm not sure they have access to everything that Google has access to (such as paywalled sites). I could be wrong though as I'm not following the project very closely.
- anoncake 6y ago> and I'm not sure they have access to everything that Google has access to (such as paywalled sites). Which, by the way, is another hindrance for potential competitors.
- chongli 6y agoDepends how you look at it. If you want to build a search engine that avoids annoyance like paywalls and JavaScript-only sites, then this isn’t so much of a hindrance.
- Grimm1 6y agoHaving used it in some early work on our search engine, CC does not update near often enough or crawl a big enough portion of the web to be competitive with commercial crawling solutions. It's a great thing that exists, don't get me wrong, but it's better for researchers looking to amass a dataset versus competing in search.
- julienfr112 6y agoCould Google give it some money à la firefox so that hey, it's an competitive market !