4 ms·
It would be nice if there was a common crawler offering deltas on top of base checkpoints of the entire crawl; I am guessing most AI companies would prefer not
by benlivengood 10mo ago
It would be nice if there was a common crawler offering deltas on top of base checkpoints of the entire crawl; I am guessing most AI companies would prefer not having to mess with their own scrapers. Google could probably make a mint selling access.
- ccgreg 10mo agocommoncrawl.org Our public web dataset goes back to 2008, and is widely used by academia and startups.
- pdimitar 10mo agoI always wanted to ask: - How often is that updated? - How current is it at any point in time? - Does it have historical / temporal access i.e. be able to check the history of a page a la The Internet Archive?
- ccgreg 10mo ago- monthly - it's a historical archive, the concept of "current" is hard to turn into a metric - not only is our archive historical, it is included in the Internet Archive's wayback machine.