4 ms·

Plug for https://commoncrawl.org/ https://commoncrawl.org/ if you need billions of pages but don't want to deal with scraping the web yourself.

deleted 4y ago
[deleted]
squiggy22 4y ago
Is there subsets of common crawl anywhere for individual sites. E.g. YouTube for example?
- magundu 4y ago
  You can query subset of specific sites from common crawl itself.
mdaniel 4y ago
It would thrill me if common crawl were updated with such frequency that it would allow new search engines to enter the market I haven't dug into it enough to know if there's some technical reason it's not currently the case, or just lack of (interest|willpower)
- input_sh 4y ago
  I'd argue that one broad crawl every 2-3 months in addition to their updated-daily news crawl[0] should be good enough to make a rudimentary search engine. [0] https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html
  - mdaniel 4y ago
    You're right about the "rudimentary" part, because I don't know how they do it but the major players have some not-kidding-around freshness: https://www.google.com/search?hl=en&q=%22thrill%20me%22%20%22common%20crawl%22%20site%3Anews.ycombinator.com&tbs=qdr%3Ad https://www.google.com/search?hl=en&q=%22thrill%20me%22%20%2... https://www.bing.com/search?q=%22thrill+me%22+%22common+crawl%22+site%3Anews.ycombinator.com https://www.bing.com/search?q=%22thrill+me%22+%22common+craw... (and DDG similarly, because bing) ed: I was curious if maybe HN publishes a sitemap, and it seems no. Then again, hnreplies knows about the HN API so maybe it's special-cased by the big crawlers https://github.com/ggerganov/hnreplies#hnreplies https://github.com/ggerganov/hnreplies#hnreplies
    - jjeaff 4y ago
      I think it is as simple as an algorithm that rates how frequently new updates show up vs how many people visit a site. If you rarely make updates to your site, Google crawls it infrequently and new things won't show up very quickly. But if you do have frequent updates and lots of traffic, like any popular forum style site, you will get lots of crawler traffic. And I would bet the algo does the same for all endpoints on a site. So the "about us" page on a popular site probably ends up not being crawled nearly as much as the new threads page.

deleted 4y ago

[deleted]

squiggy22 4y ago

Is there subsets of common crawl anywhere for individual sites. E.g. YouTube for example?

magundu 4y ago
You can query subset of specific sites from common crawl itself.

mdaniel 4y ago

It would thrill me if common crawl were updated with such frequency that it would allow new search engines to enter the market I haven't dug into it enough to know if there's some technical reason it's not currently the case, or just lack of (interest|willpower)

input_sh 4y ago
I'd argue that one broad crawl every 2-3 months in addition to their updated-daily news crawl[0] should be good enough to make a rudimentary search engine. [0] https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html
- mdaniel 4y ago
  You're right about the "rudimentary" part, because I don't know how they do it but the major players have some not-kidding-around freshness: https://www.google.com/search?hl=en&q=%22thrill%20me%22%20%22common%20crawl%22%20site%3Anews.ycombinator.com&tbs=qdr%3Ad https://www.google.com/search?hl=en&q=%22thrill%20me%22%20%2... https://www.bing.com/search?q=%22thrill+me%22+%22common+crawl%22+site%3Anews.ycombinator.com https://www.bing.com/search?q=%22thrill+me%22+%22common+craw... (and DDG similarly, because bing) ed: I was curious if maybe HN publishes a sitemap, and it seems no. Then again, hnreplies knows about the HN API so maybe it's special-cased by the big crawlers https://github.com/ggerganov/hnreplies#hnreplies https://github.com/ggerganov/hnreplies#hnreplies
  - jjeaff 4y ago
    I think it is as simple as an algorithm that rates how frequently new updates show up vs how many people visit a site. If you rarely make updates to your site, Google crawls it infrequently and new things won't show up very quickly. But if you do have frequent updates and lots of traffic, like any popular forum style site, you will get lots of crawler traffic. And I would bet the algo does the same for all endpoints on a site. So the "about us" page on a popular site probably ends up not being crawled nearly as much as the new threads page.