6 ms·

Yeah, there are certainly more problems these days. For one, the size of the web is larger, more of it is spam causing issues with pure page rank to detect netw

Yeah, there are certainly more problems these days. For one, the size of the web is larger, more of it is spam causing issues with pure page rank to detect networks that heavily link to each other. Important sites have a bunch of anti-crawling detection set up (especially news sites). It's even worse that the best user-generated content is behind walled gardens in facebook groups, slack channels, quora threads, etc... The rest of the good sites are javascript-heavy and you often have to run chrome headless to render the page and find the content - but that is detectable so you end up renting IP's from mobile number farms or trying to build your own 4G network. On the upside, https://commoncrawl.org/ https://commoncrawl.org/ now exists and makes the prototype crawling work much easier. It's not the full internet, but gives you plenty to work with and test against so you can skip to the part where you figure out if you can produce anything useful should you actually try to crawl the whole internet.

nonrandomstring 4y ago
> but that is detectable so you end up renting IP's from mobile number farms or trying to build your own 4G network. Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. The established idea of a "search engine" seems stuck, limited and based on some 90's technology that worked on a 90's web that no longer exists. Surely after 30 years we can build some kind of content discovery layer on top of what's out there?
- Xeoncross 4y ago
  https://blogsurf.io/ https://blogsurf.io/ is an example of a small search engine that just stuck to a directory of known blogs instead of indexing the big sites or randomly crawling the web and ending up with mostly gibberish pages from all the spam sites.
  - mannyistyping 4y ago
    thank you for sharing this! I read through the site's about and I really enjoy how the creator wanted to stick to a specific area for quality over quantity.
- jeffbee 4y ago
  I think that is not what they mean. I think what they meant is the site will detect your headless robot and serve it good content, while serving spam and malware to everyone else. The crawlers need their own distributed networks of unrelated addresses to prevent or detect this behavior.
- thanksgiving 4y ago
  > Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. I work on a small - medium ecommerce website and my code just... sucks. I kind of don't want to admit it but it is true. When there is some Chinese search engine that tries to crawl all the product detail pages during the day (presumably at night for them?), it slows down the site to a crawl. I mean technically I should have the pages set up so they can't pierce through the cloudflare cache but it is easier to just ask cloudflare to challenge user (captcha?) if there are more than n (I think currently set to something small like ten) requests per second from any single source. I don't understand all the business decisions but yeah, I'd suspect the biggest reason is we simply have poor codebases and can't spend too much time fixing this while we have so many backlog items from marketing to work on...
  - ALittleLight 4y ago
    Why are page loads so slow or demanding? I can't imagine how a web crawler could be DoS'ing you if it's in good faith. What is the TPS? What caching are you doing? What's your stack like?
    - Gh0stRAT 4y ago
      Not GP, but from having run a small/niche search engine that got hammered by a crawler in the past: Webserver was a single VM running a Java + Spring webserver in Tomcat, connecting to an overworked Solr cluster to do the actual faceted searching. Caches kept most page loads for organic traffic within respectable bounds, but the crawler destroyed our cache hit rate when it was scraping our site and at one point did exhaust a concurrent connection limit of some kind because there were so many slow/timing-out requests in progress at the same time.
      - ALittleLight 4y ago
        I would expect that a small to medium e-commerce site would cache all their pages.
- jonhohle 4y ago
  Maybe we need a categorized, hand curated directory of sites that users can submit their own sites to for inclusion and categorization. Maybe like an open directory. Perhaps Mozilla could operate it, or maybe Yahoo!
  - noduerme 4y ago
    I know, right? Imagine if you went to the front page of Yahoo! and it was like a curated directory of websites. Like... a portal. It could look something like this: https://web.archive.org/web/20000302042007/http://www1.yahoo.com/ https://web.archive.org/web/20000302042007/http://www1.yahoo...
    - mysterydip 4y ago
      I would love to see a combo of a search engine and a portal. Basically at any step drilling down into links for categories and subcategories, I could type in a search bar and every site at that level and below would be searched.
      - marginalia_nu 4y ago
        I've been sort of considering going this path with my own search engine, but I feel whenever there is any sort of interactivity, there is the serious problem of spam and abuse; I'm not quite sure how to solve that. Feels like accepting user content no longer flies on the web in 2022 due to the effectiveness of bot spam and lack of viable countermeasures :-/
        mysterydip 4y ago
        I think non-crowdsourced manual curation (by you/your trusted volunteers) would be a feature. Sure, search would have more limited results, but really with how much cruft there is I don't see that as a bad thing. Start with the "known good/useful" domains and branch out from there.
        4y ago
- altdataseller 4y ago
  >> If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. Then they should treat all bots equally and block Google as well. If they block Google as well, then yes, we should leave them alone. Why give unfair treatment to Google? That's anti-competitive behavior and it just prevents new search engines from being created.
  - nonrandomstring 4y ago
    I think I understand, combined with jeffbee's answer, that these sites are behaving selectively according to who you are. So we're back to "No Blacks or Irish" on the 2022 Internet? What do you think they have against smaller search engines? I can't quite fathom the motives.
    - wolfgang42 4y ago
      There are a lot of crawlers out there, and many of them are ill-behaved. When GoogleBot crawls your site, you get more visitors. When FizzBuzzBot/0.1.3 comes along, you’re more likely to get an overloaded server, weird URLs in your crash logs, spam, or any other manner of mess. Small search engines getting blocked is just collateral damage from websites trying to solve this problem with a blunt ban-hammer.
      - bryanrasmussen 4y ago
        This is probably because, as I have learned here on HN, 1 in 4 programmers can't write a decent FizzBuzzBot and should never be hired.
    - autoexec 4y ago
      > So we're back to "No Blacks or Irish" on the 2022 Internet? I think this is the inevitable result of surveillance capitalism. Once you can reliably be identified as holding certain views or meeting some arbitrary criteria pages can be dynamically altered to restrict what you have access to. A racist social platform can look completely benign to anyone who isn't part of the in-group, forums for teens can automatically lock out anyone suspected of being over a certain age, companies can show products or services only to people within a certain class, etc. We already restrict a lot of access to content based on guesses about location. Youtube doesn't even let you see that certain content on their platform exists on their site unless you're logged in from an IP in specific counties (as opposed to showing you the videos exist and throwing up an error saying it's not available for you when you try to view it) and netflix blocks VPNs just to try to keep a bunch of their content away from the wrong kinds of people. These are the kinds of barriers the internet should have freed us from, but instead it's being used to put up more gates and to force people into them.
- amelius 4y ago
  > Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? I think it is not about being found. It is more about being copied. These sites are afraid their content is stolen, so they only allow Google to crawl them.
- noncoml 4y ago
  They don’t want to be indexed unless you are Google
- throwaway81523 4y ago
  > If sites don't want to be found and indexed why go to any effort to include them? The news sites etc. want to be indexed by Google, and they whitelist Google so they show up in search results. They don't want to be crawled by random aggregators and scrapers which don't bring them ad views and subscriptions. Indexing for me and not for thee. Other places like Slack channels are private or semi-private and shouldn't be indexed, I agree. There was some kind of scheme a while back (before Freenode went down the tubes in a completely different and unrelated way) to index Freenode. People rightfully freaked out about it and the scheme was shot down.
- anotherrandom 4y ago
  Sounds like federation might be making a comeback eventually
ArrayBoundCheck 4y ago
I don't know how people can use the data. There's so much of it! I don't see any harddrives that are 80TB. It seems like people would need some kind of raid setup that can handle 200+TB of uncompressed data
- Xeoncross 4y ago
  You don't need to download the whole thing. You can parse the WARC files from S3 to only extract the information you want (like pages with content). It's a lot smaller when you only keep the links and text.
- francoismassot 4y ago
  A search index is often made of smaller independent pieces often called segments. So you can download & process progressively the data locally and upload it to an object storage. And run queries on it. That's what we did here for this project: https://quickwit.io/blog/commoncrawl https://quickwit.io/blog/commoncrawl Also an interesting blog post here: https://fulmicoton.com/posts/commoncrawl/ https://fulmicoton.com/posts/commoncrawl/
  - marginalia_nu 4y ago
    Huh, that's pretty expensive. For comparison, my search engine has an operational cost of ~$50/mo.
Xeoncross 4y ago
There isn't a one-size-fits all approach, but I've never worked on a project that encompasses as many computer science algorithms as a search engine. - Tries (patricia, radix, etc...) - Trees (b-trees, b+trees, merkle trees, log-structured merge-tree, etc..) - Consensus (raft, paxos, etc..) - Block storage (disk block size optimizations, mmap files, delta storage, etc..) - Probabilistic filters (hyperloloog, bloom filters, etc...) - Binary Search (sstables, sorted inverted indexes) - Ranking (pagerank, tf/idf, bm25, etc...) - NLP (stemming, POS tagging, subject identification, etc...) - HTML (document parsing/lexing) - Images (exif extraction, removal, resizing / proxying, etc...) - Queues (SQS, NATS, Apollo, etc...) - Clustering (k-means, density, hierarchical, gaussian distributions, etc...) - Rate limiting (leaky bucket, windowed, etc...) - text processing (unicode-normalization, slugify, sanitation, lossless and lossy hashing like metaphone and document fingerprinting) - etc... I'm sure there is plenty more I've missed. There are lots of generic structures involved like hashes, linked-lists, skip-lists, heaps and priority queues and this is just to get 2000's level basic tech. - https://github.com/quickwit-oss/tantivy https://github.com/quickwit-oss/tantivy - https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic - https://github.com/mosuka/phalanx https://github.com/mosuka/phalanx - https://github.com/meilisearch/MeiliSearch https://github.com/meilisearch/MeiliSearch - https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve - https://github.com/thomasjungblut/go-sstables https://github.com/thomasjungblut/go-sstables A lot of people new to this space mistakenly think you can just throw elastic search or postgres fulltext search in front of terabytes of records and have something decent. That might work for something small like a curated collection of a few hundred sites.
- kreeben 4y ago
  Yes, yes, yes :D There are so many topics in this space that are so interesting it's like a dream. I would add to your list - sentiment analysis - roaring bitmaps - compression - applied linear algebra - ai In a vent diagram intersecting all of these topics, is search. Coding a search engine from scratch is a beautiful way to spend ones days, if you're into programming.
- boyter 4y ago
  > That might work for something small like a curated collection of a few hundred sites. Probably more like a few million but otherwise 100% true. Once you really need to scale you have to start losing some accuracy or correctness. It helps that the goal of a search engine is not to find all the results but instead delight the user by finding the things they want.
deleted 4y ago
[deleted]
speedgoose 4y ago
Are the Google crawling bots using similar IP ranges than Google Cloud?
tim-- 4y ago
Even though Common Crawl exists, if you wanted to get the data locally to have a bit of a poke around, it's practically impossible. The next search engine will never be made by a few kids playing around in the shed. The latest Common Crawl is roughly 120TB, enough to fit on 48 LTO-6 tapes. LTO-6 is the perfect middle ground expense-wise if you did want to play with the entire dataset at home. The tapes cost ~$30 (AUD) each, a cheap loader from eBay can be had for around ~$1000-4000 (like a Dell PowerVault or Fujitsu Eternus). Your looking at around $8k just to have your data close to your computer. Either that, or you are going to just run everything on AWS/Google Cloud/Azure. Two of these are your competitors. No one is going to copy 48 tapes for you, either :) 120TB would take around three months to download - if my ISP didn't cut me off first for probably being the heaviest consumer user of bandwidth. https://www.chatnoir.eu/ https://www.chatnoir.eu/ has a search engine that is built from the CommonCrawl data running on Elasticsearch, and it runs on 130 nodes [0]. I would love to be able to run my own search engine, but unless there are a number of breakthrough algorithms, I don't see how it can be easily achieved. I did ask some guys from the Internet Archive if they would be happy to copy some data for me onto some tapes. I wonder if there is a service for this? [0] https://webis.de/downloads/publications/papers/bevendorff_2018.pdf https://webis.de/downloads/publications/papers/bevendorff_20...
- cosmolev 4y ago
  Why dealing with LTO-6 tapes? Isn't it easier to deploy 8 x 16TB HDDs?
  - tim-- 4y ago
    Tapes are cheaper when sneaker footing the data around.
- marginalia_nu 4y ago
  I've sort of mostly dismissed common crawl for the same reason when building my search engine. It's simply too unwieldy. It's far easier (and cheaper) to do my own crawling at a manageable scale, than it is to work with CC's datasets. I'm also not entirely sure which problem CC is intending to solve. It's not like their data is in any way more complete than my own crawl sets, what I can't access to crawl, they don't seem to crawl either.

nonrandomstring 4y ago

> but that is detectable so you end up renting IP's from mobile number farms or trying to build your own 4G network. Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. The established idea of a "search engine" seems stuck, limited and based on some 90's technology that worked on a 90's web that no longer exists. Surely after 30 years we can build some kind of content discovery layer on top of what's out there?

Xeoncross 4y ago
https://blogsurf.io/ https://blogsurf.io/ is an example of a small search engine that just stuck to a directory of known blogs instead of indexing the big sites or randomly crawling the web and ending up with mostly gibberish pages from all the spam sites.
- mannyistyping 4y ago
  thank you for sharing this! I read through the site's about and I really enjoy how the creator wanted to stick to a specific area for quality over quantity.
jeffbee 4y ago
I think that is not what they mean. I think what they meant is the site will detect your headless robot and serve it good content, while serving spam and malware to everyone else. The crawlers need their own distributed networks of unrelated addresses to prevent or detect this behavior.
thanksgiving 4y ago
> Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. I work on a small - medium ecommerce website and my code just... sucks. I kind of don't want to admit it but it is true. When there is some Chinese search engine that tries to crawl all the product detail pages during the day (presumably at night for them?), it slows down the site to a crawl. I mean technically I should have the pages set up so they can't pierce through the cloudflare cache but it is easier to just ask cloudflare to challenge user (captcha?) if there are more than n (I think currently set to something small like ten) requests per second from any single source. I don't understand all the business decisions but yeah, I'd suspect the biggest reason is we simply have poor codebases and can't spend too much time fixing this while we have so many backlog items from marketing to work on...
- ALittleLight 4y ago
  Why are page loads so slow or demanding? I can't imagine how a web crawler could be DoS'ing you if it's in good faith. What is the TPS? What caching are you doing? What's your stack like?
  - Gh0stRAT 4y ago
    Not GP, but from having run a small/niche search engine that got hammered by a crawler in the past: Webserver was a single VM running a Java + Spring webserver in Tomcat, connecting to an overworked Solr cluster to do the actual faceted searching. Caches kept most page loads for organic traffic within respectable bounds, but the crawler destroyed our cache hit rate when it was scraping our site and at one point did exhaust a concurrent connection limit of some kind because there were so many slow/timing-out requests in progress at the same time.
    - ALittleLight 4y ago
      I would expect that a small to medium e-commerce site would cache all their pages.
jonhohle 4y ago
Maybe we need a categorized, hand curated directory of sites that users can submit their own sites to for inclusion and categorization. Maybe like an open directory. Perhaps Mozilla could operate it, or maybe Yahoo!
- noduerme 4y ago
  I know, right? Imagine if you went to the front page of Yahoo! and it was like a curated directory of websites. Like... a portal. It could look something like this: https://web.archive.org/web/20000302042007/http://www1.yahoo.com/ https://web.archive.org/web/20000302042007/http://www1.yahoo...
  - mysterydip 4y ago
    I would love to see a combo of a search engine and a portal. Basically at any step drilling down into links for categories and subcategories, I could type in a search bar and every site at that level and below would be searched.
    - marginalia_nu 4y ago
      I've been sort of considering going this path with my own search engine, but I feel whenever there is any sort of interactivity, there is the serious problem of spam and abuse; I'm not quite sure how to solve that. Feels like accepting user content no longer flies on the web in 2022 due to the effectiveness of bot spam and lack of viable countermeasures :-/
      - mysterydip 4y ago
        I think non-crowdsourced manual curation (by you/your trusted volunteers) would be a feature. Sure, search would have more limited results, but really with how much cruft there is I don't see that as a bad thing. Start with the "known good/useful" domains and branch out from there.
        4y ago
altdataseller 4y ago
>> If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. Then they should treat all bots equally and block Google as well. If they block Google as well, then yes, we should leave them alone. Why give unfair treatment to Google? That's anti-competitive behavior and it just prevents new search engines from being created.
- nonrandomstring 4y ago
  I think I understand, combined with jeffbee's answer, that these sites are behaving selectively according to who you are. So we're back to "No Blacks or Irish" on the 2022 Internet? What do you think they have against smaller search engines? I can't quite fathom the motives.
  - wolfgang42 4y ago
    There are a lot of crawlers out there, and many of them are ill-behaved. When GoogleBot crawls your site, you get more visitors. When FizzBuzzBot/0.1.3 comes along, you’re more likely to get an overloaded server, weird URLs in your crash logs, spam, or any other manner of mess. Small search engines getting blocked is just collateral damage from websites trying to solve this problem with a blunt ban-hammer.
    - bryanrasmussen 4y ago
      This is probably because, as I have learned here on HN, 1 in 4 programmers can't write a decent FizzBuzzBot and should never be hired.
  - autoexec 4y ago
    > So we're back to "No Blacks or Irish" on the 2022 Internet? I think this is the inevitable result of surveillance capitalism. Once you can reliably be identified as holding certain views or meeting some arbitrary criteria pages can be dynamically altered to restrict what you have access to. A racist social platform can look completely benign to anyone who isn't part of the in-group, forums for teens can automatically lock out anyone suspected of being over a certain age, companies can show products or services only to people within a certain class, etc. We already restrict a lot of access to content based on guesses about location. Youtube doesn't even let you see that certain content on their platform exists on their site unless you're logged in from an IP in specific counties (as opposed to showing you the videos exist and throwing up an error saying it's not available for you when you try to view it) and netflix blocks VPNs just to try to keep a bunch of their content away from the wrong kinds of people. These are the kinds of barriers the internet should have freed us from, but instead it's being used to put up more gates and to force people into them.
amelius 4y ago
> Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? I think it is not about being found. It is more about being copied. These sites are afraid their content is stolen, so they only allow Google to crawl them.
noncoml 4y ago
They don’t want to be indexed unless you are Google
throwaway81523 4y ago
> If sites don't want to be found and indexed why go to any effort to include them? The news sites etc. want to be indexed by Google, and they whitelist Google so they show up in search results. They don't want to be crawled by random aggregators and scrapers which don't bring them ad views and subscriptions. Indexing for me and not for thee. Other places like Slack channels are private or semi-private and shouldn't be indexed, I agree. There was some kind of scheme a while back (before Freenode went down the tubes in a completely different and unrelated way) to index Freenode. People rightfully freaked out about it and the scheme was shot down.
anotherrandom 4y ago
Sounds like federation might be making a comeback eventually

ArrayBoundCheck 4y ago

I don't know how people can use the data. There's so much of it! I don't see any harddrives that are 80TB. It seems like people would need some kind of raid setup that can handle 200+TB of uncompressed data

Xeoncross 4y ago
You don't need to download the whole thing. You can parse the WARC files from S3 to only extract the information you want (like pages with content). It's a lot smaller when you only keep the links and text.
francoismassot 4y ago
A search index is often made of smaller independent pieces often called segments. So you can download & process progressively the data locally and upload it to an object storage. And run queries on it. That's what we did here for this project: https://quickwit.io/blog/commoncrawl https://quickwit.io/blog/commoncrawl Also an interesting blog post here: https://fulmicoton.com/posts/commoncrawl/ https://fulmicoton.com/posts/commoncrawl/
- marginalia_nu 4y ago
  Huh, that's pretty expensive. For comparison, my search engine has an operational cost of ~$50/mo.

Xeoncross 4y ago

There isn't a one-size-fits all approach, but I've never worked on a project that encompasses as many computer science algorithms as a search engine. - Tries (patricia, radix, etc...) - Trees (b-trees, b+trees, merkle trees, log-structured merge-tree, etc..) - Consensus (raft, paxos, etc..) - Block storage (disk block size optimizations, mmap files, delta storage, etc..) - Probabilistic filters (hyperloloog, bloom filters, etc...) - Binary Search (sstables, sorted inverted indexes) - Ranking (pagerank, tf/idf, bm25, etc...) - NLP (stemming, POS tagging, subject identification, etc...) - HTML (document parsing/lexing) - Images (exif extraction, removal, resizing / proxying, etc...) - Queues (SQS, NATS, Apollo, etc...) - Clustering (k-means, density, hierarchical, gaussian distributions, etc...) - Rate limiting (leaky bucket, windowed, etc...) - text processing (unicode-normalization, slugify, sanitation, lossless and lossy hashing like metaphone and document fingerprinting) - etc... I'm sure there is plenty more I've missed. There are lots of generic structures involved like hashes, linked-lists, skip-lists, heaps and priority queues and this is just to get 2000's level basic tech. - https://github.com/quickwit-oss/tantivy https://github.com/quickwit-oss/tantivy - https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic - https://github.com/mosuka/phalanx https://github.com/mosuka/phalanx - https://github.com/meilisearch/MeiliSearch https://github.com/meilisearch/MeiliSearch - https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve - https://github.com/thomasjungblut/go-sstables https://github.com/thomasjungblut/go-sstables A lot of people new to this space mistakenly think you can just throw elastic search or postgres fulltext search in front of terabytes of records and have something decent. That might work for something small like a curated collection of a few hundred sites.

kreeben 4y ago
Yes, yes, yes :D There are so many topics in this space that are so interesting it's like a dream. I would add to your list - sentiment analysis - roaring bitmaps - compression - applied linear algebra - ai In a vent diagram intersecting all of these topics, is search. Coding a search engine from scratch is a beautiful way to spend ones days, if you're into programming.
boyter 4y ago
> That might work for something small like a curated collection of a few hundred sites. Probably more like a few million but otherwise 100% true. Once you really need to scale you have to start losing some accuracy or correctness. It helps that the goal of a search engine is not to find all the results but instead delight the user by finding the things they want.

deleted 4y ago

[deleted]

speedgoose 4y ago

Are the Google crawling bots using similar IP ranges than Google Cloud?

tim-- 4y ago

Even though Common Crawl exists, if you wanted to get the data locally to have a bit of a poke around, it's practically impossible. The next search engine will never be made by a few kids playing around in the shed. The latest Common Crawl is roughly 120TB, enough to fit on 48 LTO-6 tapes. LTO-6 is the perfect middle ground expense-wise if you did want to play with the entire dataset at home. The tapes cost ~$30 (AUD) each, a cheap loader from eBay can be had for around ~$1000-4000 (like a Dell PowerVault or Fujitsu Eternus). Your looking at around $8k just to have your data close to your computer. Either that, or you are going to just run everything on AWS/Google Cloud/Azure. Two of these are your competitors. No one is going to copy 48 tapes for you, either :) 120TB would take around three months to download - if my ISP didn't cut me off first for probably being the heaviest consumer user of bandwidth. https://www.chatnoir.eu/ https://www.chatnoir.eu/ has a search engine that is built from the CommonCrawl data running on Elasticsearch, and it runs on 130 nodes [0]. I would love to be able to run my own search engine, but unless there are a number of breakthrough algorithms, I don't see how it can be easily achieved. I did ask some guys from the Internet Archive if they would be happy to copy some data for me onto some tapes. I wonder if there is a service for this? [0] https://webis.de/downloads/publications/papers/bevendorff_2018.pdf https://webis.de/downloads/publications/papers/bevendorff_20...

cosmolev 4y ago
Why dealing with LTO-6 tapes? Isn't it easier to deploy 8 x 16TB HDDs?
- tim-- 4y ago
  Tapes are cheaper when sneaker footing the data around.
marginalia_nu 4y ago
I've sort of mostly dismissed common crawl for the same reason when building my search engine. It's simply too unwieldy. It's far easier (and cheaper) to do my own crawling at a manageable scale, than it is to work with CC's datasets. I'm also not entirely sure which problem CC is intending to solve. It's not like their data is in any way more complete than my own crawl sets, what I can't access to crawl, they don't seem to crawl either.