4 ms·

Google can access any site without being blocked. They dominate the search space and give little incentive for site owners to allow other bots. I'd say bypassin

by soheil 5y ago

Google can access any site without being blocked. They dominate the search space and give little incentive for site owners to allow other bots. I'd say bypassing these measures is fair game while there is a monopoly in search space. We don't want a web that only Google can access. By the way great work on Marginalia search engine, I love it.

marginalia_nu 5y ago
I've honestly not had much problem at all crawling the web as an indie search engine operator. If you want to get past CloudFlare you can register your bot fingerprint with them. A small number of sites has blocked my crawler , but that's almost always been my own fault, and happened a few instances when the crawler was misbehaving and actually fetching too aggressively (or repeatedly). In every case just sending an email to the site explaining what happened and humbly asking for a second chance been enough to be allowed back in. Most website owners don't seem to mind small search engines at all, what they don't want is scrapers that aggressively scrape their entire site 10 times a day, ignoring robots.txt, and being a general nuisance.
- gzer0 5y ago
  How did you go about registering your bot fingerprint with Cloudflare? I did a quick search but cannot find anything with regards to that.
  - marginalia_nu 5y ago
    Good question, I can't find the link now either. I found it somewhere on Cloudflare's site. --edit-- Nevermind, here it is: https://support.cloudflare.com/hc/en-us/articles/360035387431#h_5itGQRBabQ51RwT5cNJX8u https://support.cloudflare.com/hc/en-us/articles/36003538743... It should skip to "I run a good bot...", but if it doesn't, that's where you want to scroll.
- kristjansson 5y ago
  There's also the alignment of interests between you and the site operators, even for a small search engine. Cooperation improves outcomes for both sides. Whereas the more aggressive scrapers almost certainly have interests that conflict with the site operators, regardless of the costs to serve the traffic itself.

by soheil 5y ago

marginalia_nu 5y ago

I've honestly not had much problem at all crawling the web as an indie search engine operator. If you want to get past CloudFlare you can register your bot fingerprint with them. A small number of sites has blocked my crawler , but that's almost always been my own fault, and happened a few instances when the crawler was misbehaving and actually fetching too aggressively (or repeatedly). In every case just sending an email to the site explaining what happened and humbly asking for a second chance been enough to be allowed back in. Most website owners don't seem to mind small search engines at all, what they don't want is scrapers that aggressively scrape their entire site 10 times a day, ignoring robots.txt, and being a general nuisance.

gzer0 5y ago
How did you go about registering your bot fingerprint with Cloudflare? I did a quick search but cannot find anything with regards to that.
- marginalia_nu 5y ago
  Good question, I can't find the link now either. I found it somewhere on Cloudflare's site. --edit-- Nevermind, here it is: https://support.cloudflare.com/hc/en-us/articles/360035387431#h_5itGQRBabQ51RwT5cNJX8u https://support.cloudflare.com/hc/en-us/articles/36003538743... It should skip to "I run a good bot...", but if it doesn't, that's where you want to scroll.
kristjansson 5y ago
There's also the alignment of interests between you and the site operators, even for a small search engine. Cooperation improves outcomes for both sides. Whereas the more aggressive scrapers almost certainly have interests that conflict with the site operators, regardless of the costs to serve the traffic itself.