3 ms·
The challenges to OSS developers are numerous. First of all, many popular sites on the internet block crawlers other than Google and Bing, because only those on
by Mediterraneo10 6y ago
The challenges to OSS developers are numerous. First of all, many popular sites on the internet block crawlers other than Google and Bing, because only those ones seem to matter to their business, and any small upstart would be assumed to be a dodgy bot. Secondly, Google amasses the database it has only with vast data centers, incredible amounts of bandwidth, and power requirements unavailable to a startup.
- creese 6y agoHow would anyone block a crawler? A crawler is just a headless browser.
- tleb_ 6y agorobots.txt https://www.robotstxt.org/ https://www.robotstxt.org/ https://en.wikipedia.org/wiki/Robots_exclusion_standard https://en.wikipedia.org/wiki/Robots_exclusion_standard
- Xylakant 6y agoNote that robots.txt is a hint to well-behaved crawlers, not blocking them in any regard. You can block crawlers if you can identify them, but reliably identifying them is hard.
- tleb_ 6y agoWe should probably classify the crawler identifying problem as impossible and move along. Less resources wasted and easier automation for everyone. Assuming a crawler is malicious is narrow-minded.
- ddorian43 6y agohttps://developers.google.com/search/docs/advanced/verifying-googlebot https://developers.google.com/search/docs/advanced/verifying...
- Xylakant 6y agoThis helps to verify that a bot that announces itself as google bot is indeed a google bot. It doesn’t help identify a bot that pretends to be a user/browser.
- aembleton 6y agoJust use the string 'googlebot' in your user agent. After all, Google uses 'Mozilla' in their Google Bot user agent String for similar reasons - because sites might expect it.