4 ms·
Cloudflare.com's Robots.txt
- jsheard 2y agoThis is what happens if your robot isn't nice > curl -I -H "User-Agent: Googlebot" https://www.cloudflare.com HTTP/2 403
- jamesog 2y agoThat's not from robots.txt, but their Bot Management feature which blocks things calling themselves Googlebot that don't come from known Google IPs.
- speedgoose 2y agoAre GCP IPs considered Google IPs?
- crop_rotation 2y agoNo I am very sure they are not.
- jgrahamc 2y agoNo.
- judge2020 2y agoFor reference https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/...
- yapyap 2y agoThat’s cool, if any scrapers would still respect the robots.txt that is
- bityard 2y agoThink of robots.txt as less of a no trespassing sign and more of a, "You can visit but here are the rules to follow if you don't want to get shot" sign.
- iterance 2y agoIf you do not respect the sign I shall be very cross with you. Very cross indeed. Perhaps I shall have to glare at you, yes, very hard. I think I shall glare at you. Perhaps if you are truly irritating I shall be forced to remove you from the premises for a bit.
- blacksmith_tb 2y agoThere's a lot of talk of deregulation in the air, maybe we'll see Gibson-esque Black Ice, where rude crawlers provoke an automated DoS, a new Wild West.
- dartos 2y agoI was surprised any ever did, honestly
- marginalia_nu 2y agoThey may or may not, though respecting robots.txt is a nice way of not having your IP range end up on blacklists. With cloudflare in particular, that can be a bit of a pain. They're pretty nice to deal with if you're upfront about what you are doing and clearly identify your bot, as well as register it with their bot detection. There's a form floating around somewhere for that.
- andrethegiant 2y agoFWIW, that’s why I’m working on a platform[1] to help devs deploy polite crawlers and scrapers out of the box that respect robots.txt (and 429s, Retry-After response headers, etc). It also happens to be entirely built on Cloudflare. [1] https://crawlspace.dev https://crawlspace.dev
- palsecam 2y agoThat’s a funny one! Anyone knows of others like that? Here is mine: https://FreeSolitaire.win/robots.txt https://FreeSolitaire.win/robots.txt
- jsheard 2y agoGoogle used to have a /killer-robots.txt which forbid the T-1000 and T-800 from accessing Larry Page and Sergey Brin, but they took that down at some point.
- shaunpud 2y agohttps://web.archive.org/web/20160530160330/https://www.google.com/killer-robots.txt https://web.archive.org/web/20160530160330/https://www.googl...
- segphault 2y agoStripe has a humans.txt: https://stripe.com/humans.txt https://stripe.com/humans.txt
- ttepasse 2y agohumans.txt is a small scale initiative: https://humanstxt.org/ https://humanstxt.org/
- orliesaurus 2y agoHas anyone worked on anything like this for AI scrapers?
- dartos 2y agoA robots.txt that asks AI scrapers not to scrape? There’s a couple services that keep updated lists of known scraper user agents. A quick search reveals a handful.
- zorked 2y agohttps://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt https://github.com/ai-robots-txt/ai.robots.txt/blob/main/rob...
- gnaman 2y agohttps://llmstxt.org/ https://llmstxt.org/ https://www.answer.ai/posts/2024-09-03-llmstxt.html https://www.answer.ai/posts/2024-09-03-llmstxt.html
- CodesInChaos 2y agoWhat's the purpose of "User-Agent: DemandbaseWebsitePreview/0.1"? I couldn't find anything about that agent, but I assume it's somehow related to demandbase.com? But why are it and twitter the only whitelisted entries? Google and bing missing is a bit surprising, but I assume they're whitelisted through a different mechanism (like a google webmaster account)?
- saddist0 2y agoIt is one of the service they use. As per the cookie policy page [1]: > DemandBase - Enables us to identify companies who intend to purchase our products and solutions and deliver more relevant messages and offers to our Website visitors. [1]: https://www.cloudflare.com/en-in/cookie-policy/ https://www.cloudflare.com/en-in/cookie-policy/
- Maken 2y agoMy guess is that the Twitter one is for previews when you link to a web in Twitter.
- op00to 2y agoIf those robots could read, they'd be very upset.
- seanwilson 2y agoI have an ASCII art Easter egg like this in an SEO product I made. :) https://www.checkbot.io/robots.txt https://www.checkbot.io/robots.txt I should probably add this SEO tip too because the purpose of robots.txt is confusing: If you want to remove/deindex a page from Google search, you counterintuitively need to allow the page to be crawled in the robots.txt file, and then add a noindex response header or noindex meta tag to the page. This way the crawler gets to see the noindex instruction. Robots.txt controls which pages can be crawled, not which pages can be indexed.
- dazc 2y agoThe consequences of robots.txt misuse can also be disastrous for a regular site. For example, I've seen instances where multiple warnings of 'page indexed but blocked by robots.txt' have led to sites being severely down-ranked as a consequence. My assumption being that search engines don't want to be listing too many pages that everyone can read and they can not.
- m-app 2y agoWhat does “OUR TREE IS A REDWOOD” refer to? A quick search doesn’t yield any definite results.
- sandworm101 2y ago[flagged]
- withinboredom 2y agohttps://en.wikipedia.org/wiki/Roko%27s_basilisk https://en.wikipedia.org/wiki/Roko%27s_basilisk ???
- chrisweekly 2y agoOne nice thing about CF's robots.txt is its inclusion of a sitemap: https://www.cloudflare.com/sitemap.xml https://www.cloudflare.com/sitemap.xml which contains links to educational materials like https://www.cloudflare.com/learning/ddos/layer-3-ddos-attacks/ https://www.cloudflare.com/learning/ddos/layer-3-ddos-attack... Potentially interesting to see their flattened IA....
- palsecam 2y agoLittle-known fact: a syndication feed (RSS or Atom) can be used as a sitemap. Quoting https://www.sitemaps.org/protocol.html#otherformats https://www.sitemaps.org/protocol.html#otherformats: > The Sitemap protocol enables you to provide details about your pages to search engines, […] in addition to the XML protocol, we support RSS feeds and text files, which provide more limited information. > You can provide an RSS (Real Simple Syndication) 2.0 or Atom 0.3 or 1.0 feed. Generally, you would use this format only if your site already has a syndication feed.
- ck2 2y agoeasy guess that length breaks some legacy stuff but every robots.txt should have a auto-ban trap line ie. crawl it and die basically a script that puts the requesting IP into firewall of course it's possible to abuse that so it has to be monitored
- okdood64 2y agoHow do you discern a crawler agent and a human? Is it easily as the fact that they might cover something like 80%+ of the site in one visit fairly quickly?
- SoftTalker 2y agoCrawlers/archivers will be hitting your site much faster than a human user.
- johneth 2y agoI thought about doing something like that, but then I realised: what if someone linked to the trap URL it from another site and a crawler followed that link to the trap? You might end up penalising Googlebot or Bingbot. If anyone knew what that trap URL did, and felt malicious, this could happen.
- CodesInChaos 2y agoA crawler could easily avoid that by fetching the target domain's robots.txt before fetching the link target. However a website could also embed the honeypot link in an <img> tag and get the user banned when their browser attempts to load the image.