7 ms·
As a search engine operator, I can confirm webmasters put a lot of weird crap in robots.txt. ASCII art, poems, emoji, love letters, sudoku puzzles, philosophica
by marginalia_nu 5y ago
As a search engine operator, I can confirm webmasters put a lot of weird crap in robots.txt. ASCII art, poems, emoji, love letters, sudoku puzzles, philosophical questions, whatever. It's cute, but odds are at least some parser will choke on them and depending on how the crawler is coded, either ignore your robots.txt; or skip crawling your site. It's not great either way as the format is more of a set of conventions than a proper well defined language all specified up in BNF.
Webmasters also put highly specific URLs they don't want the world to see in robots.txt. Also not a great idea.
- hughrr 5y agoI made my robots.txt 2Gb of garbage from dev random to see how many spiders I could break. At least three Chinese ones downloaded the whole file.
- 5e92cb50239222b 5y agoNext time drop a symlink from robots.txt to /dev/urandom and see how it goes.
- dspillett 5y agoYou might want to rate limit that if you don't magically have free unlimited bandwidth quota.
- marginalia_nu 5y agoI hear it's decades of bad luck to do unless you check that the crawler's user-agent isn't "search.marginalia.nu"; but you could just tarpit the connection.
- dijit 5y agoWhile I'm in general not a fan of wildcard certificates, yours doesn't work for the bare domain: https://i.imgur.com/uKVK5xd.png https://i.imgur.com/uKVK5xd.png I was hitting it plaintext first, so a simple redirect to some subdomain instead of a bare redirect to https would probably work fine.
- marginalia_nu 5y agoHmm, curious behavior. I'm not getting that in any of the browsers I can run. Maybe some safari-specific interpretation? I don't trust letsencrypt and I don't want to give them or anyone else a list of which subdomains I use.
- timvdalen 5y agoI can at least reproduce it in Chrome, for what it's worth. Perhaps you have the redirect to www cached?
- marginalia_nu 5y agoOn an Apple or another OS?
- seszett 5y agoI can reproduce it even just with curl on Linux. The certificate is only for "*.marginalia.nu", which simply doesn't cover "marginalia.nu". It should give the same error on any platform and browser, unless their SSL implementation is broken. Some browsers try to be smart and insert www automatically though.
- Ronsenshi 5y agoI would recommend getting certificate which includes root domain and wildcard subdomains to cover all bases.
- jffry 5y agoThe cert I am seeing [1] does include both the wildcard subdomain AND the apex domain [1] https://crt.sh/?id=6125359537 https://crt.sh/?id=6125359537
- seszett 5y agoThat's not the certificate I get served though (this one is from 5 february, the one served is from 25 january) do you actually get this certificate on https://marginalia.nu https://marginalia.nu?
- rosndo 5y agoFree unmetered bandwidth is pretty much the norm unless you have fallen for some “cloud” scam. edit: s/unlimited/unmetered/
- franga2000 5y agoBoth old-school webhosting and modern cloud hosting commonly have data caps. Unlimited data is common only on mid-to-high-end VPS or dedicated servers and unlimited bandwidth doesn't really exist.
- rndgermandude 5y ago>unlimited bandwidth doesn't really exist. True, when there is no deliberate cap stated, it's unmetered, not unlimited, meaning you can use what however much fits through the pipes (usually best effort instead of guaranteed bandwidth), and that's the natural limit. A 1 Gbit/s link for example will get you close to 300 TiB in each direction in a 30 day month if you are able to saturate that link 24/7, while a 100MBit/s link will get you a tenth so "only" close to 30TiB.
- dspillett 5y agoFree unlimited bandwidth always has a catch somewhere, that this use could trip on, unless the service is fairly expensive. There is often a “fair use policy”, and even without anything like that you need to consider the potential effect on the stuff you are actually running the service for (while a search engine is pulling random data as fast as they can, that transfer is competing against other network load). I have a couple of inexpensive hosted servers with genuinely unlimited bandwidth, but only at 100 and 250mbps respectively, better rates cost a fair chunk more without some sort of cap (usually of the form “after some-TB-or-some-tens-of-TB we'll throttle you to 100mpbs until your next billing month”. Unlimited throughput on gbit/more lines is commonly available, but never both cheap and reliable.
- rosndo 5y ago> Unlimited throughput on gbit/more lines is commonly available, but never both cheap and reliable It’s been a long time since you could call OVH “unreliable”, Hetzner chugs along just fine too. I think you might just be stuck in 2004, reliable unmetered gbit+ has been cheap for years.
- hughrr 5y agoIt’s almost worth burning my terabyte quota on Linode to try this out :)
- kenniskrag 5y agocan I do that with non existing domains or mal. login attemps in a nginx config? That would maybe slow down the attackers a little :)
- DigitallyFidget 5y agoThis is brilliant. I'm doing this for my personal site. I have a gig uplink and no data caps.
- cookiengineer 5y ago> 2Gb of garbage I serve my robots.txt files with Transfer-Encoding: gzip and Content-Encoding: gzip, chunked and then just serve a two layered 10G gzip bomb (240kB file size) instead. Can recommend. Works pretty often.
- pikseladam 5y agoyou are evil :)
- samhw 5y agoDoesn't it mostly penalise the honest bots, though? Or do some bots use robots.txt files as a proxy sitemap? I'm not that familiar with this area.
- cookiengineer 5y agoYou can also insert a Disallow: /wp-admin/ in there and just serve the gzip bomb then :D
- berkes 5y agoI have anyone requesting this automatic going to fail2ban¹, but serving a bomb at that url is actually much smarter. ¹anything .php,.asp and some other tech we never use, also phpmyadmin, cgi-bin, and some more, actually.
- hughrr 5y agoActually now you mention that I’m going to see if I can set a trap for our pen testers :)
- spookthesunset 5y agoDamn, that is a good idea. I wish I thought of that back when I ran a web forum.
- 5y ago
- altdataseller 5y agoI did something like that once and the next day my hosting provider warned me I was under a DDOS attack. Might have been a coincidence though.
- rei_ayanami 5y agodevelish
- deleted 5y ago[deleted]
- gwd 5y ago> Webmasters also put highly specific URLs they don't want the world to see in robots.txt. Also not a great idea. I can confirm; I tried to helpfully put a bunch of addresses which would be pointless for a web crawler to crawl in there (links to the login page, etc), but as soon as you say "don't" somebody's gotta go find out why. I like the idea of putting gzip bombs at commonly-scanned URLs.
- bsmartt 5y agoThat’s fascinating. But assuming you aren’t one of the top two or three search engines, wouldn’t it hurt you as much if not more to stop crawling a major site like Nike which visitors surely expect to be crawled? If you are one of the top two or three, I would like to think their crawlers are amazingly complex and robust but maybe I’m just naive.
- JoeyBananas 5y agoA parser that can't handle comments doesn't care about obeying the robots.txt in the first place.
- marginalia_nu 5y agoThere is an RFC, but in practice the format is super informally specified. You'll encounter # comments // comments <!-- comments --> /* comments */ plain text lines as comments TeX-markup script tags HTML PHP code Whatever this is: https://www.costablancaworld.com/robots.txt https://www.costablancaworld.com/robots.txt search-engine specific directives (Yandex has a few strange ones)
- colejohnson66 5y ago> Whatever this is: https://www.costablancaworld.com/robots.txt https://www.costablancaworld.com/robots.txt That's an RTF file that was saved as a TXT (without removing formatting)... https://en.wikipedia.org/wiki/Rich_Text_Format https://en.wikipedia.org/wiki/Rich_Text_Format
- supercoffee 5y ago> https://www.costablancaworld.com/robots.txt https://www.costablancaworld.com/robots.txt Somebody screwed up and made their robots.txt in a rich-text editor like wordpad.
- vmoore 5y ago> webmasters put a lot of weird crap in robots.txt Even stranger things can be found in a `humans.txt`[0] file! [0] https://humanstxt.org/ https://humanstxt.org/
- vmoore 5y ago> at least some parser will choke on them Ah the never ending joy I get by putting a 1GB file as the robots.txt or even having a 1GB `favicon.ico` file.
- marginalia_nu 5y agoDunno how other search engines do it, but I only ever read 512Kb from a connection :P
- vmoore 5y agoI once renamed a Ubuntu .ISO file to favicon.ico and placed it in the root. Some greedy bots choked on it, others had more sense and timed out.
- bhartzer 5y ago>> either ignore robots.txt or skip crawling your site. That’s not really how it works now. Google pretty much ignores robots.txt now and crawls your site anyway.
- marginalia_nu 5y agoGoogle does obey robots.txt, but there are a lot of crawlers that identify as GoogleBot that are not from google.
- zinekeller 5y agoSome web firewalls now even detect fake crawlers by using instructions published by search engines (Google: https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/..., Microsoft: https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26 https://www.bing.com/webmasters/help/how-to-verify-bingbot-3..., Yandex: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html https://yandex.com/support/webmaster/robot-workings/check-ya..., Apple: restrict to 17.0.0.0/8 (their IP range, https://support.apple.com/en-us/HT204683 https://support.apple.com/en-us/HT204683))
- Dma54rhs 5y agoIt does not, they obey it regarding indexing pages but not crawling. They can and will crawl even if you say no, they've officially said so as well in their docs.
- dwild 5y agoCan you point to your source? From a quick glance on their documentation, it seems like you got it backward. > A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page. https://developers.google.com/search/docs/advanced/robots/intro https://developers.google.com/search/docs/advanced/robots/in... They won't crawl it, but it they already did crawl it or if there's a link to it, it will still be part of their index (thus search results). They even go further on their noindex page, by saying that it need to be crawlable (thus not part of the robots.txt) so that they can see the no index directive on the page... > Important: For the noindex directive to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex directive, and the page can still appear in search results, for example if other pages link to it. https://developers.google.com/search/docs/advanced/crawling/block-indexing https://developers.google.com/search/docs/advanced/crawling/...
- DerekRobot 5y agoMy high school's was the strangest before they overhauled their site. One of the URLs was some sort of CGI spell check app: just a button and a field that said "I am spelled wrang"
- deleted 5y ago[deleted]