7 ms·
I made my robots.txt 2Gb of garbage from dev random to see how many spiders I could break. At least three Chinese ones downloaded the whole file.
by hughrr 5y ago
I made my robots.txt 2Gb of garbage from dev random to see how many spiders I could break. At least three Chinese ones downloaded the whole file.
- 5e92cb50239222b 5y agoNext time drop a symlink from robots.txt to /dev/urandom and see how it goes.
- dspillett 5y agoYou might want to rate limit that if you don't magically have free unlimited bandwidth quota.
- marginalia_nu 5y agoI hear it's decades of bad luck to do unless you check that the crawler's user-agent isn't "search.marginalia.nu"; but you could just tarpit the connection.
- dijit 5y agoWhile I'm in general not a fan of wildcard certificates, yours doesn't work for the bare domain: https://i.imgur.com/uKVK5xd.png https://i.imgur.com/uKVK5xd.png I was hitting it plaintext first, so a simple redirect to some subdomain instead of a bare redirect to https would probably work fine.
- marginalia_nu 5y agoHmm, curious behavior. I'm not getting that in any of the browsers I can run. Maybe some safari-specific interpretation? I don't trust letsencrypt and I don't want to give them or anyone else a list of which subdomains I use.
- timvdalen 5y agoI can at least reproduce it in Chrome, for what it's worth. Perhaps you have the redirect to www cached?
- marginalia_nu 5y agoOn an Apple or another OS?
- seszett 5y agoI can reproduce it even just with curl on Linux. The certificate is only for "*.marginalia.nu", which simply doesn't cover "marginalia.nu". It should give the same error on any platform and browser, unless their SSL implementation is broken. Some browsers try to be smart and insert www automatically though.
- Ronsenshi 5y agoI would recommend getting certificate which includes root domain and wildcard subdomains to cover all bases.
- jffry 5y agoThe cert I am seeing [1] does include both the wildcard subdomain AND the apex domain [1] https://crt.sh/?id=6125359537 https://crt.sh/?id=6125359537
- seszett 5y agoThat's not the certificate I get served though (this one is from 5 february, the one served is from 25 january) do you actually get this certificate on https://marginalia.nu https://marginalia.nu?
- jffry 5y agoI misunderstood what URL we were talking about. What I am seeing: https://marginalia.nu https://marginalia.nu = https://crt.sh/?id=6046506678 https://crt.sh/?id=6046506678 (includes wildcard but NOT apex) https://search.marginalia.nu https://search.marginalia.nu = https://crt.sh/?id=6125359537 https://crt.sh/?id=6125359537 (includes both wildcard and apex)
- rosndo 5y agoFree unmetered bandwidth is pretty much the norm unless you have fallen for some “cloud” scam. edit: s/unlimited/unmetered/
- franga2000 5y agoBoth old-school webhosting and modern cloud hosting commonly have data caps. Unlimited data is common only on mid-to-high-end VPS or dedicated servers and unlimited bandwidth doesn't really exist.
- rndgermandude 5y ago>unlimited bandwidth doesn't really exist. True, when there is no deliberate cap stated, it's unmetered, not unlimited, meaning you can use what however much fits through the pipes (usually best effort instead of guaranteed bandwidth), and that's the natural limit. A 1 Gbit/s link for example will get you close to 300 TiB in each direction in a 30 day month if you are able to saturate that link 24/7, while a 100MBit/s link will get you a tenth so "only" close to 30TiB.
- dspillett 5y agoFree unlimited bandwidth always has a catch somewhere, that this use could trip on, unless the service is fairly expensive. There is often a “fair use policy”, and even without anything like that you need to consider the potential effect on the stuff you are actually running the service for (while a search engine is pulling random data as fast as they can, that transfer is competing against other network load). I have a couple of inexpensive hosted servers with genuinely unlimited bandwidth, but only at 100 and 250mbps respectively, better rates cost a fair chunk more without some sort of cap (usually of the form “after some-TB-or-some-tens-of-TB we'll throttle you to 100mpbs until your next billing month”. Unlimited throughput on gbit/more lines is commonly available, but never both cheap and reliable.
- rosndo 5y ago> Unlimited throughput on gbit/more lines is commonly available, but never both cheap and reliable It’s been a long time since you could call OVH “unreliable”, Hetzner chugs along just fine too. I think you might just be stuck in 2004, reliable unmetered gbit+ has been cheap for years.
- hughrr 5y agoIt’s almost worth burning my terabyte quota on Linode to try this out :)
- kenniskrag 5y agocan I do that with non existing domains or mal. login attemps in a nginx config? That would maybe slow down the attackers a little :)
- DigitallyFidget 5y agoThis is brilliant. I'm doing this for my personal site. I have a gig uplink and no data caps.
- cookiengineer 5y ago> 2Gb of garbage I serve my robots.txt files with Transfer-Encoding: gzip and Content-Encoding: gzip, chunked and then just serve a two layered 10G gzip bomb (240kB file size) instead. Can recommend. Works pretty often.
- pikseladam 5y agoyou are evil :)
- samhw 5y agoDoesn't it mostly penalise the honest bots, though? Or do some bots use robots.txt files as a proxy sitemap? I'm not that familiar with this area.
- cookiengineer 5y agoYou can also insert a Disallow: /wp-admin/ in there and just serve the gzip bomb then :D
- berkes 5y agoI have anyone requesting this automatic going to fail2ban¹, but serving a bomb at that url is actually much smarter. ¹anything .php,.asp and some other tech we never use, also phpmyadmin, cgi-bin, and some more, actually.
- hughrr 5y agoActually now you mention that I’m going to see if I can set a trap for our pen testers :)
- spookthesunset 5y agoDamn, that is a good idea. I wish I thought of that back when I ran a web forum.
- 5y ago
- altdataseller 5y agoI did something like that once and the next day my hosting provider warned me I was under a DDOS attack. Might have been a coincidence though.
- rei_ayanami 5y agodevelish
- deleted 5y ago[deleted]