10 ms·
Next time drop a symlink from robots.txt to /dev/urandom and see how it goes.
by 5e92cb50239222b 5y ago
Next time drop a symlink from robots.txt to /dev/urandom and see how it goes.
- dspillett 5y agoYou might want to rate limit that if you don't magically have free unlimited bandwidth quota.
- marginalia_nu 5y agoI hear it's decades of bad luck to do unless you check that the crawler's user-agent isn't "search.marginalia.nu"; but you could just tarpit the connection.
- dijit 5y agoWhile I'm in general not a fan of wildcard certificates, yours doesn't work for the bare domain: https://i.imgur.com/uKVK5xd.png https://i.imgur.com/uKVK5xd.png I was hitting it plaintext first, so a simple redirect to some subdomain instead of a bare redirect to https would probably work fine.
- marginalia_nu 5y agoHmm, curious behavior. I'm not getting that in any of the browsers I can run. Maybe some safari-specific interpretation? I don't trust letsencrypt and I don't want to give them or anyone else a list of which subdomains I use.
- timvdalen 5y agoI can at least reproduce it in Chrome, for what it's worth. Perhaps you have the redirect to www cached?
- marginalia_nu 5y agoOn an Apple or another OS?
- seszett 5y agoI can reproduce it even just with curl on Linux. The certificate is only for "*.marginalia.nu", which simply doesn't cover "marginalia.nu". It should give the same error on any platform and browser, unless their SSL implementation is broken. Some browsers try to be smart and insert www automatically though.
- Ronsenshi 5y agoI would recommend getting certificate which includes root domain and wildcard subdomains to cover all bases.
- jffry 5y agoThe cert I am seeing [1] does include both the wildcard subdomain AND the apex domain [1] https://crt.sh/?id=6125359537 https://crt.sh/?id=6125359537
- seszett 5y agoThat's not the certificate I get served though (this one is from 5 february, the one served is from 25 january) do you actually get this certificate on https://marginalia.nu https://marginalia.nu?
- jffry 5y agoI misunderstood what URL we were talking about. What I am seeing: https://marginalia.nu https://marginalia.nu = https://crt.sh/?id=6046506678 https://crt.sh/?id=6046506678 (includes wildcard but NOT apex) https://search.marginalia.nu https://search.marginalia.nu = https://crt.sh/?id=6125359537 https://crt.sh/?id=6125359537 (includes both wildcard and apex)
- marginalia_nu 5y agoAh, I see what's happening. search.marginalia.nu goes through cloudflare which does its creepy MITM thing and re-encrypts on its end; the others do not.
- wizzwizz4 5y agoWhy does marginalia.nu use Cloudflare? Have you been hit by DDOSs? I can't think how a search engine would benefit from Cloudflare caching things.
- input_sh 5y agoIt's simple, your cert points to *.domain.tld, domain.tld doesn't have a subdomain, <anything>.domain.tld does. You should expand it to include both domain.tld and *.domain.tld.
- jffry 5y agoIt seems like a bad cert is being served at the redirect; the LE cert at https://search.marginalia.nu https://search.marginalia.nu DOES have both the wildcard and apex domain See https://news.ycombinator.com/item?id=30301015 https://news.ycombinator.com/item?id=30301015
- MonaroVXR 5y ago> .domain.tld Oof, I'm not sure if I would do that. Since "" stands for all sub domains.
- jffry 5y agoThere may be some type of caching problem. When I visit https://marginalia.nu https://marginalia.nu directly, I'm served with this cert, which does not include the apex domain: https://crt.sh/?id=6046506678 https://crt.sh/?id=6046506678 When I visit https://search.marginalia.nu https://search.marginalia.nu I'm served with this different cert, which does include both wildcard and apex: https://crt.sh/?id=6125359537 https://crt.sh/?id=6125359537
- deleted 5y ago[deleted]
- rosndo 5y agoFree unmetered bandwidth is pretty much the norm unless you have fallen for some “cloud” scam. edit: s/unlimited/unmetered/
- franga2000 5y agoBoth old-school webhosting and modern cloud hosting commonly have data caps. Unlimited data is common only on mid-to-high-end VPS or dedicated servers and unlimited bandwidth doesn't really exist.
- rndgermandude 5y ago>unlimited bandwidth doesn't really exist. True, when there is no deliberate cap stated, it's unmetered, not unlimited, meaning you can use what however much fits through the pipes (usually best effort instead of guaranteed bandwidth), and that's the natural limit. A 1 Gbit/s link for example will get you close to 300 TiB in each direction in a 30 day month if you are able to saturate that link 24/7, while a 100MBit/s link will get you a tenth so "only" close to 30TiB.
- dspillett 5y agoFree unlimited bandwidth always has a catch somewhere, that this use could trip on, unless the service is fairly expensive. There is often a “fair use policy”, and even without anything like that you need to consider the potential effect on the stuff you are actually running the service for (while a search engine is pulling random data as fast as they can, that transfer is competing against other network load). I have a couple of inexpensive hosted servers with genuinely unlimited bandwidth, but only at 100 and 250mbps respectively, better rates cost a fair chunk more without some sort of cap (usually of the form “after some-TB-or-some-tens-of-TB we'll throttle you to 100mpbs until your next billing month”. Unlimited throughput on gbit/more lines is commonly available, but never both cheap and reliable.
- rosndo 5y ago> Unlimited throughput on gbit/more lines is commonly available, but never both cheap and reliable It’s been a long time since you could call OVH “unreliable”, Hetzner chugs along just fine too. I think you might just be stuck in 2004, reliable unmetered gbit+ has been cheap for years.
- hughrr 5y agoIt’s almost worth burning my terabyte quota on Linode to try this out :)
- kenniskrag 5y agocan I do that with non existing domains or mal. login attemps in a nginx config? That would maybe slow down the attackers a little :)
- DigitallyFidget 5y agoThis is brilliant. I'm doing this for my personal site. I have a gig uplink and no data caps.