3 ms·
One funny thing I've discovered as a result of certificate transparency logs is that the second your host gets given an SSL cert, you are immediately blasted wi
by nemothekid 4mo ago
One funny thing I've discovered as a result of certificate transparency logs is that the second your host gets given an SSL cert, you are immediately blasted with ai crawlers.
I put a project online - it was online for a month, and the second I added an SSL cert it went from 0 traffic to 1000 requests/min.
- CyberDildonics 4mo agoMake a new certificate, let crawlers blast you and add those IPs to a block list.
- nikcub 4mo agothese old network security techniques don't really work anymore. the common bots are at known IP ranges, the problem bots are all on datacenter + residential proxies.
- CyberDildonics 4mo agoWhy would blocking those be a problem?
- nikcub 4mo agothere are 150M+ of them and you'll be taking out a lot of human users with it modern blocking is behaviour / heuristic based
- CyberDildonics 4mo agoThere are 150 million bots all using residential IP addresses?
- chadgpt3 4mo agobecause you are blocking all of Comcast, Verizon, T-Mobile, British Telecom, .... at the end you have blocked every network with human visitors and only datacenter IPs can access your site. The proxies rotate IP every day, so you either have ineffective blocking or you block the whole network.
- mh- 4mo agoIn my experience, these aren't the crawlers from legit companies, so they have infinite IPs via residential botnets/proxies. edit: 'nikcub beat me to it by 30 seconds :)
- 8cvor6j844qw_d6 4mo ago> One funny thing I've discovered as a result of certificate transparency logs is that the second your host gets given an SSL cert I've been thinking of using wildcard certs for Caddy in regards to this.
- jgalt212 4mo agoand then what? serve your app under some obscure / customer unfriendly subdomain?
- sadeshmukh 4mo agoEven if you use a common subdomain, anecdotally I get orders of magnitude less bot traffic than not using a wildcard cert.
- RajT88 4mo agoToday AI crawlers, years ago vulnerability scanners from Russia or China. Either way! People monitor cert registries for targets.