6 ms·
I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools.
by bodantogat 2y ago
I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools.
I haven't found an easy way to block them. They nearly took out my site a few days ago too.
- newsclues 2y agoThe amateurs at home are going to give the big companies what they want: an excuse for government regulation.
- throwaway290 2y agoIf it doesn't say it's a bot and it doesn't come from a corporate IP it doesn't mean it's NOT a bot and not run by some "AI" company.
- bodantogat 2y agoI have no way to verify this, I suspect these are either stealth AI companies or data collectors, who hope to sell training data to them
- datadrivenangel 2y agoI've heard that some mobile SDKs / Apps earn extra revenue by providing an IP address for VPN connections / scraping.
- odo1242 2y agoChrome extensions too
- int_19h 2y agoDon't worry, the governments are perfectly capable of coming up with excuses all on their own.
- echelon 2y agoYou could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy. Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills. Maybe you don't even need a full LLM. Just a simple transformer that inverts negative and positive statements, changes nouns such as locations, and subtly nudges the content into an erroneous state.
- tyre 2y agoTheir problem is they can’t detect which are bots in the first place. If they could, they’d block them.
- echelon 2y agoThen have the users solve ARC-AGI or whatever nonsense. If the bots want your content, they'll have to solve $3,000 of compute to get it.
- Tostino 2y agoThat only works until The benchmark questions and answers are public. Which they necessarily would be in this case.
- EVa5I7bHFq9mnYK 2y agoOr maybe solve a small sha2(sha2()) leading zeroes challenge, taking ~1 second of computer time. Normal users won't notice, and bots will earn you Bitcoins :)
- marcus0x62 2y agoSelf plug, but I made this to deal with bots on my site: https://marcusb.org/hacks/quixotic.html https://marcusb.org/hacks/quixotic.html. It is a simple markov generator to obfuscate content (static-site friendly, no server-side dynamic generation required) and an optional link-maze to send incorrigible bots to 100% markov-generated non-sense (requires a server-side component.)
- kmoser 2y agoMy cheap and dirty way of dealing with bots like that is to block any IP address that accesses any URLs in robots.txt. It's not a perfect strategy but it gives me pretty good results given the simplicity to implement.
- Capricorn2481 2y agoI don't understand this. You don't have routes your users might need in robots.txt? This article is about bots accessing resources that other might use.
- IncreasePosts 2y agoIt seems better to put fake honeypot urls in robots.txt, and block any up that accesses those.
- Capricorn2481 2y agoAh I see
- trod1234 2y agoBlocking will never work. You need to impose cost. Set up QoS buckets, slow suspect connections down dramatically (almost to the point of timeout).
- Beijinger 2y agoHow can I implement this?
- aorth 2y agoAnother related idea: use fail2ban to monitor the server access logs. There is one filter that will ban hosts that request non-existent URLs like WordPress login and other PHP files. If your server is not hosting PHP at all it's an obvious sign that the requests are from bots that are probing maliciously.
- 2y ago
- _lvbh 2y agoTLS fingerprinting still beats most of them. For really high compute endpoints I suppose some sort of JavaScript challenge would be necessary. Quite annoying to set up yourself. I hate cloudflare as a visitor but they do make life so much easier for administrators
- petre 2y agoYou rate limit them and then block the abusers. Nginx allows rate limiting. You can then block them using fail2ban for an hour if they're rate limited 3 times. If they get blocked 5 times you can block them forever using the recidive jail. I've had massive AI bot traffic from M$, blocked several IPs by adding manual entries into the recidive jail. If they come back and disregard robots.txt with disallow * I will run 'em through fail2ban.
- herbst 2y agoWhatever M$ was doing still baffles me. I still have several azure ranges in my blocklist because whatever this was appeared to change strategie once I implemented a ban method.
- petre 2y agoThey were hammering our closed ticketing system for some reason. I blocked an entire C block and an individual IP. If needed I will not hesitate banning all their ranges, which means we won't get any mail from Azure, M$ office 365, since this is also our mail server. But scew'em, I'll do it anyway until someone notices, since it's clearly abuse.