4 ms·
You're painting this as a problem that is somehow related to overly long form text based web pages. It isn't. If you host a local cleaning company site, or a ga
by Fade_Dance 1y ago
You're painting this as a problem that is somehow related to overly long form text based web pages. It isn't. If you host a local cleaning company site, or a game walkthrough site, or a roleplaying forum, the bots will flood the gates all the same.
You are right that it doesn't look like it is slowing down, but the developing result of this will not be people posting a shorter recipe, it will be a further contraction of the public facing, open internet.
- hombre_fatal 1y agoKinda funny to see someone casually mention a roleplaying forum. That's what I run and it got 10x traffic overnight from AI bots. Made it when I was a teenager and got stuck running it the rest of my life. Of course, the bots go super deep into the site and bust your cache.
- lawlessone 1y agoyou'd be within your right to serve the bots fake gibberish data. Maybe they'll crawl less when it starts damaging models.
- Analemma_ 1y agoThat's only a stopgap measure, eventually they'll realize what's happening and use distributed IPs and fake user agents to look like normal users. The Tencent and Bytedance scrapers are already doing this.