4 ms·
What if content providers reduced the 30k word page for a recipe down to just the actual recipe, would this reduce the amount of data these bots are pulling dow
by tmaly 1y ago
What if content providers reduced the 30k word page for a recipe down to just the actual recipe, would this reduce the amount of data these bots are pulling down?
I don't see this slowing down. If websites don't adapt to the AI deep search reality, the bot will just go somewhere else. People don't want to read these massive long form pages geared at outdated Google SEO techniques.
- Fade_Dance 1y agoYou're painting this as a problem that is somehow related to overly long form text based web pages. It isn't. If you host a local cleaning company site, or a game walkthrough site, or a roleplaying forum, the bots will flood the gates all the same. You are right that it doesn't look like it is slowing down, but the developing result of this will not be people posting a shorter recipe, it will be a further contraction of the public facing, open internet.
- hombre_fatal 1y agoKinda funny to see someone casually mention a roleplaying forum. That's what I run and it got 10x traffic overnight from AI bots. Made it when I was a teenager and got stuck running it the rest of my life. Of course, the bots go super deep into the site and bust your cache.
- lawlessone 1y agoyou'd be within your right to serve the bots fake gibberish data. Maybe they'll crawl less when it starts damaging models.
- Analemma_ 1y agoThat's only a stopgap measure, eventually they'll realize what's happening and use distributed IPs and fake user agents to look like normal users. The Tencent and Bytedance scrapers are already doing this.
- gowld 1y agoText content at the sub-page level is approximately 0% of web traffic. It's a non-issue.
- immibis 1y agoThen remove the 30MB of advertisements, serve just the 3kB of text, and your server load will be completely fine.