4 ms·

My personal experience is that OpenAI's crawler was hitting a very, very low traffic website I manage 10s of 1000s of times a minute non-stop. I had to block it

My personal experience is that OpenAI's crawler was hitting a very, very low traffic website I manage 10s of 1000s of times a minute non-stop. I had to block it from Cloudflare.

danaris 1y ago
Same here. I run a very small browser game (~120 weekly users currently), and until I put its Wiki (utterly uninteresting to anyone who doesn't already play the game) behind a login-wall, the bots were causing massive amounts of spurious traffic. Due to some of the Wiki's data coming live from the game through external data feeds, the deluge of bots actually managed to crash the game several times, necessitating a restart of the MariaDB process.
- mrweasel 1y ago
  Wikis seems to attract AI bots like crazy, especially the bad kind that will attempt any type of cache invalidation available to them.
Leynos 1y ago
Where is caching breaking so badly that this is happening? Are OpenAI failing to use etags or honour cache validity?
- Analemma_ 1y ago
  Their crawler is vibe-coded.
- deleted 1y ago
  [deleted]