3 ms·
Our bot traffic is up 10-fold since LLM Cambrian explosion.
by jgalt212 2y ago
Our bot traffic is up 10-fold since LLM Cambrian explosion.
- parpfish 2y agoCambrian explosion implies that there’s a huge variety of different creatures out there, but I suspect those bots are all just wrappers around OpenAI/anthropic models. This is more like the rise of Cyanobacteria as a single early dominant lifeform
- visarga 2y agoThere are 112,391 language models on HuggingFace, most of them fine-tunes of a few base models, but still, a staggering number.
- simonw 2y agoWriting a crawler that's a wrapper around OpenAI or Anthropic doesn't make sense to me: what is your crawler doing? Piping all that crawler data through an existing LLM would cost you millions of dollars, and for what purpose? Crawling to train your own LLM from scratch makes a lot more sense.
- AshamedCaptain 2y agoI agree. I used to have a website serving some code and some tarballs of my software. I used to be able to handle the traffic (including from ALL Linux distributions, who are packaging this software) from a home server and home connection, over for the 30+ years I've been serving it. In the last few months, there's so much crawler traffic (specially going over all the source files over and over), ignoring crawl-delay and the entirety of robots.txt , that they have brought the server down more than once.