2 ms·
Volume and incentives. Before this LLM craze, the biggest crawlers were search engines. They had a motivation to not bring down their targets, because who need
by pona-a 2y ago
Volume and incentives.
Before this LLM craze, the biggest crawlers were search engines. They had a motivation to not bring down their targets, because who needs an index full of dead links. With LLM crawlers, all you need is text, and if the site is forced to shut down because of you, that's just less data for your competitors.
- throw10920 2y agoAlso, nobody else steals your stuff like AI does. Doesn't take much thought to figure out the difference.