3 ms·
> The web is increasingly turning into closed platform unindexable silos. So it is the quality of the open web itself that has been declining, and search is ju
by mattmcknight 3y ago
> The web is increasingly turning into closed platform unindexable silos.
So it is the quality of the open web itself that has been declining, and search is just reflection of that. LLM doesn't solve that, unless the LLM has access across the silos.
- maven29 3y agoLLM training is difficult to audit, and as such good-faith barriers such as robots.txt or API endpoint scraping are all easily ignored. Even ignoring the query side, and just from an indexing perspecrive, you see a considerable advantage. LLMs (at runtime) and LLM mills (during training) have nothing to gain from restricting themselves to the open web. This is in contrast to Google, who is in the business of selling links to information and not information in itself.