3 ms·
The LLM crawler just needs to respect robots.txt and they won't be affected. I don't understand how anyone can find this to be a problem.
by threeseed 2y ago
The LLM crawler just needs to respect robots.txt and they won't be affected.
I don't understand how anyone can find this to be a problem.