4 ms·
“… he complained that LLM companies were crawling data without respecting robosts.txt“ Maybe that is the problem? They misspelled robots.txt
by cudgy 2y ago
“… he complained that LLM companies were crawling data without respecting robosts.txt“
Maybe that is the problem? They misspelled robots.txt