4 ms·
Is there anything like gitignore.io for robots.txt files? Like, select a set of known bots and get a list of rules tailored to them. Bytedance's crawler should
by extraduder_ire 3y ago
Is there anything like gitignore.io for robots.txt files? Like, select a set of known bots and get a list of rules tailored to them.
Bytedance's crawler should probably be blocked too, based on a recent mattKC video.
I assume google's training bard with their search engine crawling data, so you have an incentive to allow them.