3 ms·
He explains how and why in the article, and gives arguments. You do none. The problem with whitelist-only robots.txt is that they favor monopolies and startups
by Aissen 14y ago
He explains how and why in the article, and gives arguments. You do none.
The problem with whitelist-only robots.txt is that they favor monopolies and startups are the ones getting the "fuck you". But maybe you don't care about that.