3 ms·
Hopefully this could be market regulated, if a particular crawler gets caught making things up or doing otherwise a bad job crawling, indexers could simply stop
by MadWombat 7y ago
Hopefully this could be market regulated, if a particular crawler gets caught making things up or doing otherwise a bad job crawling, indexers could simply stop using it in favor of a different crawler.