3 ms·

So, market/companies refused to regulate themselves (by adhering to the robots.txt) so we're now forced to innovate some solutions against them.