17 ms·
This argument would be made a lot stronger with numbers... What percentage of the top million webpages allow Googlebot? What percentage allow other robots? Wh
by londons_explore 6y ago
This argument would be made a lot stronger with numbers...
What percentage of the top million webpages allow Googlebot? What percentage allow other robots?
Why not simply pretend to be Googlebot? After all, browsers pretend to be Mozilla...