4 ms·
It's a stretch to expect a human initiated action to abide by robot.txt. Also, once you click on a link in chrome it's pretty much all robot parsed and rendere
by jpambrun 3y ago
It's a stretch to expect a human initiated action to abide by robot.txt.
Also, once you click on a link in chrome it's pretty much all robot parsed and rendered from there as well..
- colonelspace 3y agoAt bottom, all robot actions are human initiated.
- s3p 2y agoExactly, so they should ignore robots.txt !