2 ms·
long, long ago I had several GB of text archives up on the web, with the internal links all scripted out as relative paths (like "../../some/file.txt"). The who
by h2odragon 5y ago
long, long ago I had several GB of text archives up on the web, with the internal links all scripted out as relative paths (like "../../some/file.txt"). The whole thing was visible different ways on different networks and this kept things simpler with multiple views that way.
Many web crawlers had trouble with that. several didn't care about `robots.txt` then either and so i had firewall blocks for some of them.
I don't recall the MS bot's name from then but i do recall it had a charming failure mode where it prepended "./" to a url already beginning with one, and started crawling that as a new tree. I actually wrote them an email about path normalization then i think.