Yeah, the https://en.m.wikipedia.org/wiki/Robots_exclusion_standard https://en.m.wikipedia.org/wiki/Robots_exclusion_standard dates from a more civilized time,

Yeah, the https://en.m.wikipedia.org/wiki/Robots_exclusion_standard https://en.m.wikipedia.org/wiki/Robots_exclusion_standard dates from a more civilized time, when URLs were more, erm, uniformly structured.