3 ms·

They said "Using robots.txt as a model for anything doesn't work." but it does work for the case described in the text

• benatkin 3y ago
  What works? This is an idea, not anything that's been throughly tried.