3 ms·

Which they ultimately should do when crawling a site. OpenAI does according to ChatGPT.

• warrenm 4y ago
  as a coworker of mine liked to say, "'should' is a funny word"