3 ms·I think it's valid to consider robots.txt to be a part of a site's API.by scott_s 16y agoI think it's valid to consider robots.txt to be a part of a site's API.