4 ms·
From their html source: <meta name="robots" content="noindex" /> <meta name="ROBOTS" content="NOINDEX, FOLLOW" /> <meta name="ROBOTS" content="INDEX, NOF
by singlow 10y ago
From their html source:
<meta name="robots" content="noindex" />
<meta name="ROBOTS" content="NOINDEX, FOLLOW" />
<meta name="ROBOTS" content="INDEX, NOFOLLOW" />
<meta name="ROBOTS" content="NOINDEX, NOFOLLOW" />
Not sure what they are intending here. But my guess is they don't either.
[edit: submitted too soon by mistake]
- tremon 10y agoI think they mean that case-sensitive crawlers are disallowed, and they reserve the right to arbitrarily sue case-insensitive crawlers under the CFAA.