3 ms·

Surprisingly the content of robots.txt is: User-agent: * Disallow: / http://webplatform.org/robots.txt http://webplatform.org/robots.txt

• wwwtyro 14y ago
  I can't begin to fathom a motivation for that. Can anyone think of a reason, or is this likely simply a mistake?
  • desbest 14y ago
    Maybe someone from Facebook made the robots.txt file, ayyy!
  • criswell 14y ago
    The site doesn't even seem close to being useful or complete yet. I think it's intentional.
  • acdha 14y ago
    They didn't want search traffic until the content and structure start to stabilize? Google, et al. will crawl for years after a URL 404s so it might make sense to hold off until the first round of stabilization
  • thenonsequitur 14y ago
    The site was in super-alpha. I think it was intentional.
• HendrikR 14y ago
  This has already been changed to #User-agent: * #Disallow: /