3 ms·

Is robots.txt still a thing?

• dotancohen 11mo ago
  It is. A typically ignored thing.
  • culi 11mo ago
    Maybe less so soon with content-signals https://contentsignals.org/ https://contentsignals.org/
    • dotancohen 11mo ago
      With no enforcement mechanism?
      • culi 11mo ago
        Some European laws give robots.txt files some legal weight. That's why you often see this in robots.txt that have content-signals # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.