3 ms·
A parser that can't handle comments doesn't care about obeying the robots.txt in the first place.
by JoeyBananas 5y ago
A parser that can't handle comments doesn't care about obeying the robots.txt in the first place.
- marginalia_nu 5y agoThere is an RFC, but in practice the format is super informally specified. You'll encounter # comments // comments <!-- comments --> /* comments */ plain text lines as comments TeX-markup script tags HTML PHP code Whatever this is: https://www.costablancaworld.com/robots.txt https://www.costablancaworld.com/robots.txt search-engine specific directives (Yandex has a few strange ones)
- colejohnson66 5y ago> Whatever this is: https://www.costablancaworld.com/robots.txt https://www.costablancaworld.com/robots.txt That's an RTF file that was saved as a TXT (without removing formatting)... https://en.wikipedia.org/wiki/Rich_Text_Format https://en.wikipedia.org/wiki/Rich_Text_Format
- supercoffee 5y ago> https://www.costablancaworld.com/robots.txt https://www.costablancaworld.com/robots.txt Somebody screwed up and made their robots.txt in a rich-text editor like wordpad.