6 ms·
I coded a few robots.txt parsers i.e.: (these are just the ones i cared to publish) https://github.com/franzenzenhofer/simple-functional-robots-txt-parser htt
by franze 6y ago
I coded a few robots.txt parsers
i.e.: (these are just the ones i cared to publish)
https://github.com/franzenzenhofer/simple-functional-robots-txt-parser https://github.com/franzenzenhofer/simple-functional-robots-...
https://github.com/franzenzenhofer/robotstxt https://github.com/franzenzenhofer/robotstxt
and even though robots.txt seems like a very, very simple text based protocoll there are unanswered mysteries
the biggest mystery is user agent groups and comments
i.e.:
User-agent: googlebot
User-agent: bing
User-agent: yandex
Disallow: /
so i am disallowing everything for google, bing, yandex; easy enough, but:
User-agent: googlebot
User-agent: yandex
Disallow: /
means that googlebot has no instructions, but yandex is disallowed all.
if a whole line is commented out
User-agent: googlebot
#User-agent: bing
User-agent: yandex
Disallow: /
is a commented out line a blank line or a non existing line?
if it is interpreted as a blank linke, the disallow only counts for yandex, if the line is non existant, it counts for googlebot and yandex.
i like simple things, but sometimes the complexity is between the lines.
- ThreeFx 6y agoWhat's the solution? Semantically it makes sense if it counts for both googlebot and yandex. Have you observed one/both behaviours?
- franze 6y agoi am consulting a lot of aggregators, and we coose to interpret sometimes as this, sometimes in the other scenario. the way i interpret the spec, it is not defined behaviour.
- iams 6y agoUser-agent: googlebot #User-agent: bing User-agent: yandex Disallow: / why would it be interpreted as a blank line? If you remove everything after the #, that includes the new line characters at the end of the line. Leaving: User-agent: googlebot User-agent: yandex Disallow: /
- franze 6y agohere the problem comes in, as people write User-agent: googlebot User-agent: bing #behaved-badly User-agent: yandex Disallow: / if # removes everythin afer the #, then it would be User-agent: googlebot User-agent: bing User-agent: yandex Disallow: / which results into the whole line " User-agent: bing User-agent: yandex" beeing thrown out as malformed, so only googlebot would be disallowed.
- eyelidlessness 6y agoIt's ambiguous, and this is why significant whitespace can be so frustrating. Unless it's specified, some people will interpret a full-line comment as a blank line with a comment ending it (`^#.$`), and others will interpret it as you have (`^#.?\n`). Neither is obviously correct (even if it's obvious to you). Edit: I don't know how to escape in HN formatting. Obviously there are italics where literal asterisks should be.
- traes 6y ago*** You can just use three asterisks. *** * You can just use three asterisks. * Unfortunately you need something after them though. *** Unfortunately you need something after them though.
- eyelidlessness 6y agoThanks, that's good to know!
- DoreenMichele 6y agoIf you indent it two spaces, I think that should solve that problem: *(`^#.$`), and others will interpret it as you have (`^#.?\n`)* (Not sure where you intended those asterisks. I made my best inference.)
- eyelidlessness 6y agoYeah I was hoping to keep them inline, but indenting surely would have helped. I intended them after the dots (`.*`, dot asterisk in case I get the escaping wrong, is 0 or more characters in regex).