3 ms·

Do you even need to match Google's robots.txt parsing behavior? With less than 1000 lines you can be pretty sure they are not doing it right and are breaking pl

Do you even need to match Google's robots.txt parsing behavior? With less than 1000 lines you can be pretty sure they are not doing it right and are breaking plenty of people's assumptions about it. Either way you have to test it on real world data.

joshuamorton 7y ago
> Do you even need to match Google's robots.txt parsing behavior? With less than 1000 lines you can be pretty sure they are not doing it right and are breaking plenty of people's assumptions about it. This seems like a weird assertion. The specification isn't particularly complex (ignoring the implicit complexities of unicode). There are ~5 keywords and like 3 control characters. Why would you expect to need all that much?
- zzzcpan 7y ago
  Very few people follow the specification or even know it exists.
  - joshuamorton 7y ago
    I'm not talking about the formal specification, but the implicit specification of what people have been using for decades. That only has 5 keywords and a couple control characters. The formal spec is based on that informal spec, which again, isn't that complicated. To be more direct: what are all of these assumptions you assume google's parser is mishandling?
    - zzzcpan 7y ago
      Top comment [1] talks about noindex directive for example. Some people definitely expect it to work. [1] https://news.ycombinator.com/item?id=20326098 https://news.ycombinator.com/item?id=20326098
jerf 7y ago
The point of this code release seems to be to release Google's precise logic. That you may incorporate it into something else is, IMHO, less interesting; we've got plenty of other solutions that "do robots.txt" well enough. If it was just about that, Google's release of this would not be worth anything. The point is so that non-Google parties can see exactly what Google is seeing in your robots.txt. That's why I'm saying there's no point trying to re-implement this. If you were going to re-implement this, there's probably already a library that will work well enough for you. The value here is solely in being exactly what Google uses; anything that is a "re-implementation" of this code but isn't exactly what Google uses is missing the point. If they formalize it into a spec, others may then implement the spec, but they can and should do that by implementing the spec, not porting this code.
- zzzcpan 7y ago
  As I understand the point about Go complaint is to parse actual real world robots.txt. For which you don't need to behave exactly as this library does.

joshuamorton 7y ago

> Do you even need to match Google's robots.txt parsing behavior? With less than 1000 lines you can be pretty sure they are not doing it right and are breaking plenty of people's assumptions about it. This seems like a weird assertion. The specification isn't particularly complex (ignoring the implicit complexities of unicode). There are ~5 keywords and like 3 control characters. Why would you expect to need all that much?

zzzcpan 7y ago
Very few people follow the specification or even know it exists.
- joshuamorton 7y ago
  I'm not talking about the formal specification, but the implicit specification of what people have been using for decades. That only has 5 keywords and a couple control characters. The formal spec is based on that informal spec, which again, isn't that complicated. To be more direct: what are all of these assumptions you assume google's parser is mishandling?
  - zzzcpan 7y ago
    Top comment [1] talks about noindex directive for example. Some people definitely expect it to work. [1] https://news.ycombinator.com/item?id=20326098 https://news.ycombinator.com/item?id=20326098

jerf 7y ago

The point of this code release seems to be to release Google's precise logic. That you may incorporate it into something else is, IMHO, less interesting; we've got plenty of other solutions that "do robots.txt" well enough. If it was just about that, Google's release of this would not be worth anything. The point is so that non-Google parties can see exactly what Google is seeing in your robots.txt. That's why I'm saying there's no point trying to re-implement this. If you were going to re-implement this, there's probably already a library that will work well enough for you. The value here is solely in being exactly what Google uses; anything that is a "re-implementation" of this code but isn't exactly what Google uses is missing the point. If they formalize it into a spec, others may then implement the spec, but they can and should do that by implementing the spec, not porting this code.

zzzcpan 7y ago
As I understand the point about Go complaint is to parse actual real world robots.txt. For which you don't need to behave exactly as this library does.