5 ms·

Google is also really generous with how they will let you spell "disallow": https://github.com/google/robotstxt/blob/59f3643d3a3ac88f613326dd4dfc8c9b9a545e45/ro

• knorker 7y ago
  I'm not surprised. Some people think humans read robots.txt and get super angry when the crawler doesn't understand.
  • c3534l 7y ago
    I read robots.txt, but I'm not a massive corporation.
  • deleted 7y ago
    [deleted]
• kortilla 7y ago
  > (absl::StartsWithIgnoreCase(key, "disallaw"))))); Ah, the southern version. :)
• Fnoord 7y ago
  Makes sense because not everyone speaks English as native language. Disalow is pretty close to disallow, phonetically.
  • ryandrake 7y ago
    Yuck though! Imagine if you were writing a compiler. Would you make it accept “unsinged” “unnsigned” “unssined” and “unsined” as keywords, just to catch spelling mistakes? Not sure I like that pattern.
    • bscphil 7y ago
      It's a little different in that case, since the person using the parser is also the person writing the input to the parser. So if the input fails the parser, the author of the code can simply correct it. As I understand it, there's no single standard that captures how all robots.txt files are formatted, so there's no "standard parser" that the authors of these files could be expected to pass.
  • anoncake 7y ago
    That is not an excuse. Non-native speakers can learn to spell.
• avip 7y ago
  This is great. #kAllowFrequentTypos
  • samstave 7y ago
    Ive never made a tpyo in my life!!!
    • quickthrower2 7y ago
      Whats a tpyo? My typo array for "typo" only has tipo, typpo and thai-pho
      • bigiain 7y ago
        If you're gonna include thai-pho you also need to include thai-fur...