Due to this trend, the de facto crawler behaviour will be to do as Apple do and observe Googlebot's rules as if they were your own [1]. (That is assuming a giv

Due to this trend, the de facto crawler behaviour will be to do as Apple do and observe Googlebot's rules as if they were your own [1]. (That is assuming a given crawler follows robots.txt at all!) [1] https://support.apple.com/en-us/HT204683 https://support.apple.com/en-us/HT204683 (HN discussion was at https://news.ycombinator.com/item?id=9497264 https://news.ycombinator.com/item?id=9497264 )