3 ms·
I don't understand how these pages could have been crawled - could someone enlighten?
by cleverjake 14y ago
I don't understand how these pages could have been crawled - could someone enlighten?
- nico-roddz 14y agoIt's seems that Facebook uses robots.txt to block this pages https://www.facebook.com/robots.txt https://www.facebook.com/robots.txt But, depending of the amount of inbound links, Google will index the urls anyway. It's a common issue.
- cleverjake 14y agoGoogle ignores robots.txt if the number of inbound links is > N? Also - any speculation as to how so many sites were lining to peoples login pages?