2 ms·
It's like a if/elsif/else statement. `User-agent: *` means for all other bots that didn't match any section above. Here are the rules for Google: User-ag
by lcedp 13y ago
It's like a if/elsif/else statement.
`User-agent: *` means for all other bots that didn't match any section above.
Here are the rules for Google:
User-agent: Googlebot
Disallow: /ac.php
Disallow: /ae.php
Disallow: /ajax/
Disallow: /album.php
Disallow: /ap.php
Disallow: /autologin.php
Disallow: /checkpoint/
Disallow: /confirmemail.php
Disallow: /contact_importer/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /l.php
Disallow: /o.php
Disallow: /p.php
Disallow: /photo.php
Disallow: /photo_comments.php
Disallow: /photo_search.php
Disallow: /photos.php
Disallow: /sharer/
- meaning anything else allowed.