4 ms·
BBC News Robots.txt
- JKFSOM 1y agoThe BBC News robots.txt file includes an interesting selection of disallowed pages: - "Jailed refugee wins right to stay" (http://news.bbc.co.uk/1/hi/england/1678616.stm http://news.bbc.co.uk/1/hi/england/1678616.stm) - "Teacher's HIV sack claim rejected" (http://news.bbc.co.uk/1/hi/england/beds/bucks/herts/3663494.stm http://news.bbc.co.uk/1/hi/england/beds/bucks/herts/3663494....) - "Should the UK vote on the EU constitution?" subpage (http://news.bbc.co.uk/1/shared/spl/hi/pop_ups/05/uk_should_the_uk_vote_on_the_eu_constitution0/html/8.stm http://news.bbc.co.uk/1/shared/spl/hi/pop_ups/05/uk_should_t...) - Retail students cash in their talent (http://news.bbc.co.uk/1/hi/education/3826037.stm http://news.bbc.co.uk/1/hi/education/3826037.stm)