Back in the days I wrote my own crawler and layout analyzer to collect news articles for my research (cf. http://www.unixuser.org/~euske/python/webstemmer/ http

Back in the days I wrote my own crawler and layout analyzer to collect news articles for my research (cf. http://www.unixuser.org/~euske/python/webstemmer/ http://www.unixuser.org/~euske/python/webstemmer/ ). I thought in 2005 it was mostly acceptable as long as you don't hog their bandwidth, but today I feel it would be looked differently. It is kinda sad to see that the more and more part of the web is treated not public. It seems that everyone likes to build their wall.