3 ms·
interesting project. I was doing a similar thing with yahoo pipes, but it got blocked because of robots.txt. What do you do about it?
by roll 16y ago
interesting project. I was doing a similar thing with yahoo pipes, but it got blocked because of robots.txt.
What do you do about it?
- shadowpwner 16y agoYou can always disregard robots.txt.. ;)