You can plug into someone else's crawler. For example http://spinn3r.com/ http://spinn3r.com/. In general take a look at http://highscalability.com/ http://hig

You can plug into someone else's crawler. For example http://spinn3r.com/ http://spinn3r.com/. In general take a look at http://highscalability.com/ http://highscalability.com/.