3 ms·
A Little Web Spider I Wrote Last Night (code included)
- timf 18y agoIt's hard to tell if you're after learning or something to use etc., but if you haven't seen it yet, check out http://scrapy.org/ http://scrapy.org/ for ideas (or even something to use if that is what you are after).
- rhett 18y agoHere is a web spider in 1 line: perl -MLWP::UserAgent -MHTML::LinkExtor -MURI::URL -lwe '$ua = LWP::UserAgent->new; while (my $link = shift @ARGV) { print STDERR "working on $link";HTML::LinkExtor->new( sub { my ($t, %a) = @_; my @links = map { url($_, $link)->abs() } grep { defined } @a{qw/href img/}; print STDERR "+ $_" foreach @links; push @ARGV, @links} )->parse(do { my $r = $ua->simple_request (HTTP::Request->new("GET", $link)); $r->content_type eq "text/html" ? $r-> content : ""; } ) }' http://www.google.com http://www.google.com
- yogione 18y agoI tested the above script - It works. You are good.
- rhett 18y agothanks, i didn't write it. I remembered that in some magazine from 1999. The link to the author is in the comment below
- quilby 18y agoMore info: http://www.foo.be/docs/tpj/issues/vol4_3/tpj0403-0013.html http://www.foo.be/docs/tpj/issues/vol4_3/tpj0403-0013.html
- jerf 18y agoHere's a little web spider I just wrote: "wget -r". I haven't got a problem with people writing their own quickie scripts, but they aren't really worth putting online without some other compelling reason.
- a-priori 18y agoFor completeness, and to keep with the recent meme around here, here's a web spider I wrote a while ago in Erlang. http://github.com/michaelmelanson/spider http://github.com/michaelmelanson/spider