4 ms·
I made a crawler https://github.com/jahaynes/crawler https://github.com/jahaynes/crawler It outputs to the warc file format (https://en.wikipedia.org/wiki/Web_
by mrkeen 9y ago
I made a crawler https://github.com/jahaynes/crawler https://github.com/jahaynes/crawler
It outputs to the warc file format (https://en.wikipedia.org/wiki/Web_ARChive https://en.wikipedia.org/wiki/Web_ARChive), in case your workflow is to gather web pages and then process them afterwards.