3 ms·
Sorry for bugging you. Did you store results from the response¹ metadata object for every domain and process it later or use Regex to parse the HTML content? I
by heyalexej 11y ago
Sorry for bugging you. Did you store results from the response¹ metadata object for every domain and process it later or use Regex to parse the HTML content?
I crawl large-ish websites (most recently https://code.google.com https://code.google.com with 1.8MM repos) often and am really looking forward to your dataset & code.
[1] http://phantomjs.org/api/webpage/handler/on-resource-received.html http://phantomjs.org/api/webpage/handler/on-resource-receive...
- jacquesm 11y ago> Did you store results from the response¹ metadata object for every domain and process it later or use Regex to parse the HTML content? That would have constrained throughput too much so I opted for culling it during the crawl to just content-type and url, this was then processed to extract the various bits of information. I did use the 'resource received' trick you linked above. Very useful.