4 ms·
This is very interesting. Will you release the data and code at some point?
by heyalexej 11y ago
This is very interesting. Will you release the data and code at some point?
- jacquesm 11y agoYes, I will definitely release the code and the dataset required to bootstrap the rest. It takes a long long time to run and you'll need a good bit of bandwidth. I won't be releasing the raw data because there is simply too much of it.
- adriancooney 11y agoIf bandwidth is a worry there are open data projects by Amazon[1] and I think Google that could host your data. [1] Amazon Public Data Sets, http://aws.amazon.com/public-data-sets/ http://aws.amazon.com/public-data-sets/ [2] Google Public Data, http://www.google.com/publicdata/directory http://www.google.com/publicdata/directory
- heyalexej 11y agoSorry for bugging you. Did you store results from the response¹ metadata object for every domain and process it later or use Regex to parse the HTML content? I crawl large-ish websites (most recently https://code.google.com https://code.google.com with 1.8MM repos) often and am really looking forward to your dataset & code. [1] http://phantomjs.org/api/webpage/handler/on-resource-received.html http://phantomjs.org/api/webpage/handler/on-resource-receive...
- jacquesm 11y ago> Did you store results from the response¹ metadata object for every domain and process it later or use Regex to parse the HTML content? That would have constrained throughput too much so I opted for culling it during the crawl to just content-type and url, this was then processed to extract the various bits of information. I did use the 'resource received' trick you linked above. Very useful.
- jacquesm 11y agoCode released to: https://github.com/jacquesmattheij/remoteresources https://github.com/jacquesmattheij/remoteresources