4 ms·
For pagerank you are going to need a system for large scale graph processing, a la pregel. Neo4J has an algorithms module with pagerank built in, but I don't th
by bryanrasmussen 9y ago
For pagerank you are going to need a system for large scale graph processing, a la pregel. Neo4J has an algorithms module with pagerank built in, but I don't think it will scale large enough.
On the personal projects I'm working on right now I've decided to do as follows -
crawl and make graph/index of links, initial data to get overview of site(s), secondary crawlers might be sent out to get extra data if needed (this is generally if I am using some library that analyzes DOM in-browser), if I just need to analyze whole DOM with my own library I send that dom for analysis right away while crawling to next links.
collecting and parsing incoming data really depends on what you want to do, for stuff that is going to have to access the whole html of the page I'm passing that html wrapped in object to resque queue. Crawling the page is going to need to have a javascript aware crawler given the current state of the web. one thing I generally do is wait for links to be rendered, count the number of links on page if < than what I think a page should have wait +seconds for page to be rendered by react, angular, whatever and then analyze page again.
storing data is a decision regarding money, and how much time you have to do stuff, and probably what your competencies are.
however what I am doing is not an internet scale operation, it is at best Customer/Project X uses Crawler Y to crawl Site Z scale.