I cannot recommend FlashGraph strongly enough. FlashGraph was one of the first graph computation engines to enable near trivial analysis of the Web Data Commons

I cannot recommend FlashGraph strongly enough. FlashGraph was one of the first graph computation engines to enable near trivial analysis of the Web Data Commons Hyperlink Graph - 3.5 billion web pages and 128 billion links. For smaller graphs, analysis using FlashGraph is hilariously quick. If you're interested in how this is achieved, refer to [1]. From memory, Da Zheng said he created FlashGraph primarily as he wanted to prove how efficient the storage system was. Full details on the FlashGraph (though FlashGraph runs far faster now!) in the paper at [2]. Note: I'm a data scientist at Common Crawl, the dataset that the Web Data Commons Hyperlink Graph is based upon and the main developer of Flash Graph, Da Zheng, wrote a guest post for us on this very topic[3], so I'm rightfully biased as to thinking this is an amazing project! [1]: http://www.cs.jhu.edu/~zhengda/sc13.pdf http://www.cs.jhu.edu/~zhengda/sc13.pdf [2]: https://www.usenix.org/system/files/conference/fast15/fast15-paper-zheng.pdf https://www.usenix.org/system/files/conference/fast15/fast15... [3]: http://blog.commoncrawl.org/2015/02/analyzing-a-web-graph-with-129-billion-edges-using-flashgraph/ http://blog.commoncrawl.org/2015/02/analyzing-a-web-graph-wi...