Hi HN, I submitted this article because it has sat in my bookmarks for a while and I've referenced it quite a number of times over the past few months. I thoug

Hi HN, I submitted this article because it has sat in my bookmarks for a while and I've referenced it quite a number of times over the past few months. I thought it may be helpful to some of you who work on similar problems. I like it because when most people think of "giant datasets" they think they need some special tool to process it. Infact, there are such tools and I use them everyday (namely Hadoop), but this article is a reminder that with the proper forethought and consideration for the data structures in question, it is quite possible to wrestle a large dataset on a single machine. Also, people always wonder how to better optimize their code. I think this is one of the few examples I've seen where the author went through a series of steps to obtain the optimization they had in mind and documented their strategy well. It serves a practical purpose too. If you want to try your hand out at this problem you can obtain the dataset here: http://archive.ics.uci.edu/ml/datasets/Netflix+Prize http://archive.ics.uci.edu/ml/datasets/Netflix+Prize and follow the forums here: http://www.netflixprize.com/community/ http://www.netflixprize.com/community/