The canonical Hadoop example is counting the number of words in a large text file http://wiki.apache.org/hadoop/WordCount http://wiki.apache.org/hadoop/WordCoun

The canonical Hadoop example is counting the number of words in a large text file http://wiki.apache.org/hadoop/WordCount http://wiki.apache.org/hadoop/WordCount You could use the same approach to simply take the unique tokens of the output of the final "count". May be overkill if you you can read the file in less than an hour, but this approach (divide and conquer) may be a good inspiration.