If this is just a one off job, I would guess that running sort -u $FILE will take less time than writing some specific software for the purpose. Of course, if

If this is just a one off job, I would guess that running sort -u $FILE will take less time than writing some specific software for the purpose. Of course, if you want to learn hadoop then go for it. But it's probably more practical just to let it run :-)