'The interviewer wanted him to give a technical answer to a technical question, not enter into a discussion about what "The ten thousand most-used words in the

'The interviewer wanted him to give a technical answer to a technical question, not enter into a discussion about what "The ten thousand most-used words in the English language" actually means or whether it's feasible to collect a corpus of such words.' Is it not technical to consider the sample size of your data? It's not even interesting how you do the +1 to the frequency count of a given word if the # of words get huge - nobody has any business talking about hash table collisions blah, it's too low level. I'll estimate a half million english words and use the Java hash table or a database with a words table, now I will setup my web crawler to get data... but how do I know what text is a word and is not? That is actually the only interesting bit of that question! So first I need to scrape dictionary.com and then start spidering say livejournal vs msnbc vs whatever. If someone talks about hash table collisions for this problem, they are likely to suffer from "not invented here".