At their scale I doubt they are using LSA or anything of similar complexity. My guess would be something like tf-idf coupled with any thing from naive bayes to

At their scale I doubt they are using LSA or anything of similar complexity. My guess would be something like tf-idf coupled with any thing from naive bayes to logic programming for entity recognition. Final output would likely use some hand curated rules on how to combine the above with infromation from curated databases like dbepida. ________________ There is this thing called random projection based on the observation that in high dimensions, most direction are nearly orthogonal to each other. This is a wonderful observation because it allows us to write stupendously simple (and fast) algorithms that give good results for mathematically sound reasons. I strongly suggest anyone who is interested in LSA but wants something scalable, simpler to write and faster to look into Random Indexing.