4 ms·
Great question! A combination of techniques were involved in cleaning up institution names. First, I deleted all department names so that only the university/la
by ck_one 6y ago
Great question! A combination of techniques were involved in cleaning up institution names. First, I deleted all department names so that only the university/lab name was left. Second, I used local sensitive hashing (wiki: https://en.wikipedia.org/wiki/Locality-sensitive_hashing https://en.wikipedia.org/wiki/Locality-sensitive_hashing) to find similar strings. LSH maps similar strings to the same bucket. This resolved some ambiguity like: UC Berkeley/University of California Berkeley etc. But there was also a good chunk of manual clean up necessary at the end.