4 ms·

But "forschungsgemeinschaft" is German, and it's mentioned frequently: at least 100,000 times each in the book corpus. I don't trust his corpus compares to the

by nodata 14y ago

But "forschungsgemeinschaft" is German, and it's mentioned frequently: at least 100,000 times each in the book corpus. I don't trust his corpus compares to the original English corpus.

arrrg 14y ago
As you might have seen, all those words with twenty letters or more are very weird. That’s because there aren’t very many commonly used long English words. Also, at least 100,000 times isn’t so frequent, given Google’s huge corpus. Here is the relative frequency of “Forschungsgemeinschaft” over time: http://books.google.com/ngrams/graph?content=Forschungsgemeinschaft&year_start=1500&year_end=2008&corpus=15&smoothing=0 http://books.google.com/ngrams/graph?content=Forschungsgemei... Here are results on Google Books when searching for “Forschungsgemeinschaft”: http://www.google.com/search?q=%22Forschungsgemeinschaft%22&tbs=bks:1&lr=lang_en http://www.google.com/search?q=%22Forschungsgemeinschaft%22&... Google doesn’t promise that their English corpus only contains English books, just that it contains mostly English books. Given the raw size of their corpus this algorithmic solution seems like a reasonable tradeoff to me. The frequency of “Forschungsgemeinschaft” is also easy enough to explain: The “Deutsche Forschungsgemeinschaft” (German Research Foundation) provides lots of funding for research in Germany. It is consequently mentioned in lots of research papers, many of which are written in English.
pretoriusB 14y ago
A lot of foreign words are frequently met in English texts, especially in narrow domains like philosophy, biology and medicine, etc. And seing that: http://en.wikipedia.org/wiki/Deutsche_Forschungsgemeinschaft http://en.wikipedia.org/wiki/Deutsche_Forschungsgemeinschaft is a research institute, one would expect tons of mentions for this word in _english_ scientific papers. But that is an outlier, and given the vastness of the corpus those would have been sorted out. That said, there would have been an easy way to filter non English books out of the way automatically: do a statistical analysis on each book (e.g on letter frequency) and reject the ones that stray too much from the norm -- or send them to a secondary filtering stage, e.g by word presence or a verification by a human. Done carefully that filtering would not harm the actual results at all (e.g by presuposing a specific letter frequency, because it would only reject extreme outliers that would indeed by non-english works).
- arrrg 14y ago
  It’s not a research institute. It gives grants to researchers (it’s the biggest organization in Germany giving research grants), it’s a foundation. As such it is also often mentioned in scientific papers. (“This study was funded in part by a grant from the …”)

by nodata 14y ago

But "forschungsgemeinschaft" is German, and it's mentioned frequently: at least 100,000 times each in the book corpus. I don't trust his corpus compares to the original English corpus.

arrrg 14y ago

As you might have seen, all those words with twenty letters or more are very weird. That’s because there aren’t very many commonly used long English words. Also, at least 100,000 times isn’t so frequent, given Google’s huge corpus. Here is the relative frequency of “Forschungsgemeinschaft” over time: http://books.google.com/ngrams/graph?content=Forschungsgemeinschaft&year_start=1500&year_end=2008&corpus=15&smoothing=0 http://books.google.com/ngrams/graph?content=Forschungsgemei... Here are results on Google Books when searching for “Forschungsgemeinschaft”: http://www.google.com/search?q=%22Forschungsgemeinschaft%22&tbs=bks:1&lr=lang_en http://www.google.com/search?q=%22Forschungsgemeinschaft%22&... Google doesn’t promise that their English corpus only contains English books, just that it contains mostly English books. Given the raw size of their corpus this algorithmic solution seems like a reasonable tradeoff to me. The frequency of “Forschungsgemeinschaft” is also easy enough to explain: The “Deutsche Forschungsgemeinschaft” (German Research Foundation) provides lots of funding for research in Germany. It is consequently mentioned in lots of research papers, many of which are written in English.

pretoriusB 14y ago

A lot of foreign words are frequently met in English texts, especially in narrow domains like philosophy, biology and medicine, etc. And seing that: http://en.wikipedia.org/wiki/Deutsche_Forschungsgemeinschaft http://en.wikipedia.org/wiki/Deutsche_Forschungsgemeinschaft is a research institute, one would expect tons of mentions for this word in _english_ scientific papers. But that is an outlier, and given the vastness of the corpus those would have been sorted out. That said, there would have been an easy way to filter non English books out of the way automatically: do a statistical analysis on each book (e.g on letter frequency) and reject the ones that stray too much from the norm -- or send them to a secondary filtering stage, e.g by word presence or a verification by a human. Done carefully that filtering would not harm the actual results at all (e.g by presuposing a specific letter frequency, because it would only reject extreme outliers that would indeed by non-english works).

arrrg 14y ago
It’s not a research institute. It gives grants to researchers (it’s the biggest organization in Germany giving research grants), it’s a foundation. As such it is also often mentioned in scientific papers. (“This study was funded in part by a grant from the …”)