5 ms·

Google Ngram Experiments

47 points by solipsist 16y ago

moultano 16y ago
Here's my set of them: http://moultano.blogspot.com/2010/12/history-through-google-books-ngrams.html http://moultano.blogspot.com/2010/12/history-through-google-... I got totally addicted to this for 24 hours and couldn't do anything else. Completely amazing. I believe history now!
- RickRoll 16y ago
  >> I believe history now! This statement makes so little sense ...
  - moultano 16y ago
    For me this is the first time I've (indirectly) looked at primary sources. I'm no longer relying on the benevolence of some other expert, I can check the data myself!
    - RickRoll 16y ago
      Well 1st IMHO you can say you believe the version of history you've been thought in school (you didn't believe before there is any history?). Obviously there are different official versions in differ countries. For example according to US schoolbooks US was the primary reason Germany&Co lost WW2, but not so in Russian schoolbooks. 2nd I don't see how you've checked any data. Say you see word 'terror' is mentioned many times after 2001. What if most of the books say 9/11 was an inside job and there were no terrorist? The fact that the word 'terror' is used often in books doesn't mean there was actual terror going on.
albertsun 16y ago
Here's another curated set. http://ngrams.tumblr.com/ http://ngrams.tumblr.com/
kapilkaisare 16y ago
Interestingly, science begins to trump religion at approximately the same time that freedom begins to trump justice. Not that I'm complaining, but what happened during the 1920s?
- nl 16y ago
  The modern world began. Seriously - WW1 ended, and people began to understand the impact of technology. Motorized and air transport, radio, Communism, womens rights and the rise to dominance of the US all were trends that were well underway in the 1920s, but were either non-existent or very rare in 1910.
  - RickRoll 16y ago
    Ppl began to understand the impact of technology in 1918? What have you been smoking? ;)
    - nl 16y ago
      I assume you disagree? Are you saying they understood it earlier or later? Prior to WW1 (in Europe anyway), there was a distinct class-based disconnect on the impact of technology on society. Yes, the upper/political classes knew that it meant many more people had moved to the cities, and many industrialists were becoming rich. But it was the impact of technology on warfare that forced the political classes to consider idea that technology leadership was a matter of national existence. At the same time, the industrial scale of the war, and the huge number of people who lost their life led to wide-spread questioning of the political classes by the working class, which in turn led to an interest in the things that did affect them so much during the war: technology.
      - RickRoll 16y ago
        Nonsense. So hunter-gatherers didn't understand the impact of sharper stone knifes. And ALL technology prior to 20th century was an accident. Nobody could see the impact of an iron sword. And people in 19th century western Europe didn't understand that say Opium wars were won through better technology.
nl 16y ago
My favorite thing is seeing memes that die out for various reasons. The best example I've found is "king cotton" (the idea that European powers would intervene in the American civil war to protect their supplies of cotton): http://ngrams.googlelabs.com/graph?content=king+cotton&year_start=1820&year_end=2008&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=king+cotton&y... Another that compares a few and their cultural impact. Did you know that Phrenology probably had more cultural impact than UFO's did? http://ngrams.googlelabs.com/graph?content=Phrenology%2CSpiritualism%2CUFO&year_start=1820&year_end=2008&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=Phrenology%2CSpir... Here's a dead scientific theory: http://ngrams.googlelabs.com/graph?content=luminiferous&year_start=1700&year_end=2008&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=luminiferous&... The success of a socio-political movement: http://ngrams.googlelabs.com/graph?content=Woman%27s+Suffrage&year_start=1700&year_end=2008&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=Woman%27s+Suffrag... The death of Wade-Giles and rise of Pinyin for Chinese: http://ngrams.googlelabs.com/graph?content=Peking%2CBeijing&year_start=1820&year_end=2008&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=Peking%2CBeijing&...
srean 16y ago
Some older comments here: http://news.ycombinator.com/item?id=2013346 http://news.ycombinator.com/item?id=2013346 Surprisingly enough I could not get a hit with k-grams of "Almost but not quite entirely unlike tea." But k-grams of "Colorless green ideas sleep furiously" worked.
zitterbewegung 16y ago
Most interesting one that I found was woot http://ngrams.googlelabs.com/graph?content=woot&year_start=1820&year_end=2008&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=woot&year_sta... its actually used in the 1800s
RickRoll 16y ago
http://ngrams.googlelabs.com/graph?content=google,microsoft,yahoo&year_start=1975&year_end=2008&corpus=0&smoothing=1 http://ngrams.googlelabs.com/graph?content=google,microsoft,...
- LeBleu 16y ago
  It is case-sensitive, try the following instead: http://ngrams.googlelabs.com/graph?content=Google%2CMicrosoft%2CYahoo&year_start=1975&year_end=2008&corpus=0&smoothing=1 http://ngrams.googlelabs.com/graph?content=Google%2CMicrosof...
nitrogen 16y ago
I love the spate of unbalanced parentheses around 1700. Probably an OCR failure, but still amusing.
guscost 16y ago
http://ngrams.googlelabs.com/graph?content=satan,antichrist,brimstone,blasphemy,fuck&year_start=1700&year_end=2000&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=satan,antichrist,... Most interesting one I could find. Jesus hates swear words, apparently.
- shadowpwner 16y ago
  Do you understand the history behind this? I didn't imagine that "fuck' was used so often.
  - guscost 16y ago
    Well, I do know that it's never meant anything but what it means today. It was probably more common in oral circulation than written down, but it probably did show up in text too. I think that the Second Great Awakening would have effectively eliminated it from print during the period the graph shows. From Wikipedia: "Though it appeared in John Ash's 1775 A New and Complete Dictionary, listed as "low" and "vulgar," and appearing with several definitions,[10] fuck did not appear in any widely-consulted dictionary of the English language from 1795 to 1965. Its first appearance in the Oxford English Dictionary (along with the word cunt) was in 1972. There is anecdotal evidence of its use during the American Civil War." @tom: Ah, very interesting. OCR error would have been my second guess. http://books.google.com/books?id=EIBIJunqzeUC&pg=PA278&dq=%22fuck%22&hl=en&ei=yFslTYSdBM2r8AapmbCJAg&sa=X&oi=book_result&ct=result&resnum=6&ved=0CEcQ6AEwBTgK#v=onepage&q=%22fuck%22&f=false http://books.google.com/books?id=EIBIJunqzeUC&pg=PA278&#... "War will fuck away their souls."
  - tomhenderson 16y ago
    This article gives a good explanation: http://searchengineland.com/when-ocr-goes-bad-googles-ngram-viewer-the-f-word-59181 http://searchengineland.com/when-ocr-goes-bad-googles-ngram-... Also interesting: http://ngrams.googlelabs.com/graph?content=suck%2Cfuck&year_start=1700&year_end=2000&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=suck%2Cfuck&y...
jeremydavid 16y ago
I would love for someone to download all the data and put together lists of the most frequent 10,000 1, 2, 4, and 5 ngrams. The datasets are just too large for my computer to handle. http://ngrams.googlelabs.com/datasets http://ngrams.googlelabs.com/datasets
- bobf 16y ago
  What would you do with it? (I know there are lots of uses, but what's yours?)
  - jeremydavid 16y ago
    I'm working on a language app and an authoritative (and public domain) list of the most frequent words / phrases would be tremendously helpful :)
thmzlt 16y ago
FORTRAN and Lisp: http://ngrams.googlelabs.com/graph?content=FORTRAN,Lisp&year_start=1960&year_end=2008&corpus=0&smoothing=10 http://ngrams.googlelabs.com/graph?content=FORTRAN,Lisp&...

47 points by solipsist 16y ago

Here's my set of them: http://moultano.blogspot.com/2010/12/history-through-google-books-ngrams.html http://moultano.blogspot.com/2010/12/history-through-google-... I got totally addicted to this for 24 hours and couldn't do anything else. Completely amazing. I believe history now!

RickRoll 16y ago
>> I believe history now! This statement makes so little sense ...
- moultano 16y ago
  For me this is the first time I've (indirectly) looked at primary sources. I'm no longer relying on the benevolence of some other expert, I can check the data myself!
  - RickRoll 16y ago
    Well 1st IMHO you can say you believe the version of history you've been thought in school (you didn't believe before there is any history?). Obviously there are different official versions in differ countries. For example according to US schoolbooks US was the primary reason Germany&Co lost WW2, but not so in Russian schoolbooks. 2nd I don't see how you've checked any data. Say you see word 'terror' is mentioned many times after 2001. What if most of the books say 9/11 was an inside job and there were no terrorist? The fact that the word 'terror' is used often in books doesn't mean there was actual terror going on.

albertsun 16y ago

Here's another curated set. http://ngrams.tumblr.com/ http://ngrams.tumblr.com/

kapilkaisare 16y ago

Interestingly, science begins to trump religion at approximately the same time that freedom begins to trump justice. Not that I'm complaining, but what happened during the 1920s?

nl 16y ago
The modern world began. Seriously - WW1 ended, and people began to understand the impact of technology. Motorized and air transport, radio, Communism, womens rights and the rise to dominance of the US all were trends that were well underway in the 1920s, but were either non-existent or very rare in 1910.
- RickRoll 16y ago
  Ppl began to understand the impact of technology in 1918? What have you been smoking? ;)
  - nl 16y ago
    I assume you disagree? Are you saying they understood it earlier or later? Prior to WW1 (in Europe anyway), there was a distinct class-based disconnect on the impact of technology on society. Yes, the upper/political classes knew that it meant many more people had moved to the cities, and many industrialists were becoming rich. But it was the impact of technology on warfare that forced the political classes to consider idea that technology leadership was a matter of national existence. At the same time, the industrial scale of the war, and the huge number of people who lost their life led to wide-spread questioning of the political classes by the working class, which in turn led to an interest in the things that did affect them so much during the war: technology.
    - RickRoll 16y ago
      Nonsense. So hunter-gatherers didn't understand the impact of sharper stone knifes. And ALL technology prior to 20th century was an accident. Nobody could see the impact of an iron sword. And people in 19th century western Europe didn't understand that say Opium wars were won through better technology.

nl 16y ago

My favorite thing is seeing memes that die out for various reasons. The best example I've found is "king cotton" (the idea that European powers would intervene in the American civil war to protect their supplies of cotton): http://ngrams.googlelabs.com/graph?content=king+cotton&year_start=1820&year_end=2008&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=king+cotton&y... Another that compares a few and their cultural impact. Did you know that Phrenology probably had more cultural impact than UFO's did? http://ngrams.googlelabs.com/graph?content=Phrenology%2CSpiritualism%2CUFO&year_start=1820&year_end=2008&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=Phrenology%2CSpir... Here's a dead scientific theory: http://ngrams.googlelabs.com/graph?content=luminiferous&year_start=1700&year_end=2008&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=luminiferous&... The success of a socio-political movement: http://ngrams.googlelabs.com/graph?content=Woman%27s+Suffrage&year_start=1700&year_end=2008&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=Woman%27s+Suffrag... The death of Wade-Giles and rise of Pinyin for Chinese: http://ngrams.googlelabs.com/graph?content=Peking%2CBeijing&year_start=1820&year_end=2008&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=Peking%2CBeijing&...

srean 16y ago

Some older comments here: http://news.ycombinator.com/item?id=2013346 http://news.ycombinator.com/item?id=2013346 Surprisingly enough I could not get a hit with k-grams of "Almost but not quite entirely unlike tea." But k-grams of "Colorless green ideas sleep furiously" worked.

zitterbewegung 16y ago

Most interesting one that I found was woot http://ngrams.googlelabs.com/graph?content=woot&year_start=1820&year_end=2008&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=woot&year_sta... its actually used in the 1800s

RickRoll 16y ago

http://ngrams.googlelabs.com/graph?content=google,microsoft,yahoo&year_start=1975&year_end=2008&corpus=0&smoothing=1 http://ngrams.googlelabs.com/graph?content=google,microsoft,...

LeBleu 16y ago
It is case-sensitive, try the following instead: http://ngrams.googlelabs.com/graph?content=Google%2CMicrosoft%2CYahoo&year_start=1975&year_end=2008&corpus=0&smoothing=1 http://ngrams.googlelabs.com/graph?content=Google%2CMicrosof...

nitrogen 16y ago

I love the spate of unbalanced parentheses around 1700. Probably an OCR failure, but still amusing.

guscost 16y ago

http://ngrams.googlelabs.com/graph?content=satan,antichrist,brimstone,blasphemy,fuck&year_start=1700&year_end=2000&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=satan,antichrist,... Most interesting one I could find. Jesus hates swear words, apparently.

shadowpwner 16y ago
Do you understand the history behind this? I didn't imagine that "fuck' was used so often.
- guscost 16y ago
  Well, I do know that it's never meant anything but what it means today. It was probably more common in oral circulation than written down, but it probably did show up in text too. I think that the Second Great Awakening would have effectively eliminated it from print during the period the graph shows. From Wikipedia: "Though it appeared in John Ash's 1775 A New and Complete Dictionary, listed as "low" and "vulgar," and appearing with several definitions,[10] fuck did not appear in any widely-consulted dictionary of the English language from 1795 to 1965. Its first appearance in the Oxford English Dictionary (along with the word cunt) was in 1972. There is anecdotal evidence of its use during the American Civil War." @tom: Ah, very interesting. OCR error would have been my second guess. http://books.google.com/books?id=EIBIJunqzeUC&pg=PA278&dq=%22fuck%22&hl=en&ei=yFslTYSdBM2r8AapmbCJAg&sa=X&oi=book_result&ct=result&resnum=6&ved=0CEcQ6AEwBTgK#v=onepage&q=%22fuck%22&f=false http://books.google.com/books?id=EIBIJunqzeUC&pg=PA278&#... "War will fuck away their souls."
- tomhenderson 16y ago
  This article gives a good explanation: http://searchengineland.com/when-ocr-goes-bad-googles-ngram-viewer-the-f-word-59181 http://searchengineland.com/when-ocr-goes-bad-googles-ngram-... Also interesting: http://ngrams.googlelabs.com/graph?content=suck%2Cfuck&year_start=1700&year_end=2000&corpus=0&smoothing=3 http://ngrams.googlelabs.com/graph?content=suck%2Cfuck&y...

jeremydavid 16y ago

I would love for someone to download all the data and put together lists of the most frequent 10,000 1, 2, 4, and 5 ngrams. The datasets are just too large for my computer to handle. http://ngrams.googlelabs.com/datasets http://ngrams.googlelabs.com/datasets

bobf 16y ago
What would you do with it? (I know there are lots of uses, but what's yours?)
- jeremydavid 16y ago
  I'm working on a language app and an authoritative (and public domain) list of the most frequent words / phrases would be tremendously helpful :)

thmzlt 16y ago

FORTRAN and Lisp: http://ngrams.googlelabs.com/graph?content=FORTRAN,Lisp&year_start=1960&year_end=2008&corpus=0&smoothing=10 http://ngrams.googlelabs.com/graph?content=FORTRAN,Lisp&...