6 ms·

Text Mining South Park

202 points by eamonncarey 11y ago

peg_leg 11y ago
This should be nominated for an igNobel
dropdatabase 11y ago
This is amazing, I wonder what results you'd get from The Simpsons
- charlieegan3 11y ago
  Not sure subtitles contain character information but the people running https://frinkiac.com/ https://frinkiac.com/ might have the data.
deleted 11y ago
[deleted]
agentgt 11y ago
I wonder how the results would change if it was based not on words but rather by lines (not string lines but actor lines in conversation). Its also funny how Stan talks more than Kyle given the show now has a recurring joke that makes fun of Kyle's long educational dialogues.
- cdubzzz 11y ago
  Maybe because of Kyle's decision to not give long speeches last season (:
- flashman 11y ago
  It would definitely change. For instance I'd expect Kyle's words-per-sentence (or at least his 90th percentile sentence length) to be higher than Stan's, due to his speeches.
gulbrandr 11y ago
Error establishing a database connection Someone has a cached version please?
- eeturunen 11y ago
  http://i.imgur.com/IEudyni.jpg http://i.imgur.com/IEudyni.jpg
cadab 11y ago
I've found an image, which i'm guessing it taken from the site: http://imgur.com/IEudyni http://imgur.com/IEudyni, worth looking at if the sites still down.
seankross 11y ago
Here's the accompanying GitHub repo: https://github.com/walkerkq/textmining_southpark https://github.com/walkerkq/textmining_southpark
wodenokoto 11y ago
> Reducing the sparsity brought that down to about 3,100 unique words [from 30,600 unique words] What does that mean? Does he remove words that are only said once or twice? Can anyone point me to a text explaining the difference between Identifying Characteristic Words using Log Likelihood and using tfidf. ?
- minimaxir 11y ago
  Relevant line in code: # remove sparse terms all.tdm.75 <- removeSparseTerms(all.tdm, 0.75) # 3117 / 728215 I believe it corresponds to the tfidf factor.
nanis 11y ago
I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carrying out statistical tests. If I know ALL the words spoken by someone, then I know which words they say the most without resorting to any tests simply by counting. No concept of "statistical significance" is applicable because there is no sample. We can calculate the population value of any parameter we can think of, because, we have the entire population (in this specific instance, ALL the words spoken by all the characters). FYI, all budding data "scientists" ...
- nanis 11y ago
  Also, I am going to go out on a limb here and guess that R's `read.csv` doesn't do what one hopes it would when fed this CSV: 10,3,Brian,"You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? Peter: Come on, Mohammad, let's get some tea. Mr. T: Try my ""Mr. T. ...tea."" " Well, it seems people are not understanding the problem with this line. Here is the screenshot of the original script: http://imgur.com/pcu5N2U http://imgur.com/pcu5N2U Brian: You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? [flashback #3] Peter: Come on, Mohammad, let's get some tea. [Mohammad is covered by a black box with the words "IMAGE CENSORED BY FOX" printed several times from top to bottom inside the box. They stop at a tea stand.] Mr. T: Try my "Mr. T. ...tea." [squints] There, three characters speak. However, R's read.csv will assign all three characters' speech to Brian: http://imgur.com/gLpPKdl http://imgur.com/gLpPKdl > x[596, ] Season Episode Character 596 10 3 Brian Line 596 You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? \nPeter:\nCome on, Mohammad, let's get some tea. \n > x[597,] Season Episode Character 597 10 3 Brian Line 597 You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? \nPeter:\nCome on, Mohammad, let's get some tea. \nMr. T:\nTry my "Mr. T. ...tea." \n as well as seemingly duplicating part of the conversation. PS: In addition, both Muhammad and Mohammad appear, presumably under-counting the references to the prophet.
LoSboccacc 11y ago
I would have loved to see log characterization for the canadians characters, even if they aren't part of the main cast
rhema 11y ago
Pretty interesting. This Large Scale Study of Myspace (http://www.cc.gatech.edu/projects/doi/Papers/Caverlee_ICWSM_2008.pdf http://www.cc.gatech.edu/projects/doi/Papers/Caverlee_ICWSM_...) paper shows a similar method for finding characteristic terms, using Mutual Information.
pg_is_a_butt 11y ago
more like text mining reddit, nerds.

202 points by eamonncarey 11y ago

peg_leg 11y ago

This should be nominated for an igNobel

dropdatabase 11y ago

This is amazing, I wonder what results you'd get from The Simpsons

charlieegan3 11y ago
Not sure subtitles contain character information but the people running https://frinkiac.com/ https://frinkiac.com/ might have the data.

deleted 11y ago

[deleted]

agentgt 11y ago

I wonder how the results would change if it was based not on words but rather by lines (not string lines but actor lines in conversation). Its also funny how Stan talks more than Kyle given the show now has a recurring joke that makes fun of Kyle's long educational dialogues.

cdubzzz 11y ago
Maybe because of Kyle's decision to not give long speeches last season (:
flashman 11y ago
It would definitely change. For instance I'd expect Kyle's words-per-sentence (or at least his 90th percentile sentence length) to be higher than Stan's, due to his speeches.

gulbrandr 11y ago

Error establishing a database connection Someone has a cached version please?

eeturunen 11y ago
http://i.imgur.com/IEudyni.jpg http://i.imgur.com/IEudyni.jpg

cadab 11y ago

I've found an image, which i'm guessing it taken from the site: http://imgur.com/IEudyni http://imgur.com/IEudyni, worth looking at if the sites still down.

seankross 11y ago

Here's the accompanying GitHub repo: https://github.com/walkerkq/textmining_southpark https://github.com/walkerkq/textmining_southpark

wodenokoto 11y ago

> Reducing the sparsity brought that down to about 3,100 unique words [from 30,600 unique words] What does that mean? Does he remove words that are only said once or twice? Can anyone point me to a text explaining the difference between Identifying Characteristic Words using Log Likelihood and using tfidf. ?

minimaxir 11y ago
Relevant line in code: # remove sparse terms all.tdm.75 <- removeSparseTerms(all.tdm, 0.75) # 3117 / 728215 I believe it corresponds to the tfidf factor.

nanis 11y ago

I was in the process of reading this when I thought to check who this person is. Of course, by that time the site had failed, so I haven't read the whole thing yet. But, it seems to me that the author is falling in to a trap many an unwary data "scientist" falls by not understanding the discipline of Statistics. When one has the entire population data (i.e. a census), rather than a sample, there is no point in carrying out statistical tests. If I know ALL the words spoken by someone, then I know which words they say the most without resorting to any tests simply by counting. No concept of "statistical significance" is applicable because there is no sample. We can calculate the population value of any parameter we can think of, because, we have the entire population (in this specific instance, ALL the words spoken by all the characters). FYI, all budding data "scientists" ...

nanis 11y ago
Also, I am going to go out on a limb here and guess that R's `read.csv` doesn't do what one hopes it would when fed this CSV: 10,3,Brian,"You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? Peter: Come on, Mohammad, let's get some tea. Mr. T: Try my ""Mr. T. ...tea."" " Well, it seems people are not understanding the problem with this line. Here is the screenshot of the original script: http://imgur.com/pcu5N2U http://imgur.com/pcu5N2U Brian: You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? [flashback #3] Peter: Come on, Mohammad, let's get some tea. [Mohammad is covered by a black box with the words "IMAGE CENSORED BY FOX" printed several times from top to bottom inside the box. They stop at a tea stand.] Mr. T: Try my "Mr. T. ...tea." [squints] There, three characters speak. However, R's read.csv will assign all three characters' speech to Brian: http://imgur.com/gLpPKdl http://imgur.com/gLpPKdl > x[596, ] Season Episode Character 596 10 3 Brian Line 596 You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? \nPeter:\nCome on, Mohammad, let's get some tea. \n > x[597,] Season Episode Character 597 10 3 Brian Line 597 You mean like the time you had tea with Mohammad, the prophet of the Muslim faith? \nPeter:\nCome on, Mohammad, let's get some tea. \nMr. T:\nTry my "Mr. T. ...tea." \n as well as seemingly duplicating part of the conversation. PS: In addition, both Muhammad and Mohammad appear, presumably under-counting the references to the prophet.

LoSboccacc 11y ago

I would have loved to see log characterization for the canadians characters, even if they aren't part of the main cast

rhema 11y ago

Pretty interesting. This Large Scale Study of Myspace (http://www.cc.gatech.edu/projects/doi/Papers/Caverlee_ICWSM_2008.pdf http://www.cc.gatech.edu/projects/doi/Papers/Caverlee_ICWSM_...) paper shows a similar method for finding characteristic terms, using Mutual Information.

pg_is_a_butt 11y ago

more like text mining reddit, nerds.