4 ms·

This is my first project in unsupervised NLP, so let me know if there's anything obviously wrong with the article or methodology.

tssva 8y ago
I know nothing about NLP but would a run where it predicts the authorship of the known texts be useful to get some idea of the level of accuracy?
- wodenokoto 8y ago
  Not the author, but I do know a bit about NLP, so to answer your question: Yes it would :)
blt 8y ago
Why did you choose an unsupervised method to solve a classification problem?
- a-dub 8y ago
  There's no ground truth. :) They were written under a common psuedonym.
  - supermdguy 8y ago
    Couldn't works known to be written by hypothesis authors be used to train a supervised classifier?
    - jonluca 8y ago
      That thought crossed my mind. I was thinking of trying to get a training set and teaching these models based on previous works by Jay, Madison, and Hamilton. However a lot of what I found for each of them was behind a paywall, or too hard to grep through to actually get the data. For instance all I could find in my (admittedly superficial) digging on John Jay was a book by UVA called "The Selected Papers of John Jay". It costs $90 and contains correspondence both too and from John Jay. It seemed like too much overhead for a weekend project so I just settled on unsupervised.
      - tonysdg 8y ago
        May be of interese (see "See Also" for James Madison stuff too): https://en.m.wikipedia.org/wiki/The_Selected_Papers_of_John_Jay https://en.m.wikipedia.org/wiki/The_Selected_Papers_of_John_...
a-dub 8y ago
Maybe consider including the results of others' analyses alongside yours? Even more interesting would be a deep-dive into why they disagree. (Adventures in interpretation) Also maybe do a PCA and show scatter plots of the first two PCs for each doc? I'm no expert, but these could be fun avenues to explore.
- a-dub 8y ago
  You could also go super crazy and try to do an LDA (may have to go beyond syntax and lex). Assume each document is a mixture of author influence. :)
autokad 8y ago
nice article, there is a kaggle competition (Spooky Author) that you had to identify which of three authors wrote a sentence. the problem is very much the same, so not only can you try your technique on the data, you can also read kernels people posted in the competition. Unlike what the commentor above said that its not modern and you should have did word2vec, bags of words are very robust and work well in these situations. word2vec was trained on a completely different corupus, and this data is quite small. some things you might try are: - cosine distance between words - ad LDA (latent dirichelet allocation) topic probabilities - add verb speed (how fast they used the first verb in sentence - run an LSTM NN, add the predicted prob as features (careful in overfiting)
nurettin 8y ago
How were the properties chosen? Did you do any information gain analysis on them before doing the clustering?
thepythiccoder1 8y ago
Hmm the issue is that there might be some correlation between the syntactic and lexical similarity and the actual subject matter the authors are talking about. I've been working on a similar project. Three things that I would suggest would be - add documents definitively written by the authors (Maddison, John jay, etc) from outside the federalist papers to your train set. http://oll.libertyfund.org/titles/jay-the-correspondence-and-public-papers-of-john-jay-vol-1-1763-1781 http://oll.libertyfund.org/titles/jay-the-correspondence-and... http://www.gutenberg.org/ebooks/author/14 http://www.gutenberg.org/ebooks/author/14, etc - Add another feature which looks at the frequency of the function (closed class words) such as articles, prepositions etc these are very stylistic and hard for an author to control, they are also independent of the content, this is a classical feature in forensics. - Add a distractor case to your train and validation set I.e documents written by a non federalist such as Thomas Jefferson and confirm that they don't get clustered into one of the other federalist authors. If you have questions feel free to tweet me it seems like a cool project @pythiccoder

tssva 8y ago

I know nothing about NLP but would a run where it predicts the authorship of the known texts be useful to get some idea of the level of accuracy?

wodenokoto 8y ago
Not the author, but I do know a bit about NLP, so to answer your question: Yes it would :)

blt 8y ago

Why did you choose an unsupervised method to solve a classification problem?

a-dub 8y ago
There's no ground truth. :) They were written under a common psuedonym.
- supermdguy 8y ago
  Couldn't works known to be written by hypothesis authors be used to train a supervised classifier?
  - jonluca 8y ago
    That thought crossed my mind. I was thinking of trying to get a training set and teaching these models based on previous works by Jay, Madison, and Hamilton. However a lot of what I found for each of them was behind a paywall, or too hard to grep through to actually get the data. For instance all I could find in my (admittedly superficial) digging on John Jay was a book by UVA called "The Selected Papers of John Jay". It costs $90 and contains correspondence both too and from John Jay. It seemed like too much overhead for a weekend project so I just settled on unsupervised.
    - tonysdg 8y ago
      May be of interese (see "See Also" for James Madison stuff too): https://en.m.wikipedia.org/wiki/The_Selected_Papers_of_John_Jay https://en.m.wikipedia.org/wiki/The_Selected_Papers_of_John_...

a-dub 8y ago

Maybe consider including the results of others' analyses alongside yours? Even more interesting would be a deep-dive into why they disagree. (Adventures in interpretation) Also maybe do a PCA and show scatter plots of the first two PCs for each doc? I'm no expert, but these could be fun avenues to explore.

a-dub 8y ago
You could also go super crazy and try to do an LDA (may have to go beyond syntax and lex). Assume each document is a mixture of author influence. :)

autokad 8y ago

nice article, there is a kaggle competition (Spooky Author) that you had to identify which of three authors wrote a sentence. the problem is very much the same, so not only can you try your technique on the data, you can also read kernels people posted in the competition. Unlike what the commentor above said that its not modern and you should have did word2vec, bags of words are very robust and work well in these situations. word2vec was trained on a completely different corupus, and this data is quite small. some things you might try are: - cosine distance between words - ad LDA (latent dirichelet allocation) topic probabilities - add verb speed (how fast they used the first verb in sentence - run an LSTM NN, add the predicted prob as features (careful in overfiting)

nurettin 8y ago

How were the properties chosen? Did you do any information gain analysis on them before doing the clustering?

thepythiccoder1 8y ago

Hmm the issue is that there might be some correlation between the syntactic and lexical similarity and the actual subject matter the authors are talking about. I've been working on a similar project. Three things that I would suggest would be - add documents definitively written by the authors (Maddison, John jay, etc) from outside the federalist papers to your train set. http://oll.libertyfund.org/titles/jay-the-correspondence-and-public-papers-of-john-jay-vol-1-1763-1781 http://oll.libertyfund.org/titles/jay-the-correspondence-and... http://www.gutenberg.org/ebooks/author/14 http://www.gutenberg.org/ebooks/author/14, etc - Add another feature which looks at the frequency of the function (closed class words) such as articles, prepositions etc these are very stylistic and hard for an author to control, they are also independent of the content, this is a classical feature in forensics. - Add a distractor case to your train and validation set I.e documents written by a non federalist such as Thomas Jefferson and confirm that they don't get clustered into one of the other federalist authors. If you have questions feel free to tweet me it seems like a cool project @pythiccoder