3 ms·
The trick behind NLP that's surprisingly under discussed is that you must use apples-to-apples comparisons between datasets. Using simple tweets as a proxy for
by minimaxir 8y ago
The trick behind NLP that's surprisingly under discussed is that you must use apples-to-apples comparisons between datasets. Using simple tweets as a proxy for style against a professionally-edited NYT op-ed is silly, but this analysis (along with the previous analyses) is transparent about that fact.
- resters 8y agoThis is obvious. So irritating that someone went to the trouble of doing the analysis on an inappropriate data set.
- jmkni 8y agoNot really, they are just showing off their library in a novel way. The author doesn’t even pretend that it’s going to produce an accurate result.
- colanderman 8y agoDo you mean "opaque"? "Transparent" usually means that someone is forthcoming, but I don't see your sentiment reflected in the article.
- everdev 8y agoThe OpEd reads more like a speech. I'd assume there are some publicly available speeches (or videos with subtitles) that can be analysed instead. Some articles mention that anonymous sources specifically take note of the speech patterns of other people so they can use those to throw off the trail. I'd be very surprised if the author didn't at least try to do that, or agree to have the NYT rewrite each sentence while preserving the meaning. It's an interesting exercise and can and might be able to rule out some authors, but it seems like someone harboring such opinions and wishing to express them to some extent (and has with others -- in the resistance) will be discovered in short order. Outing this person might be an even bigger story than the OpEd itself. But I put my money on old fashion journalism rather than text analysis.
- alex_anglin 8y agoPresumably the candidates for consideration have speech writers and staff who would either write a given speech outright or influence it heavily. So, that would likely rule out any speeches related to their position.
- kevinmchugh 8y agoThe speechwriters and staff are in some cases valid guesses in their own right, but don't have a unique corpus.
- c256 8y agoIn the podcast interview (“The Daily”, also from the nyt), the contact inside the NYT says pretty clearly that they didn’t rewrite anything, to avoid detection or otherwise. I believe the quote was something like “if they write something that’s used to identify themselves, it is what it is”.
- Confiks 8y agoI don't see that transparency at all, nor in the previous analyses linked to in the article. In fact, the articles seem to explicitly be on the side of their methods being definitively able to identify "the author behind New York Time’s op-ed" [1]; this article is prefaced by the title and question "who wrote the anti-Trump New York Times op-ed?", leaving out any criticism to its methodology, making me doubt any other claim Robinson would unreservedly make. [1] https://github.com/mkearney/resist_oped https://github.com/mkearney/resist_oped
- minimaxir 8y agoThe caveats section of the one you linked to is in the .Rmd, not the .md: https://github.com/mkearney/resist_oped/blob/master/resist-oped-text-similarity.Rmd https://github.com/mkearney/resist_oped/blob/master/resist-o...
- black6 8y agoI’d like to see a comparison done against Bob Woodward’s work. As you say, 140 char micro logs are a horrible training set for analyzing a long form essay.
- comex 8y agoYou think the anonymous administration official is Bob Woodward?
- cwilkes 8y agoAlso it assumes that writers don’t change their style. In this case all a fake writer has to do is drop a word like “lodestar” in the document and everyone thinks Mike Pence wrote it. If I were the author I would take my own essay and then run it through algorithms like this and massage my word choices to match other people.