4 ms·

Author's criticism that Flesch-Kincaid isn't suitable for non-English text is on point, but history hasn't been kind to critics of readability scores in general

by cthor 7y ago

Author's criticism that Flesch-Kincaid isn't suitable for non-English text is on point, but history hasn't been kind to critics of readability scores in general: http://www.impact-information.com/impactinfo/newsletter/smartlanguage02.pdf http://www.impact-information.com/impactinfo/newsletter/smar... The given counter-arguments against their use in general ("but what about context!", "they're clearly too simple!") have been tread over for years. Readability formulas are surprisingly robust, although obviously weak to adversarial input.

deogeo 7y ago
More importantly, the test measures complexity, not how 'smart' the text is. A higher score is not necessarily good. In fact, one should strive to express a given idea as simply as possible - from that perspective, a high score is bad. As programmers, this should be very clear to us.
tgv 7y ago
Not just adversarial. FK simply uses word length as a proxy for frequency, which is obviously wrong. It makes crocodiles and elephants score higher than gybe and vaunt. It also doesn't acknowledge that embedded clauses are more difficult than a "sequence" of clauses, etc. In practice, it'll often point in the sort-of-direction-ish, but you need to take the outcome with a pinch of salt. Here's an almost random, older paper that discussed readability score differences between genres: http://csjarchive.cogsci.rpi.edu/Proceedings/2008/pdfs/p1978.pdf http://csjarchive.cogsci.rpi.edu/Proceedings/2008/pdfs/p1978.... Their conclusion: ... the analyses confirmed that several frequently used approaches for measuring vocabulary difficulty tend to be structured such that resulting text difficulty estimates overstate the difficulty of informational texts while simultaneously understating the difficulty of literary texts. These results can be explained in terms of the higher proportion of “core” vocabulary words typically found in literary texts as opposed to informational texts.
- mannykannot 7y ago
  To be fair, these measures are most useful when we are considering alternative ways to state a given set of straightforward facts. It is unlikely that one version will use 'crocodile' and another will use 'gybe'... which, on the other hand, is a good reason for regarding them as merely a first step in studying political speech. I wonder what William Buckley would have thought of the claim that the language of conservatives is simpler than that of liberals?
  - tgv 7y ago
    Even then, many of these metrics will prefer "the rat the cat the dog bit chased escaped" over "the dog bit the cat that chased the rat that escaped". They are very, very rough measures, to be used only in case of emergency. > the language of conservatives is simpler than that of liberals? Times have changed.

by cthor 7y ago

deogeo 7y ago

More importantly, the test measures complexity, not how 'smart' the text is. A higher score is not necessarily good. In fact, one should strive to express a given idea as simply as possible - from that perspective, a high score is bad. As programmers, this should be very clear to us.

tgv 7y ago

Not just adversarial. FK simply uses word length as a proxy for frequency, which is obviously wrong. It makes crocodiles and elephants score higher than gybe and vaunt. It also doesn't acknowledge that embedded clauses are more difficult than a "sequence" of clauses, etc. In practice, it'll often point in the sort-of-direction-ish, but you need to take the outcome with a pinch of salt. Here's an almost random, older paper that discussed readability score differences between genres: http://csjarchive.cogsci.rpi.edu/Proceedings/2008/pdfs/p1978.pdf http://csjarchive.cogsci.rpi.edu/Proceedings/2008/pdfs/p1978.... Their conclusion: ... the analyses confirmed that several frequently used approaches for measuring vocabulary difficulty tend to be structured such that resulting text difficulty estimates overstate the difficulty of informational texts while simultaneously understating the difficulty of literary texts. These results can be explained in terms of the higher proportion of “core” vocabulary words typically found in literary texts as opposed to informational texts.

mannykannot 7y ago
To be fair, these measures are most useful when we are considering alternative ways to state a given set of straightforward facts. It is unlikely that one version will use 'crocodile' and another will use 'gybe'... which, on the other hand, is a good reason for regarding them as merely a first step in studying political speech. I wonder what William Buckley would have thought of the claim that the language of conservatives is simpler than that of liberals?
- tgv 7y ago
  Even then, many of these metrics will prefer "the rat the cat the dog bit chased escaped" over "the dog bit the cat that chased the rat that escaped". They are very, very rough measures, to be used only in case of emergency. > the language of conservatives is simpler than that of liberals? Times have changed.