5 ms·

I wonder if there's a confounding effect here. The authors basically have a list of N-grams that they have identified as being indicative of cognitive distorti

I wonder if there's a confounding effect here. The authors basically have a list of N-grams that they have identified as being indicative of cognitive distortions and find that these N-grams have become much more prevalent in the last few decades. But the selection of these N-grams is based on how we, today, would express these cognitive distortions. It doesn't seem that the could rule out the rate of cognitive distortions remaining the same, but the language to express them changing. To be fair, looking at their table of N-grams, a lot of them seem pretty timeless. For example: everyone, everybody, should, ought, must, worst, best. But many of the N-grams strike me as being rather modern formulations. For example: a loser, a toxic, OK but, OK yet. If a team of CBT therapists constructed such a list in 1900 instead of 2020, would we see the prevalence of cognitive distortions decrease as our language changed? I think they try to control for this by choosing a null set of random N-grams, but the random N-grams were chosen from their entire corpus --- in other words by definition it's representative of the entire time span. Edit: As an example of this, the authors include in their sample of words indicative of cognitive distortions "everyone" and "everybody." But 200 years ago these formulations were much less popular than "all the world," which is not included in the author's sample. If you just include the modern words, it's no surprise that cognitive distortions are increasing. But if you made the list in 1800 and only included the older formulation, you would conclude that cognitive distortions are decreasing. https://books.google.com/ngrams/graph?content=everybody%2Call+the+world%2Ceveryone&year_start=1800&year_end=2019&corpus=26&smoothing=3&direct_url=t1%3B%2Ceverybody%3B%2Cc0%3B.t1%3B%2Call%20the%20world%3B%2Cc0%3B.t1%3B%2Ceveryone%3B%2Cc0#t1%3B%2Ceverybody%3B%2Cc0%3B.t1%3B%2Call%20the%20world%3B%2Cc0%3B.t1%3B%2Ceveryone%3B%2Cc0 https://books.google.com/ngrams/graph?content=everybody%2Cal...

cycomanic 4y ago
I think the authors acknowledge that. And I agree with you this definitely should be investigated further. Also I have not read ref 17 yet, but I don't know how well established the method of identifying cognitive dissonance through ngrams is.
darkerside 4y ago
Where did they find the text samples? It strikes me that the corpus of informal text samples has skyrocketed in the past two decades. Remind me of how most heat maps are just population density maps.
- cycomanic 4y ago
  Google books it says so in the article
- antognini 4y ago
  It's the Google N-grams dataset. It covers all the books that Google has scanned in their corpus, which, as I understand it, is virtually every book that's been published in the 20th century (at least for English). I agree that this might just be measuring increasing informality in published works.
  - lumost 4y ago
    This could also be a bias in the google books dataset. As I recall, the older google books dataset relied heavily on library preservation. It's probable that library curation would filter books for utility, longevity, and educational purpose. These filters would certainly penalize informal books.
  - TimTheTinker 4y ago
    My first thought as I scanned the abstract was that since the '90s, the volume of novels with informal dialog has gone through the roof -- especially novels for teens and young adults.
    - permo-w 4y ago
      the amount of trashy celebrity autobiographies/books written by self-help gurus/informal-style recipe books being published has almost certainly risen in recent years too
- permo-w 4y ago
  >informal text informal and by less educated people. obviously there will be big exceptions, but I would imagine the % of computer-indexed text (i.e. books and newspapers) from between 1895 and 1995 that wasn't written by someone with a degree is not very high. similar to the way you can get a distorted view of the Romans (or nearly any literate historical society) because nearly (if not) all of their literature was written by the educated upper classes
pedrosbmartins 4y ago
I believe they tackle this exact bias. From the Sampling Effects section: > Second, the choice of CDS n-grams could lead to a "recency bias” in our results, explaining their rise in prevalence in recent decades. We control for this effect with a null model that samples random n-grams more frequently from recent books, due to rapidly increasing publication volume since 1895, thereby inducing a bias toward more recent language. We observe increases of CDS n-gram prevalence well above levels predicted by this null model
- bryanrasmussen 4y ago
  > thereby inducing a bias toward more recent language. We observe increases of CDS n-gram prevalence well above levels predicted by this null model I don't get why this would work? I get the null model predicting a bias X, and I guess they have a greater bias X + Y, but I don't see how that handles their choice of cognitive distortion signifiers being biased? I mean isn't it likely their choice of signifiers matches to Y?
  - lumost 4y ago
    they are claiming that their basket of distortion N-Grams became more common faster than other randomly sampled n-grams from recent works. This seems like an interesting approach to controlling for the bias, but I'd expect the random sampling would bias lower than a specific sample as a random sampling of n-grams would pick up a lot of english grammar which hasn't changed in many years.
    - geysersam 4y ago
      Maybe they should construct a list of n-grams indicative of "non-depressed"/healthy people, and run the same analysis on that list.
- layer8 4y ago
  They also address this to some degree in the Language effects section.
permo-w 4y ago
are "OK but" and "OK yet" new formulations? "Okay" is an Americanism from the early mid 1800s that started to be used in its current form in the late 1800s. are the "but" and "yet" attachments genuinely new or recent?

cycomanic 4y ago

I think the authors acknowledge that. And I agree with you this definitely should be investigated further. Also I have not read ref 17 yet, but I don't know how well established the method of identifying cognitive dissonance through ngrams is.

darkerside 4y ago

Where did they find the text samples? It strikes me that the corpus of informal text samples has skyrocketed in the past two decades. Remind me of how most heat maps are just population density maps.

cycomanic 4y ago
Google books it says so in the article
antognini 4y ago
It's the Google N-grams dataset. It covers all the books that Google has scanned in their corpus, which, as I understand it, is virtually every book that's been published in the 20th century (at least for English). I agree that this might just be measuring increasing informality in published works.
- lumost 4y ago
  This could also be a bias in the google books dataset. As I recall, the older google books dataset relied heavily on library preservation. It's probable that library curation would filter books for utility, longevity, and educational purpose. These filters would certainly penalize informal books.
- TimTheTinker 4y ago
  My first thought as I scanned the abstract was that since the '90s, the volume of novels with informal dialog has gone through the roof -- especially novels for teens and young adults.
  - permo-w 4y ago
    the amount of trashy celebrity autobiographies/books written by self-help gurus/informal-style recipe books being published has almost certainly risen in recent years too
permo-w 4y ago
>informal text informal and by less educated people. obviously there will be big exceptions, but I would imagine the % of computer-indexed text (i.e. books and newspapers) from between 1895 and 1995 that wasn't written by someone with a degree is not very high. similar to the way you can get a distorted view of the Romans (or nearly any literate historical society) because nearly (if not) all of their literature was written by the educated upper classes

pedrosbmartins 4y ago

I believe they tackle this exact bias. From the Sampling Effects section: > Second, the choice of CDS n-grams could lead to a "recency bias” in our results, explaining their rise in prevalence in recent decades. We control for this effect with a null model that samples random n-grams more frequently from recent books, due to rapidly increasing publication volume since 1895, thereby inducing a bias toward more recent language. We observe increases of CDS n-gram prevalence well above levels predicted by this null model

bryanrasmussen 4y ago
> thereby inducing a bias toward more recent language. We observe increases of CDS n-gram prevalence well above levels predicted by this null model I don't get why this would work? I get the null model predicting a bias X, and I guess they have a greater bias X + Y, but I don't see how that handles their choice of cognitive distortion signifiers being biased? I mean isn't it likely their choice of signifiers matches to Y?
- lumost 4y ago
  they are claiming that their basket of distortion N-Grams became more common faster than other randomly sampled n-grams from recent works. This seems like an interesting approach to controlling for the bias, but I'd expect the random sampling would bias lower than a specific sample as a random sampling of n-grams would pick up a lot of english grammar which hasn't changed in many years.
  - geysersam 4y ago
    Maybe they should construct a list of n-grams indicative of "non-depressed"/healthy people, and run the same analysis on that list.
layer8 4y ago
They also address this to some degree in the Language effects section.

permo-w 4y ago

are "OK but" and "OK yet" new formulations? "Okay" is an Americanism from the early mid 1800s that started to be used in its current form in the late 1800s. are the "but" and "yet" attachments genuinely new or recent?