Very odd study. The usual reason why people introduce sentiment analysis into a model is that they don't have enough data or aren't comfortable enough with NLP

Very odd study. The usual reason why people introduce sentiment analysis into a model is that they don't have enough data or aren't comfortable enough with NLP to train a classifier against their own labels. Instead they grab pre-trained sentiment analysis model off the shelf (e.g. Vader) and now the have the much simpler problem of determining if a binary variable (sentiment positive/negative) is related some response variable which can be tackled with statistics 101. This can also be seen as an attempt to leverage transfer learning because such models are trained on a very large corpus with gold-standard labels; this can be beneficial when your corpus is too small to be a representative sample of the entire english language. But in this study, they manually label and train 8,000 sentences taken from their 10-K dataset. (A terribly small corpus, but the unsupervised word embedding step should help with that, at least a little.) Not great, not terrible. But they label them with sentiment and only later look for an association with the real variable of interest (abnormal return!) Why the extra step? If you're training your own model anyway, why not label the data with the real variable of interest from the start? I also don't see any attempt to control for the obvious confounding variables, such as the actual numbers in the 10-K. This leaves open the question of whether the effect is mediated by the other, more structured information in the filing. It's entirely possible for sentiment to have some mutual information with the variable of interest but for its benefit to entirely disappear once these are taken into account (a phenomenon known as mediation.) I would prefer to see this a single end-to-end model which is fed both the unstructured text data and the structured financial information as input features and the actual performance change as the output. The actual contribution of the unstructured text data could then be assessed with a tool like LIME (Local Interpretable Model-Agnostic Explanations.)