I tested adding things like tweets and reddit comments to our language model, and it didn't help the target model at all, even when that used less formal langua

I tested adding things like tweets and reddit comments to our language model, and it didn't help the target model at all, even when that used less formal language. Note however that the fine tuning stage adapts to the target corpus - it just doesn't require starting from random weights (so it's orders of magnitude faster).