There was an interview with Zuckerberg about how they initially split training llama chat models on purely normal text and codellama on code, but later realized

There was an interview with Zuckerberg about how they initially split training llama chat models on purely normal text and codellama on code, but later realized that if they combine the training set they get a model that is better at both tasks than each specialized one was.