Chain of thought is an in-context technique. The ensemble of models concept that GPT-4 supposedly uses works within the model itself. My understanding of the re

Chain of thought is an in-context technique. The ensemble of models concept that GPT-4 supposedly uses works within the model itself. My understanding of the relevant papers is poor, but my impression is that there are layers that help the model select which two of the 16 different “experts” should contribute to the generation of the next token. The 16 models in the ensemble are all trained in the same way, but attend to the input data differently. It’s a little like how multi-headed attention works by splitting the input embeddings into typically 8 parts and having a set of query, key, and value vectors each train on their own 1/8th slice of the input embeddings. Although there is no “meaning” to these slices, nonetheless the KQV vectors for each head will learn different relationships between the inputs simply because they were trained differently.