This could be of interest: "GLaM: Efficient Scaling of Language Models with Mixture-of-Experts"[1] Looking at OpenAI's recent introduction of ChatGPT's chat hi

This could be of interest: "GLaM: Efficient Scaling of Language Models with Mixture-of-Experts"[1] Looking at OpenAI's recent introduction of ChatGPT's chat history and sharing features, it's plausible to suggest that the creation of a set of related documents (including chat interactions and documents sourced from plugins) could pave the way for "domain expert" models to participate in a group alongside "frozen" or "foundation" models. These domain expert models could be models that receive document content for inference. I've conducted experiments where I sent document fragments (sourced from a vector search) in one query and combined it with another query without including the fragments. The results suggest that this approach can occasionally enhance the quality of the outcomes and seems to prevent the model from generating wildly inaccurate responses. [1] https://arxiv.org/pdf/2112.06905.pdf https://arxiv.org/pdf/2112.06905.pdf