4 ms·
How to improve (or “fine-tune”) your OpenAI embeddings?
- VectifyAI 3y agoHow to improve OpenAI embeddings in your RAG pipeline? At Vectify AI, we have developed the Fine-Tuning Augmented Embedding (FAE), a method that can "fine-tune" the black box OpenAI's text embedding model on your data and boost retrieval performance. Retrieval Augmented Generation (RAG) is a notable framework that bridges Large Language Models (LLMs) with external knowledge. While a general-purpose pre-trained embedding model is commonly used for retrieval, it might not be optimal for specific domains. A recent blog post from Llamaindex introduced a method for fine-tuning the embedding model on domain-specific documents without labels and achieved impressive performance boosts with the lightweight BAAI/bge-small-en model. While the fine-tuned model shows promise, it doesn't outperform the OpenAI/text-embedding-ada-002 model. Ideally, we'd directly fine-tune OpenAI's embedding model on domain-specific data. However, it's served as a black box, with no dedicated fine-tuning API available. We introduce the Fine-tuning Augmented Embedding (FAE), which augments the OpenAI's black-box model with a tunable model. By fine-tuning just the trainable model component with domain-specific data, FAE can consistently outperform OpenAI in retrieval performance. More details can be found in our blog post. Also check out the open-sourced implementation of FAE in Vectify AI's Github repo.