10 ms·

Thanks for the insights. Any particular resources you’d recommend to learn more about this? So if I’m understanding it correctly, there’s one consistent way t

Thanks for the insights. Any particular resources you’d recommend to learn more about this? So if I’m understanding it correctly, there’s one consistent way that Ollama will vectorize a set of text. Perhaps there are various ways one could, but Ollama chooses one. What about multimodality? Ie taking a context from a prompt to llava to identify an image, with further questions about the contents of that image? Any non-llava model would definitely hallucinate, but would llava?

CuriouslyC 3y ago
Multimodal models use embeddings as well, the difference there is that they've been trained to associate the same position in latent space to text and to the image that text describes, that way it can turn a textual response into an image and vice versa. A lot of models use CLIP, an embedding method from openAI.