3 ms·
> You can just keep a numpy array in memory with vectors for that and use a matmul for RAG. ... what? Tell me more.
by all2 3y ago
> You can just keep a numpy array in memory with vectors for that and use a matmul for RAG.
... what? Tell me more.
- JimmyAustin 3y agoThe model they use (BAAI/bge-small-en-v1.5) produces embeddings 384 wide, at float32 that equals 1536 bytes each. The size of all the vectors for 100 items is 153kb. Calculating the dot product of that against a query will be under measured in nano seconds, even with a naive implementation.
- cheesyFishes 3y agoThis is actually what llama-index does if you don't use a vectordb integration