3 ms·

So from what I've seen pinecone db can't handle more than 700 dimensions, but OpenAI's embeddings model has around 1500 dimensions. Are people expected to perfo

• gk1 4y ago
  Hi, I'm from Pinecone. We support up to 20,000-dimensional vectors.[1] A meaningful percentage of our customers use OpenAI models. If you got that 700 number from our site or docs I'd love to know where so we could correct it. [1] https://docs.pinecone.io/docs/limits https://docs.pinecone.io/docs/limits
  • teaearlgraycold 4y ago
    > Each p1 pod has enough capacity for 1M vectors with 768 dimensions. > Each s1 pod has enough capacity for 5M vectors with 768 dimensions. I think that’s what I was remembering. I guess I just don’t understand what that means.
    • gk1 4y ago
      Got it, thanks. Each pod has a limit to how much data it can store. We want to give an example of how many vector embeddings you can store in a single pod. Since the data size is affected by number of dimensions, we have to choose a sample dimensionality for the example. We chose 768 since that was the most common last year (SBERT), although it might make sense to change it to 1536 (OpenAI).
      • teaearlgraycold 4y ago
        I think it’s confusing because as an end user I don’t yet know what a pod is or why they’re needed. I’m guessing this is related to space partitioning? It would probably be helpful to link to some docs on pods (are pods a leaky abstraction?) and first list the max size in number of floats or bytes and then say how many embeddings that would be with different models. Edit: Thanks for the response btw!