3 ms·

Which open source libraries is pinecone wrapping?

by etaioinshrdlu 4y ago

fdgsdfogijq 4y ago
FAISS
- gk1 4y ago
  This is incorrect.
  - Sommer 4y ago
    Strange comments and denial coming from the person who submitted a post about pinecone wrapping Faiss a year ago https://news.ycombinator.com/item?id=27502458 https://news.ycombinator.com/item?id=27502458
  - etaioinshrdlu 4y ago
    Yeah, what's this landing page about? https://www.pinecone.io/managed-faiss/ https://www.pinecone.io/managed-faiss/
    - gk1 4y ago
      That's a very old and experimental page -- hence the "early access" mentions. Glad you pointed it out so we can delete it.
gk1 4y ago
I’m not sure where the other commenter gets their confidence, but Pinecone is not wrapping any open source vector-search library. We offer three index types (in-memory, in-memory graph-based, hybrid memory + disk), and all are proprietary. We do have articles about Faiss and HNSW and all sorts of other vector-search and NLP topics, so it’s possible that’s where the confusion comes from.
- obert 4y ago
  so, how does your proprietary solution compare against FAISS, eg with 10M dense vectors of 1024 dimensions?
  - pbadams 4y ago
    They used to publish some benchmarks on their site, but seem to have removed them. You can find them on archive.org[1]. I guess it is understandable, since vector search performance is pretty unpredictable, and depends on a lot of factors. If their target market is people who want vector search without needing to read a bunch of papers first, benchmarks might be more confusing than they are helpful. edit: While I do think it's understandable, it's not great for transparency. Even if they don't want to open-source their index, I would admire it if they were willing to give ann-benchmarks[2] an API key to publish some independent results. Disclaimer: I work on vector search at a different company [1] https://web.archive.org/web/20210227105542/https://www.pinecone.io/learn/similarity-search-benchmarks/ https://web.archive.org/web/20210227105542/https://www.pinec... [2] https://github.com/erikbern/ann-benchmarks https://github.com/erikbern/ann-benchmarks

Which open source libraries is pinecone wrapping?

fdgsdfogijq 4y ago

FAISS

gk1 4y ago
This is incorrect.
- Sommer 4y ago
  Strange comments and denial coming from the person who submitted a post about pinecone wrapping Faiss a year ago https://news.ycombinator.com/item?id=27502458 https://news.ycombinator.com/item?id=27502458
- etaioinshrdlu 4y ago
  Yeah, what's this landing page about? https://www.pinecone.io/managed-faiss/ https://www.pinecone.io/managed-faiss/
  - gk1 4y ago
    That's a very old and experimental page -- hence the "early access" mentions. Glad you pointed it out so we can delete it.

gk1 4y ago

I’m not sure where the other commenter gets their confidence, but Pinecone is not wrapping any open source vector-search library. We offer three index types (in-memory, in-memory graph-based, hybrid memory + disk), and all are proprietary. We do have articles about Faiss and HNSW and all sorts of other vector-search and NLP topics, so it’s possible that’s where the confusion comes from.

obert 4y ago
so, how does your proprietary solution compare against FAISS, eg with 10M dense vectors of 1024 dimensions?
- pbadams 4y ago
  They used to publish some benchmarks on their site, but seem to have removed them. You can find them on archive.org[1]. I guess it is understandable, since vector search performance is pretty unpredictable, and depends on a lot of factors. If their target market is people who want vector search without needing to read a bunch of papers first, benchmarks might be more confusing than they are helpful. edit: While I do think it's understandable, it's not great for transparency. Even if they don't want to open-source their index, I would admire it if they were willing to give ann-benchmarks[2] an API key to publish some independent results. Disclaimer: I work on vector search at a different company [1] https://web.archive.org/web/20210227105542/https://www.pinecone.io/learn/similarity-search-benchmarks/ https://web.archive.org/web/20210227105542/https://www.pinec... [2] https://github.com/erikbern/ann-benchmarks https://github.com/erikbern/ann-benchmarks