7 ms·

Hi all, I'm the author - I'll be here to answer any questions you have!

Noe2097 5y ago
Hi! Yes! Nice article (articles actually). I realize you are hinting at it as the subject of a later article, but could you share with us a little bit more about single-stage-filtering :D ? how does it work? How can metadata and vector data "coexist" in a same index?
- gk1 5y ago
  I'm from Pinecone and can tell you we are figuring out now exactly how much to say about this. We know you want to know, and we also know other vector databases want to know -- because this is a highly requested feature and it's far from trivial to implement in a low-latency way. That's why we have to give the admittedly non-satisfying answer of "stay tuned!"
- NHQ 5y ago
  Neural networks.
- heipei 5y ago
  There are different approaches for turning vectors into terms which you can then simply index with something like Elasticsearch. There are papers by the Walmart and Jet.com teams that describe different approaches, starting with both binarised vectors as well as full float vectors. Not saying that this is what Pinecone is using, so it will be interesting to read their follow-up.
- jamesbriggs 5y ago
  Yes as gk1 said, we're figuring out what can be said! I can't fully answer your questions beyond, having both indexes together helps us speed up the search (thanks to a lack of network/serialization/parsing overhead), and because of the structure we store metadata tags in, we also don't need to check the metadata for every sample (hence the cool speedup you get when filtering) - sorry that we can't say anything more yet!
throwaway_pdp09 5y ago
Your damn animated gif has sucked up >4.7GB in my browser and crashed it. Not impressed. Did animation actually add anything significant anyway.
- smoyer 5y ago
  I'm not disputing that the image may not add anything to the article but if a 10MiB GIF image can crash your browser maybe you should look at other vendors. I'm on Firefox Developer Edition with approximately 100 tabs open and didn't notice that image as a resource hog.
  - throwaway_pdp09 5y ago
    It's animated (and there are 3 on that page totalling 35MB). Some systems unfold each frame into its own image and store that. Please open firefox to that page, leave it open for a few minutes and observe the memory. I suspect FX will bloat too.

Noe2097 5y ago

Hi! Yes! Nice article (articles actually). I realize you are hinting at it as the subject of a later article, but could you share with us a little bit more about single-stage-filtering :D ? how does it work? How can metadata and vector data "coexist" in a same index?

gk1 5y ago
I'm from Pinecone and can tell you we are figuring out now exactly how much to say about this. We know you want to know, and we also know other vector databases want to know -- because this is a highly requested feature and it's far from trivial to implement in a low-latency way. That's why we have to give the admittedly non-satisfying answer of "stay tuned!"
NHQ 5y ago
Neural networks.
heipei 5y ago
There are different approaches for turning vectors into terms which you can then simply index with something like Elasticsearch. There are papers by the Walmart and Jet.com teams that describe different approaches, starting with both binarised vectors as well as full float vectors. Not saying that this is what Pinecone is using, so it will be interesting to read their follow-up.
jamesbriggs 5y ago
Yes as gk1 said, we're figuring out what can be said! I can't fully answer your questions beyond, having both indexes together helps us speed up the search (thanks to a lack of network/serialization/parsing overhead), and because of the structure we store metadata tags in, we also don't need to check the metadata for every sample (hence the cool speedup you get when filtering) - sorry that we can't say anything more yet!

throwaway_pdp09 5y ago

Your damn animated gif has sucked up >4.7GB in my browser and crashed it. Not impressed. Did animation actually add anything significant anyway.

smoyer 5y ago
I'm not disputing that the image may not add anything to the article but if a 10MiB GIF image can crash your browser maybe you should look at other vendors. I'm on Firefox Developer Edition with approximately 100 tabs open and didn't notice that image as a resource hog.
- throwaway_pdp09 5y ago
  It's animated (and there are 3 on that page totalling 35MB). Some systems unfold each frame into its own image and store that. Please open firefox to that page, leave it open for a few minutes and observe the memory. I suspect FX will bloat too.