3 ms·
Interview with David Garnitz, Creator of VectorFlow
VectorFlow (open-sourced on GH) is a new tool for ingesting data into Vector Databases such as Weaviate! There is quite an interesting End-to-End stack emerging at the ingestion layer, from:
- Retrieving data from misc. sources such as Slack, Salesforce, GitHub, Google Drive, Notion, ...
- To then Chunking the Text (maybe with the use of Visual Document Layout parsers like what Unstructured is imagining)
- Extracting Metadata potentially (say the "age" of an NBA player as in the Evaporate-Code+ research)
- Then sending this data off to embedding model inference and unpacking that can of worms from inference acceleration to load balancing
- Finally -- importing the vectors themselves to Weaviate!
I learned so much from this conversation, I really hope you enjoy listening and please check out VectorFlow!
https://www.youtube.com/watch?v=efAvOsO8Gik