4 ms·
It's all of the above. txtai uses transformers to transform data (text, images, audio) into embeddings. Those embeddings are then loaded into an approximate ne
by neumll 5y ago
It's all of the above.
txtai uses transformers to transform data (text, images, audio) into embeddings. Those embeddings are then loaded into an approximate nearest neighbor index for search. On top of that, content is loaded into a relational database to support SQL based filtering. It's trying to get the best of both vector/similarity search alongside standard structured search using a SQL syntax.
This can be run in-process or via an API - https://neuml.github.io/txtai/api/ https://neuml.github.io/txtai/api/
Components can be split up, for example there could be a server that vectorizes text into embeddings and another server that hosts the indexes.
There is also a pipeline and workflow framework (https://neuml.github.io/txtai/workflow/ https://neuml.github.io/txtai/workflow/). This component has modules that assists with splitting data, transforming, summarizing, translating, parsing tabular content. Workflows can be used purely for transformations or as a driver to load data.