4 ms·

Really just chroma + openai is all you need. Chroma makes this easy https://docs.trychroma.com/embeddings https://docs.trychroma.com/embeddings. All you have t

Really just chroma + openai is all you need. Chroma makes this easy https://docs.trychroma.com/embeddings https://docs.trychroma.com/embeddings. All you have to do is chunk... I'd start at a paragraph and experiment.

chaxor 3y ago
This is not always good advice. Many people require to not use off premise models, due to data ownership issues. I would therefore suggest a better default for this, such as BERT+Qdrant. It would be wonderful if there were a simpler (single file, SQLite or DuckDB like) database for vectors than the complex (and in some cases, unfortunately cloud-based) ones available now.
- jeffchuber 3y ago
  ah sorry, i should read OP better - chroma's default embedding model is sentence transformers - and we have many other integrated - https://github.com/chroma-core/chroma/blob/main/chromadb/utils/embedding_functions.py https://github.com/chroma-core/chroma/blob/main/chromadb/uti... > It would be wonderful if there were a simpler (single file, SQLite or DuckDB like) database for vectors than the complex (and in some cases, unfortunately cloud-based) ones available now. This is literally chroma!
  - chaxor 3y ago
    Admittedly, I don't know much about chroma, but it seems similar to Qdrant to me. Perhaps I'm missing something. It doesn't appear that it stores everything in 1 singledb file, but rather a plethora of files in some directory. Although it does appear that it runs local, which is a huge plus.
- fzliu 3y ago
  You might enjoy milvus-lite: https://zilliz.com/blog/exploring-magic-vector-databases-jupyter-notebooks https://zilliz.com/blog/exploring-magic-vector-databases-jup... pip install milvus
drewcperkins 3y ago
Chroma is just a wrapper for Clickhouse. I’d go with something like Weaviate or Qdrant that is a lot more mature as far as vector databases.
redskyluan 3y ago
how do you define chroma as a vector db? a wrapper on top of other databases and hnsw? What about a billion vectors in house~ is chroma all we need?

chaxor 3y ago

This is not always good advice. Many people require to not use off premise models, due to data ownership issues. I would therefore suggest a better default for this, such as BERT+Qdrant. It would be wonderful if there were a simpler (single file, SQLite or DuckDB like) database for vectors than the complex (and in some cases, unfortunately cloud-based) ones available now.

jeffchuber 3y ago
ah sorry, i should read OP better - chroma's default embedding model is sentence transformers - and we have many other integrated - https://github.com/chroma-core/chroma/blob/main/chromadb/utils/embedding_functions.py https://github.com/chroma-core/chroma/blob/main/chromadb/uti... > It would be wonderful if there were a simpler (single file, SQLite or DuckDB like) database for vectors than the complex (and in some cases, unfortunately cloud-based) ones available now. This is literally chroma!
- chaxor 3y ago
  Admittedly, I don't know much about chroma, but it seems similar to Qdrant to me. Perhaps I'm missing something. It doesn't appear that it stores everything in 1 singledb file, but rather a plethora of files in some directory. Although it does appear that it runs local, which is a huge plus.
fzliu 3y ago
You might enjoy milvus-lite: https://zilliz.com/blog/exploring-magic-vector-databases-jupyter-notebooks https://zilliz.com/blog/exploring-magic-vector-databases-jup... pip install milvus

drewcperkins 3y ago

Chroma is just a wrapper for Clickhouse. I’d go with something like Weaviate or Qdrant that is a lot more mature as far as vector databases.

redskyluan 3y ago

how do you define chroma as a vector db? a wrapper on top of other databases and hnsw? What about a billion vectors in house~ is chroma all we need?