7 ms·
VectorDB: Vector Database Built by Kagi Search
- altdataseller 3y agoWhere does it save the data to? How is it persisted? Is there any limitation to this? Does it work with text 500-1000 words? Does it work well with text that aren’t full sentences? Ie. Are just a collection of phrases?
- tyingq 3y agoThe README.md example just has it in memory. Looking at the source, the Storage class uses a plain file and python's pickle module: https://github.com/kagisearch/vectordb/blob/main/vectordb/storage.py https://github.com/kagisearch/vectordb/blob/main/vectordb/st...
- mathverse 3y agoWonder why Crystal was not used.
- freedomben 3y agoCrystal is a super cool and underrated language. But why do you mention it here? Does kagi use crystal in other places?
- mathverse 3y agoThey use it for search afaik.
- SushiHippie 3y agoYep[0] 70k lines of crystal code for kagi search[1] [0] https://help.kagi.com/orion/company/hiring-kagi.html#full-time-positions https://help.kagi.com/orion/company/hiring-kagi.html#full-ti... [1] CrystalConf talk from a Kagi Tech Leas https://www.youtube.com/watch?v=r7t9xPajjTM https://www.youtube.com/watch?v=r7t9xPajjTM
- darnthenuggets 3y agoI can’t decide if I hate architecture-by-drawio or not.
- YellowSuB 3y agoYes it is made with Crystal
- synergy20 3y agoI read somewhere that typically crystal is adopted by ruby developers not sure if there is something similar in Python
- tokai 3y agoNim maybe?
- 7thaccount 3y agoNim is probably closest, but it's probably more influenced by pascal, modula-2, Oberon... something like that iirc
- synergy20 3y agoit is, it's still fairly complex. wish there is something simpler but statically-compiled with python syntax, something like crystal but for python developers.
- fabianhjr 3y agoScala 3 with Braceless Syntax (scala-native for x86) https://docs.scala-lang.org/scala3/reference/other-new-features/indentation.html https://docs.scala-lang.org/scala3/reference/other-new-featu...
- synergy20 3y agokotlin seems to be the Java trend now,anyway, don't know much about java
- 3y ago
- promiseofbeans 3y agoBased on their job postings, they use it for most of their back-end: https://help.kagi.com/kagi/company/hiring-kagi.html#full-time-positions https://help.kagi.com/kagi/company/hiring-kagi.html#full-tim...
- ipsum2 3y agoThis is a wrapper around FAISS, a vector search library. FAISS has a simple API, so it might be a better for your use case if you don't need heavyweight libraries that VectorDB requires, which include PyTorch, Tensorflow, and Transformers.
- dmezzetti 3y agoThis is possible but how would you encode your vectors?
- sharemywin 3y agowhat's wrong with these? https://www.sbert.net/docs/pretrained_models.html https://www.sbert.net/docs/pretrained_models.html
- dmezzetti 3y agoAgreed and the question. If you take out the vector encoding part and just use Faiss, then what would be the proposal to encode vectors.
- lmeyerov 3y ago... Whatever you want? Even with a vector DB, embeddings are typically BYO to beginwith - dependencies not in your DB and computed in pipelines well before it. It's handy for small apps to do in DB and have support in DB for some queries, but as things get big, doing all in the DB gets weirder... Edit: I see you run a vectordb company, so your question makes more sense
- dmezzetti 3y agoTrue but the comment discussed that it should remove "heavyweight libraries". Unless you use an API service, those libraries will need to be imported somewhere. It doesn't necessarily have to run on the same server as the database.
- PossiblyKyle 3y agoI fail to understand why I should use it over a different embedded vector DB like LanceDB or Chroma. Both are written in more performant languages, have a simple API with a lot of integrations and power if one needs it
- dmezzetti 3y agoTo be fair, Chroma is also written in Python. And while LanceDB and others are written in Rust, that doesn't automatically give it super powers.
- PossiblyKyle 3y agoFair point, then you could claim it's similar to this DB with its reliance on Faiss. Despite that, Chroma at this point is more feature rich. I was mostly referring to this https://thedataquarry.com/posts/vector-db-1/ https://thedataquarry.com/posts/vector-db-1/ You are not wrong about the performance from Rust, but LanceDB is inherently written with performance in mind. SIMD support for both x86 and ARM, and an underlying vector storage approach that's built for speed (Lance)
- marcinzm 3y ago>more feature rich Not necessarily a good thing when the product is made by a VC backed startup that may die or pivot in six months leaving you the need to maintain it yourself.
- dmezzetti 3y agoI've seen a number of projects come over the last couple years. I'm the author of txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai) which I started in 2020. How you approach performance is the key point. You can write performant code in any language. For example, for standard keyword search, I wrote a component to make sparse/keyword search just as efficient as Apache Lucene in Python. https://neuml.hashnode.dev/building-an-efficient-sparse-keyword-index-in-python https://neuml.hashnode.dev/building-an-efficient-sparse-keyw....
- 3y ago
- andy99 3y agoThe two most interesting things to me in a "minimal" framework would be eliminating dependency on HF transformers and helping customize chunking. Not a knock against this project, I see where it can be helpful.
- dmezzetti 3y agoMy assumption is that this gives you the ability to locally encode vectors. This is useful for those not using API services to build their vectors.
- andy99 3y agoTransformer inference is ~60 lines of numpy[0] (closer to 500 once you add tokenization etc). It would be nice to just have this and not all of pytorch and transformers. [0] https://jaykmody.com/blog/gpt-from-scratch/ https://jaykmody.com/blog/gpt-from-scratch/
- dmezzetti 3y agoWhat about models besides GPT? Most of the popular vector encoding models aren't using this architecture. If you really didn't want PyTorch/Transformers, you could consider exporting your models to ONNX (https://github.com/microsoft/onnxruntime https://github.com/microsoft/onnxruntime).
- cherryteastain 3y agoIt's 60 lines for CPU only inference, which'll be slow. If you want GPU acceleration it'll be a lot more than 60 lines.
- jsimian 3y agoYeah, I just tried this out (props to the devs, super easy to set up) and my main gripe is the chunking algorithms aren't great - could be alot more useful with a context option that gives surrounding results to search results. The sliding window chunking method always cuts off the start of sentences.
- osigurdson 3y agoWhat is being used to actually create the embeddings?
- dmezzetti 3y agoThe library is open source. Here's where you can see how they're creating the embeddings. https://github.com/kagisearch/vectordb/blob/main/vectordb/embedding.py#L36 https://github.com/kagisearch/vectordb/blob/main/vectordb/em...
- simonw 3y agohttps://github.com/kagisearch/vectordb/blob/453bb658bb7108389cbbdbf35051d22696408a1c/vectordb/embedding.py#L37 https://github.com/kagisearch/vectordb/blob/453bb658bb710838... Looks like it uses one of these, depending on your settings: Fast model: google/universal-sentence-encoder/4 Multilingual model: universal-sentence-encoder-multilingual-large/3 Normal model (Alternative): BAAI/bge-small-en-v1.5 Best model: BAAI/bge-base-en-v1.5
- syntaxing 3y agoI always run into the issue of loading existing embedding. For instance, I want to embed a folder which have five files and two are new, is there a way to only add the two new files to the stored embedding using this or chromaDB?
- rasengan 3y agoKeep track of the files and when you see new ones only add those.
- conception 3y agohttps://github.com/wallabag/wallabag https://github.com/wallabag/wallabag No one has mentioned wallabag yet, so wanted to. Been working well for me - has apps and extensions. If you’re not excited to self-host - https://www.wallabag.it/en https://www.wallabag.it/en has been flawless with the exorbitant price of… 11 euro a year.
- 0x6c6f6c 3y agoThis isn't even slightly related to a vector database. I like Wallabag but this comes off as a shameless plug.
- edwinbalani 3y agoWrong article I think; there was another post about a bookmark manager on here.
- 0x6c6f6c 3y agoIt happens. Rereading my comment I hope that didn't come too rude
- politelemon 3y agoIf a team has an operational store already in Postgres, wouldn't it be best to just use the PGVector extension? So the data and the vector search functionality sit together, and there's one less moving part in the tech stack to manage?
- dmezzetti 3y agoThis recent thread discussed this - https://news.ycombinator.com/item?id=38416994 https://news.ycombinator.com/item?id=38416994
- m3kw9 3y agoIt shouldn’t be legal to call your vectordb vectordb
- andy99 3y agoI prefer it to, say, vectr (such contractions used to be popular) or 'barbershop' or a similar single irrelevant word name as is popular now.
- worldsayshi 3y agoAtt least with barbershop it becomes much more searchable. "barbershop db" will very likely succeed.
- andy99 3y agoGood point. It trades off searchability with understanding (or getting an indication of) what it is when you first hear of it.
- worldsayshi 3y agoI think there's two ways to name things and I think the cattle or pet metaphor is useful here. If the typical user will use the thing once and then forget about it or if they are not going to keep it in mind very often make sure to give it a descriptive name. But for some things having a descriptive name might not be the most important aspect of its name. Having a findable or memorable name might be more important. And maybe sometimes giving it a descriptive name might actually become a problem because its function might change over time.
- LtWorf 3y agoSomeone decided to call a software "informatica". A british recruiter was quite insistent to have a call… turns out because she read "informatica" on my profile (as in, laurea in informatica), asked me how many years of experience I had with the tool, I replied "I just heard of it 2 minutes ago when you first mentioned it". Then got mad at me for having written the word "informatica" on my profile.
- richardanaya 3y agoKagi search is amazing, i've used it for the last few months. If this is what they are using to power it, i'm optimistic.
- iTokio 3y agoFrom the GitHub repo: > Thanks to its low latency and small memory footprint, VectorDB is used to power AI features inside Kagi Search.
- stainablesteel 3y agosuper cool! looks like i can make a local search engine out of the massive datahoard of pdf books and articles i have
- deleted 3y ago[deleted]
- hubraumhugo 3y agoIs there any kind of comaprison of the different vector DBs? What would you choose for different use cases? How do they differ?
- dmezzetti 3y agoThis thread from a few months ago is a good read - https://news.ycombinator.com/item?id=36943318 https://news.ycombinator.com/item?id=36943318
- pknerd 3y agoThis or anyone suggest some other db/lib for local QnA like testing on Mac?
- dmezzetti 3y agoJust posted this thread that gives an in-depth look at LLM frameworks (local ones included) - https://news.ycombinator.com/item?id=38422264 https://news.ycombinator.com/item?id=38422264
- ianpurton 3y agohttps://bionic-gpt.com/blog/you-dont-need-a-vector-database/ https://bionic-gpt.com/blog/you-dont-need-a-vector-database/
- freediver 3y agoDev here. Thanks for submitting. To be fair this is not really a database, but a wrapper around few primitives such as locally-ran embeddings and FAISS/mrpt with a ton of benchmarking behind the doors to offer sane defaults that minimize latency. Here is an example Colab notebook [1] where this is used to filter the content of the massive Kagi Small Web [2] RSS feed based on stated user interests: [1] https://colab.research.google.com/drive/1pecKGCCru_Jvx7v0WRNrW441EBlcS5qS?usp=sharing https://colab.research.google.com/drive/1pecKGCCru_Jvx7v0WRN... [2] https://kagi.com/smallweb https://kagi.com/smallweb