2 ms·

Very cool! I’ve been looking at using SQLite-vss for fast vector sorting, but it requires CGO. I imagine if I do WASM compilation directly from C, I might be ab

by koeng 3y ago

Very cool! I’ve been looking at using SQLite-vss for fast vector sorting, but it requires CGO. I imagine if I do WASM compilation directly from C, I might be able to get around using CGO. Do you have any idea able the feasibility of that using your package? Thanks!

ncruces 3y ago
One problem with WASM is that, at this point (2023), the only easy way of building that, is to statically link everything C into one WASM blob (so you can't use my blob). But you can try to use my build scripts, and tweak my code, to build SQLite+sqlite-vss+Faiss into one large WASM. If you use my patches/imports/exports/config for SQLite, my binding will gladly take your WASM blob, without modification to the Go code. All you need to do is set this global variable (and not import package embed, which embeds my WASM and sets this variable for you at init): https://pkg.go.dev/github.com/ncruces/go-sqlite3#pkg-variables https://pkg.go.dev/github.com/ncruces/go-sqlite3#pkg-variabl... Ping me on GitHub if you need help. Vector search is on my meters long todo list, but I'd rather do it in Go if at all possible.
- koeng 3y ago
  How would you do it in Go? It is my understanding that FAISS is integrated into the SQL database itself (using embedding in the table), so writing it in Go wouldn't integrate it in the same way. > But you can try to use my build scripts, and tweak my code, to build SQLite+sqlite-vss+Faiss into one large WASM That makes perfect sense to me! I'm looking into your build scripts right now. I'm pretty proficient in Go, but haven't touched wasm or C much, so. I may reach out with more questions.
  - ncruces 3y ago
    You need a virtual table (same as sqlite-vss) and a library that does cosine similarity indexing (Faiss here)… which I know little about. My plan was to evaluate Go for cosine similarity indexing, and if I found something promising, port the virtual table.

by koeng 3y ago

ncruces 3y ago

One problem with WASM is that, at this point (2023), the only easy way of building that, is to statically link everything C into one WASM blob (so you can't use my blob). But you can try to use my build scripts, and tweak my code, to build SQLite+sqlite-vss+Faiss into one large WASM. If you use my patches/imports/exports/config for SQLite, my binding will gladly take your WASM blob, without modification to the Go code. All you need to do is set this global variable (and not import package embed, which embeds my WASM and sets this variable for you at init): https://pkg.go.dev/github.com/ncruces/go-sqlite3#pkg-variables https://pkg.go.dev/github.com/ncruces/go-sqlite3#pkg-variabl... Ping me on GitHub if you need help. Vector search is on my meters long todo list, but I'd rather do it in Go if at all possible.

koeng 3y ago
How would you do it in Go? It is my understanding that FAISS is integrated into the SQL database itself (using embedding in the table), so writing it in Go wouldn't integrate it in the same way. > But you can try to use my build scripts, and tweak my code, to build SQLite+sqlite-vss+Faiss into one large WASM That makes perfect sense to me! I'm looking into your build scripts right now. I'm pretty proficient in Go, but haven't touched wasm or C much, so. I may reach out with more questions.
- ncruces 3y ago
  You need a virtual table (same as sqlite-vss) and a library that does cosine similarity indexing (Faiss here)… which I know little about. My plan was to evaluate Go for cosine similarity indexing, and if I found something promising, port the virtual table.