alexgarcia-xyz

6 ms·

31.
by alexgarcia-xyz 2y ago
Yes it is! Sorry for not following up there. Actually, when I first read that ticket, it started me down the rabbit-hole of "how can I make sqlite-vss" better, which eventually turned into "I should make sqlite-vec." So
32.
by alexgarcia-xyz 2y ago
I don't have plans for compressed vector storage. There is support for int8/binary quantization, which can reduce the size of stored vectors drastically, but impacts performance quite a lot. I'd like to support something like
33.
by alexgarcia-xyz 2y ago
It's not possible to match pgvector's syntax with SQLite. SQLite doesn't have custom indexes, and since I want to have full control of how vectors are stored on disk, the only real way to do that in SQLite is with virtual tab
34.
by alexgarcia-xyz 2y ago
It could! It's based on the official SQLite WASM build, so you can use the same persistent options[0] that are offered there. Not sure if IndexedDB is specifically supported, but localStorage/OPFS VFS is available. [0] https:&#x
35.
by alexgarcia-xyz 2y ago
That's great to hear, thanks for sharing! Will definitely have benchmarks to share later Would love to hear about the scale of the vector data you're working with in your local app. Do you actually find yourself with > 1 millio
36.
by alexgarcia-xyz 2y ago
Thanks for sharing! I've looked into usearch before, it's really sleek, especially all their language bindings. Though I want sqlite-vec to have total control over what stays in-memory vs on-disk during searches, and most vector s
37.
by alexgarcia-xyz 2y ago
I definitely plan to! I have a much larger list of SQLite extensions I've built here: https://github.com/asg017/sqlite-ecosystem Here's a few other references you may enjoy if you wanna learn more about SQLit
38.
by alexgarcia-xyz 2y ago
The initial v0.1.0 release will only have linear scans, but I want to support ANN indexes like IVF/HNSW in the future! Wanted to focus on fullscans first to make things easier. In my experiments you can get pretty far with linear scans
39.
by alexgarcia-xyz 2y ago
Author here — happy to answer any questions! This is more of a "I'm working on a new project" rather than an official release, the extension itself is still a work-in-progress. A link to the project: https://github
40.
by alexgarcia-xyz 2y ago
Thanks for the kind words! I started with code first — the extension itself is already mostly written[0]. But it's one of those "20% effort for 80% of the work," where the last 20% I need to write (error handling, fuzzy tests
41.
by alexgarcia-xyz 2y ago
Author here - ya "binary vectors" means quantizing to one bit per dimension. Normally it would be 4 * dimensions bytes of space per vector (where 4=sizeof(float)). Some embedding models, like nomic v1.5[0] and mixedbread's n
42.
by alexgarcia-xyz 3y ago
Re storing vectors in BLOB columns: ya, if it's not a lot of data and it's fast enough for you, then there's no problem doing it like that. I'd even just store then in JSON/npy files first and see how long you can g
43.
by alexgarcia-xyz 3y ago
Observable has other open source libraries with similar "generic" names: Plot, Runtime, Inputs. When speaking generically, most people say "Observable Plot" or "Observable Runtime." In projects where people alr
44.
by alexgarcia-xyz 3y ago
That's great to hear! The clustered index sounds really cool. Especially since SQLite tells you about ORDER BYs in xBestIndex (with the nOrderBy[0]), so it would be super cool to have super-fast ORDER BYs with those. Very interested to
45.
by alexgarcia-xyz 3y ago
This is an awesome project, I love SQLite extensions and I think they have a ton of use. Giants props to Dan here, I haven't seen many SQLite extensions written in Zig, and I'm learning a ton reading through the source code. The c
46.
by alexgarcia-xyz 3y ago
A JSON schedule format is coming soon: https://github.com/denoland/deno/pull/21340
47.
by alexgarcia-xyz 4y ago
Author here, happy to answer any questions! My "Introducing sqlite-vss: A SQLite Extension for Vector Search" [0] blog post has more details about the motivations behind this extensions, along with a demo. Includes a semantic sear
48.
by alexgarcia-xyz 4y ago
Not yet! Spatialite is a very big and intimidating codebase that's hard to compile. I've been mulling the idea of creating several smaller GIS sqlite extensions, especially since Spatialite development is starting to get stale, bu
49.
by alexgarcia-xyz 4y ago
I've tried it! For Rust SQLite extensions, I'm able to use some cross compilers for some extra ARM targets on linux[0], I just haven't had the time to do it for all extensions yet. For MacOS arm builds cargo has the `aarch64-
50.
by alexgarcia-xyz 4y ago
I haven't worked much with Ruby, but would love to try it! These python package don't even need to compile native extensions, the pre-compiled SQLite extension just gets stuffed into the package under `package_data`. For ruby gems
51.
by alexgarcia-xyz 4y ago
It's possible but very difficult. For extensions built purely in C, you can statically compile extensions into a SQLite WASM build, which I have a few demos of with sqlite-lines [0] and sqlite-path[1]. For extensions but in Rust howeve
52.
by alexgarcia-xyz 4y ago
Author here, happy to answer questions! Simon already gave a great intro to my SQLite extensions, and I just want to point folks to Anton Zhiyanov's sqlean project, for pure-C extensions: https://github.com/nalgeon&#x2F
53.
by alexgarcia-xyz 4y ago
Author here, happy to answer any questions! Here's a few links that may be useful: Direct GitHub link: https://github.com/asg017/sqlite-loadable-rs A soon-to-be-released sqlite-loadable extension that's the f
54.
by alexgarcia-xyz 4y ago
Yeah, "dont make network requests from the database" is a common antipattern, but it doesn't quite apply with sqlite-http. It's mainly meant for data processing scripts that use SQL, which is typically on a local databas
55.
by alexgarcia-xyz 4y ago
re pipelining: unless Go's network package does this magically under the hood, then sqlite-http doesn't take advantage of that! If you think that would be a nice feature, file an issue and I'll see what can be done How comple
56.
by alexgarcia-xyz 4y ago
On SSRF: There is a "no network"[0] option for the extension, if you don't trust SQL input. This is what the example Datasette server[1] uses Also to note, this is mainly meant for use with local data processing, I doubt ther
57.
by alexgarcia-xyz 4y ago
Author here, happy to answer questions! Direct link to the project: https://github.com/asg017/sqlite-http Few other projects of interest: - sqlite-html: https://github.com/asg017/sqlite-html - sql
58.
by alexgarcia-xyz 4y ago
totally not dumb - in general there isn't much upside. Maybe there's a usecase in server-side rendering HTML where you already use SQLite somewhere, or in a CLI tool if you're using sqlite3. But I doubt sqlite-html will be as
59.
by alexgarcia-xyz 4y ago
Ah yes, my bad, gonna publish that soon! Will be an extension for things like `select http_get_body(' https://google.com ')`
60.
by alexgarcia-xyz 4y ago
Hey, author here, happy to answer questions! A few other recent posts/tools that you may be interested in: - sqlite-lines discussion from a few days ago: https://news.ycombinator.com/item?id=32288165 - htmlq, Rust CLI

More ›