11 ms·

How I Built Zig-SQLite

225 points by Sphax 4y ago

longrod 4y ago
This looks really neat and handy especially the type annotations part. I always got frustrated when accidentally putting the wrong type into a column. How's the Language Server support for this? Last time I tried ZLS, it was quite well rounded so I am curious to know if this type annotations would work with it or not. Would be really cool if they do. I have never been able to fully adopt zig due to how frustrating working with strings is but I absolutely loved the comptime functionality.
- xmorse 4y ago
  ZLS currently does not run any comptime code as far as i know It would be cool to have an official Zig language server that can do it
  - Tiberium 4y ago
    I think the only real way you can run make a language server understand comptime code is using the compiler as a library, similar to how Nim does it - nimsuggest (the tool for autocompletion, definitions, etc) is basically the compiler itself with some nimsuggest-specific stuff, so it understand macros, templates, compile time code evaluation, etc.
    - gpderetta 4y ago
      Isn't that's how most LSP servers are implemented? I'm only vaguely familiar with clangd.
      - lillecarl 4y ago
        It's moving in that direction, but far from all are based on their compiler still.
      - ranfdev 4y ago
        Yes. An LSP has to work along with the compiler, or it has to become an alternative compiler. For example, from the article "Why LSP?" [1] "It is known that compilers are complicated, and a language server is a compiler and then some." [1]: https://matklad.github.io//2022/04/25/why-lsp.html https://matklad.github.io//2022/04/25/why-lsp.html
  - Shadonototra 4y ago
    from what i understood, that is what they are planning to do, have an official built in language server
- Sphax 4y ago
  zls doesn't support this unfortunately. I don't know what the plans are for comptime support.
codethief 4y ago
This is really cool, thanks for providing such an in-depth view into Zig's comptime features and how you used them!
- ewalk153 4y ago
  Agree. This is one of the most instructive tutorials on the comptime language feature.
  - Sphax 4y ago
    I didn't really intend to write a tutorial but glad you liked the post !
noodledoodletwo 4y ago
Cool article. I can't figure out why i am supposed to care about zig beyond it's fun? I get that it's interesting and more safe than C (honestly though what the hell isn 't). Say you write rust pretty regularly for new product development, what does zig offer to make my life better, my products more stable, etc?
- mongol 4y ago
  Yes, but say you are not yet writing Rust, perhaps you should choose Zig instead of it?
  - giancarlostoro 4y ago
    I think you're both asking the same question from different angles. If I understand correctly, and I could be misremembering but Zig is supposed to be compatible with C++ not just C, which is something that is not necessarily straight forward in other C / C++ competing languages. I hear even D has some issues with mangling and what not. In all honesty, I prefer the syntax of D over all the others, it feels the most like Java or C# but with a lot of modern benefits. D is trying to do too much though it feels like and I would love for the next D standard library to support OOTB similar to what Go supports, especially a very minimalist web server, I think all modern programming languages should be capable of spinning up web servers out of the box. This is one small detail Go got right in my opinion. Here's an article from the Chromium team on challenges they faced with trying to integrate Rust (or at least evaluating it) note the entry was last updated in 2020: https://www.chromium.org/Home/chromium-security/memory-safety/rust-and-c-interoperability/ https://www.chromium.org/Home/chromium-security/memory-safet...
    - krylon 4y ago
      I like that the web server in Go's standard library comes as a library, so you can embed a web server into your application, even if it otherwise isn't web-related at all, just to allow for some introspection and/or control. (To be fair, I think Python and Ruby have HTTP servers in their standard libraries as well, at least minimalist ones.)
curist 4y ago
By using comptime, the statement couldn't be runtime composed, right? That's currently the major holdback for me to spend more time on zig: if using comptime become more common in zig community, the libs could be less flexible to use. It feels sort of like function coloring to me, that the whole call chain also need to pass down the value as comptime variable. I've only spend 2 days with zig, so I would love to learn if I'm wrong on this subject.
- Sphax 4y ago
  Only the metadata of the statement is comptime, that is the type annotation for each bind parameters. So if you have this query SELECT * FROM user WHERE age = $age{u16} You _must_ provide a u16 bind parameter. However the value itself is of course not required to be comptime-known, that would make the whole thing unusable. For what it's worth there are in zig-sqlite variants of the method which bypass the comptime checks; they're not documented properly yet but see all methods named `xyzDynamic`, for example https://github.com/vrischmann/zig-sqlite/blob/master/sqlite.zig#L493-L498 https://github.com/vrischmann/zig-sqlite/blob/master/sqlite....
- skybrian 4y ago
  Often, comptime code couldn’t be executed at runtime because the language features it accesses aren’t available then. But I agree that if a parameter could go either way, you shouldn’t have to write two versions.
- throwawaymaths 4y ago
  Generally I wouldn't call zig comptime function coloring. (I have written a prime sieve algorithm that uses the runtime code to precalculate some primes at comptime. Yes, I had to be very careful about what was in the prime number algorithm, but comptime supports that level of complexity and it was certainly possible to call runtime-intended code at comptime. Can you call comptime-intended code at runtime? No? (Yes? B/c the call site is "in" the runtime code?) But just make it runtime code instead of comptime code?
SkeuomorphicBee 4y ago
In case anyone else is unfamiliar with Zig syntax and wondering: in Zig the .{ "somevalue"} represents an anonymous list literal [1], and .{.somename = "somevalue"} is an anonymous struct literal [2]. (A bit off topic rant, but Zig documentation is quite bad, it took me a lot more effort that it should to discover the facts above.) [1] https://ziglang.org/documentation/master/#Anonymous-List-Literals https://ziglang.org/documentation/master/#Anonymous-List-Lit... [2] https://ziglang.org/documentation/master/#Anonymous-Struct-Literals https://ziglang.org/documentation/master/#Anonymous-Struct-L...
- LAC-Tech 4y ago
  Zig is still not 1.0. I can forgive its lack of docs at this point, especially how surprisingly readable the std lib is.
- jmull 4y ago
  I thought the docs were quite good. I used "ziglearn" to understand ("chapter 1" and "chapter 2" have tons of foundational stuff): https://ziglearn.org/chapter-1/ https://ziglearn.org/chapter-1/ I used the language reference for the details: https://ziglang.org/documentation/master/#Introduction https://ziglang.org/documentation/master/#Introduction I agree the standard library docs aren't useful (unless something changed a lot in the last six months)... it seemed always better to just search and read the standard library source directly.
  - messe 4y ago
    > I agree the standard library docs aren't useful (unless something changed a lot in the last six months)... it seemed always better to just search and read the standard library source directly. They're automatically generated. As I understand it they've been holding off on improving them until they have the self hosted compiler working, as it will be easier at that point.
  - throwawaymaths 4y ago
    To be fair it's really hard to "search" for tuples and anyonymous structs and get an understanding from "reading the stdlib". It's also enough of a departure from C (but not obviously readable like types on the left) to warrant some sort of big red flag in the lang docs... Understanding this tripped me up a bit
nickysielicki 4y ago
constexpr/consteval/comptime/etc is a game changer for the systems programmer and it feels so close but yet so far. Can someone speak to what language has the best support for this? Some things that I feel are missing in C++: * arbitrary file I/O. I can take my compile time data and write a python script to put it in a std::array, but I shouldn’t have to. * non-fixed-sized containers, ie vector * a generic memoization utility in the standard library for caching. If I want my function to handle any input at runtime, but I want to pre-populate a cache at compile time for values that I know will be called, it’s doable but not as easy as it should be. (In general, given the amount of algorithms that rely on memoization, I’m somewhat surprised that Python is the only language I know that makes memoization as easy as a decorator).
- formerly_proven 4y ago
  > * arbitrary file I/O. I can take my compile time data and write a python script to put it in a std::array, but I shouldn’t have to. https://ziglang.org/documentation/0.9.1/#embedFile https://ziglang.org/documentation/0.9.1/#embedFile I haven't tried, but because it should just be equivalent to a string literal you should be able to further process it using comptime as well. > * non-fixed-sized containers, ie vector One way is to just have a function count how many entries you need and use arrays. This is all pretty straightforward because comptime. Real dynamic comptime containers are dependent on making a comptime allocator available. There's a ticket for that: https://github.com/ziglang/zig/issues/1291 https://github.com/ziglang/zig/issues/1291 > * a generic memoization utility in the standard library for caching. If I want my function to handle any input at runtime, but I want to pre-populate a cache at compile time for values that I know will be called, it’s doable but not as easy as it should be. (In general, given the amount of algorithms that rely on memoization, I’m somewhat surprised that Python is the only language I know that makes memoization as easy as a decorator). I don't think it's possible to use comptime to generate wrapper functions duplicating the signature of a given function, at least I haven't found a way to do so. The function signature has to be spelled out in the comptime code.
- dataangel 4y ago
  All of this is already done in Lisp. Greenspun's 10th rule.
voigt 4y ago
I see Zig, I vote up ;) Exciting new language with compelling features! Great work of the author explaining comptime!
- deleted 4y ago
  [deleted]
jhgb 4y ago
Here's an absolutely crazy idea that I had the other day... Since Zig has comptime, couldn't a Zig reimplementation of SQLite use comptime for compile-time query compilation? Including generating native code for fixed queries? So far I haven't been able to come up with any counterargument for why this wouldn't work.
- apavlo 4y ago
  > couldn't a Zig reimplementation of SQLite use comptime for compile-time query compilation? Including generating native code for fixed queries? SQLite already compiles queries into opcodes and then uses its own VM to interpret them. You would have to reimplement the internal VM.
  - jhgb 4y ago
    I'm aware of that, but these elementary operations have some definitions in code. Even just calling them in a fixed sequence (as opposed to dispatching at runtime one opcode at a time) would be something optimizable for a sufficiently smart compiler. However, given a good-enough design of the whole system, straight code generation from the query plan shouldn't be a problem either (and at that point, the Zig compiler should be able to do even more work with the generated query code). I mean, any reimplementation would presumably use Zig features heavily anyway -- otherwise you could just link with the C code.
- pvg 4y ago
  The counter-argument is interpreting the query is not what takes up the time in executing a typical RDBMS query. Imagine a db is just a key-value store, the query is some keys and the db just spits out whatever it found in a giant hashtable. That's not going to get faster if you 'compile' the query. It's not going to get much faster if your db is a more realistic three hashtables, two big arrays and a partridge in a b-tree.
  - jhgb 4y ago
    It might not be worth for everything, but extremely simple queries on extremely simple schemas would not be the goal here.

225 points by Sphax 4y ago

This looks really neat and handy especially the type annotations part. I always got frustrated when accidentally putting the wrong type into a column. How's the Language Server support for this? Last time I tried ZLS, it was quite well rounded so I am curious to know if this type annotations would work with it or not. Would be really cool if they do. I have never been able to fully adopt zig due to how frustrating working with strings is but I absolutely loved the comptime functionality.

xmorse 4y ago
ZLS currently does not run any comptime code as far as i know It would be cool to have an official Zig language server that can do it
- Tiberium 4y ago
  I think the only real way you can run make a language server understand comptime code is using the compiler as a library, similar to how Nim does it - nimsuggest (the tool for autocompletion, definitions, etc) is basically the compiler itself with some nimsuggest-specific stuff, so it understand macros, templates, compile time code evaluation, etc.
  - gpderetta 4y ago
    Isn't that's how most LSP servers are implemented? I'm only vaguely familiar with clangd.
    - lillecarl 4y ago
      It's moving in that direction, but far from all are based on their compiler still.
    - ranfdev 4y ago
      Yes. An LSP has to work along with the compiler, or it has to become an alternative compiler. For example, from the article "Why LSP?" [1] "It is known that compilers are complicated, and a language server is a compiler and then some." [1]: https://matklad.github.io//2022/04/25/why-lsp.html https://matklad.github.io//2022/04/25/why-lsp.html
- Shadonototra 4y ago
  from what i understood, that is what they are planning to do, have an official built in language server
Sphax 4y ago
zls doesn't support this unfortunately. I don't know what the plans are for comptime support.

codethief 4y ago

This is really cool, thanks for providing such an in-depth view into Zig's comptime features and how you used them!

ewalk153 4y ago
Agree. This is one of the most instructive tutorials on the comptime language feature.
- Sphax 4y ago
  I didn't really intend to write a tutorial but glad you liked the post !

noodledoodletwo 4y ago

Cool article. I can't figure out why i am supposed to care about zig beyond it's fun? I get that it's interesting and more safe than C (honestly though what the hell isn 't). Say you write rust pretty regularly for new product development, what does zig offer to make my life better, my products more stable, etc?

mongol 4y ago
Yes, but say you are not yet writing Rust, perhaps you should choose Zig instead of it?
- giancarlostoro 4y ago
  I think you're both asking the same question from different angles. If I understand correctly, and I could be misremembering but Zig is supposed to be compatible with C++ not just C, which is something that is not necessarily straight forward in other C / C++ competing languages. I hear even D has some issues with mangling and what not. In all honesty, I prefer the syntax of D over all the others, it feels the most like Java or C# but with a lot of modern benefits. D is trying to do too much though it feels like and I would love for the next D standard library to support OOTB similar to what Go supports, especially a very minimalist web server, I think all modern programming languages should be capable of spinning up web servers out of the box. This is one small detail Go got right in my opinion. Here's an article from the Chromium team on challenges they faced with trying to integrate Rust (or at least evaluating it) note the entry was last updated in 2020: https://www.chromium.org/Home/chromium-security/memory-safety/rust-and-c-interoperability/ https://www.chromium.org/Home/chromium-security/memory-safet...
  - krylon 4y ago
    I like that the web server in Go's standard library comes as a library, so you can embed a web server into your application, even if it otherwise isn't web-related at all, just to allow for some introspection and/or control. (To be fair, I think Python and Ruby have HTTP servers in their standard libraries as well, at least minimalist ones.)

curist 4y ago

By using comptime, the statement couldn't be runtime composed, right? That's currently the major holdback for me to spend more time on zig: if using comptime become more common in zig community, the libs could be less flexible to use. It feels sort of like function coloring to me, that the whole call chain also need to pass down the value as comptime variable. I've only spend 2 days with zig, so I would love to learn if I'm wrong on this subject.

Sphax 4y ago
Only the metadata of the statement is comptime, that is the type annotation for each bind parameters. So if you have this query SELECT * FROM user WHERE age = $age{u16} You _must_ provide a u16 bind parameter. However the value itself is of course not required to be comptime-known, that would make the whole thing unusable. For what it's worth there are in zig-sqlite variants of the method which bypass the comptime checks; they're not documented properly yet but see all methods named `xyzDynamic`, for example https://github.com/vrischmann/zig-sqlite/blob/master/sqlite.zig#L493-L498 https://github.com/vrischmann/zig-sqlite/blob/master/sqlite....
skybrian 4y ago
Often, comptime code couldn’t be executed at runtime because the language features it accesses aren’t available then. But I agree that if a parameter could go either way, you shouldn’t have to write two versions.
throwawaymaths 4y ago
Generally I wouldn't call zig comptime function coloring. (I have written a prime sieve algorithm that uses the runtime code to precalculate some primes at comptime. Yes, I had to be very careful about what was in the prime number algorithm, but comptime supports that level of complexity and it was certainly possible to call runtime-intended code at comptime. Can you call comptime-intended code at runtime? No? (Yes? B/c the call site is "in" the runtime code?) But just make it runtime code instead of comptime code?

SkeuomorphicBee 4y ago

In case anyone else is unfamiliar with Zig syntax and wondering: in Zig the .{ "somevalue"} represents an anonymous list literal [1], and .{.somename = "somevalue"} is an anonymous struct literal [2]. (A bit off topic rant, but Zig documentation is quite bad, it took me a lot more effort that it should to discover the facts above.) [1] https://ziglang.org/documentation/master/#Anonymous-List-Literals https://ziglang.org/documentation/master/#Anonymous-List-Lit... [2] https://ziglang.org/documentation/master/#Anonymous-Struct-Literals https://ziglang.org/documentation/master/#Anonymous-Struct-L...

LAC-Tech 4y ago
Zig is still not 1.0. I can forgive its lack of docs at this point, especially how surprisingly readable the std lib is.
jmull 4y ago
I thought the docs were quite good. I used "ziglearn" to understand ("chapter 1" and "chapter 2" have tons of foundational stuff): https://ziglearn.org/chapter-1/ https://ziglearn.org/chapter-1/ I used the language reference for the details: https://ziglang.org/documentation/master/#Introduction https://ziglang.org/documentation/master/#Introduction I agree the standard library docs aren't useful (unless something changed a lot in the last six months)... it seemed always better to just search and read the standard library source directly.
- messe 4y ago
  > I agree the standard library docs aren't useful (unless something changed a lot in the last six months)... it seemed always better to just search and read the standard library source directly. They're automatically generated. As I understand it they've been holding off on improving them until they have the self hosted compiler working, as it will be easier at that point.
- throwawaymaths 4y ago
  To be fair it's really hard to "search" for tuples and anyonymous structs and get an understanding from "reading the stdlib". It's also enough of a departure from C (but not obviously readable like types on the left) to warrant some sort of big red flag in the lang docs... Understanding this tripped me up a bit

nickysielicki 4y ago

constexpr/consteval/comptime/etc is a game changer for the systems programmer and it feels so close but yet so far. Can someone speak to what language has the best support for this? Some things that I feel are missing in C++: * arbitrary file I/O. I can take my compile time data and write a python script to put it in a std::array, but I shouldn’t have to. * non-fixed-sized containers, ie vector * a generic memoization utility in the standard library for caching. If I want my function to handle any input at runtime, but I want to pre-populate a cache at compile time for values that I know will be called, it’s doable but not as easy as it should be. (In general, given the amount of algorithms that rely on memoization, I’m somewhat surprised that Python is the only language I know that makes memoization as easy as a decorator).

formerly_proven 4y ago
> * arbitrary file I/O. I can take my compile time data and write a python script to put it in a std::array, but I shouldn’t have to. https://ziglang.org/documentation/0.9.1/#embedFile https://ziglang.org/documentation/0.9.1/#embedFile I haven't tried, but because it should just be equivalent to a string literal you should be able to further process it using comptime as well. > * non-fixed-sized containers, ie vector One way is to just have a function count how many entries you need and use arrays. This is all pretty straightforward because comptime. Real dynamic comptime containers are dependent on making a comptime allocator available. There's a ticket for that: https://github.com/ziglang/zig/issues/1291 https://github.com/ziglang/zig/issues/1291 > * a generic memoization utility in the standard library for caching. If I want my function to handle any input at runtime, but I want to pre-populate a cache at compile time for values that I know will be called, it’s doable but not as easy as it should be. (In general, given the amount of algorithms that rely on memoization, I’m somewhat surprised that Python is the only language I know that makes memoization as easy as a decorator). I don't think it's possible to use comptime to generate wrapper functions duplicating the signature of a given function, at least I haven't found a way to do so. The function signature has to be spelled out in the comptime code.
dataangel 4y ago
All of this is already done in Lisp. Greenspun's 10th rule.

voigt 4y ago

I see Zig, I vote up ;) Exciting new language with compelling features! Great work of the author explaining comptime!

deleted 4y ago
[deleted]

jhgb 4y ago

Here's an absolutely crazy idea that I had the other day... Since Zig has comptime, couldn't a Zig reimplementation of SQLite use comptime for compile-time query compilation? Including generating native code for fixed queries? So far I haven't been able to come up with any counterargument for why this wouldn't work.

apavlo 4y ago
> couldn't a Zig reimplementation of SQLite use comptime for compile-time query compilation? Including generating native code for fixed queries? SQLite already compiles queries into opcodes and then uses its own VM to interpret them. You would have to reimplement the internal VM.
- jhgb 4y ago
  I'm aware of that, but these elementary operations have some definitions in code. Even just calling them in a fixed sequence (as opposed to dispatching at runtime one opcode at a time) would be something optimizable for a sufficiently smart compiler. However, given a good-enough design of the whole system, straight code generation from the query plan shouldn't be a problem either (and at that point, the Zig compiler should be able to do even more work with the generated query code). I mean, any reimplementation would presumably use Zig features heavily anyway -- otherwise you could just link with the C code.
pvg 4y ago
The counter-argument is interpreting the query is not what takes up the time in executing a typical RDBMS query. Imagine a db is just a key-value store, the query is some keys and the db just spits out whatever it found in a giant hashtable. That's not going to get faster if you 'compile' the query. It's not going to get much faster if your db is a more realistic three hashtables, two big arrays and a partridge in a b-tree.
- jhgb 4y ago
  It might not be worth for everything, but extremely simple queries on extremely simple schemas would not be the goal here.