6 ms·
Introducing pgzx: create PostgreSQL extensions using Zig
- canadiantim 3y agovery cool. More amazing work from xata. Special shoutout to pgroll too!
- michelpp 3y agoThis looks very cool and it's nice to see some finicky details like PG_GETARG abstracted away. Are there any plans to make pgzx a Trusted Language so that Zig functions could be written with CREATE FUNCTION? Rust recently achieved this milestone and AWS RDS now supports PL/Rust. Being a trusted language would mean it could be used to create Trusted Language Extensions (TLEs) and could be installed on cloud hosted databases without filesystem access.
- tudorg 3y agoIt would be amazing if we could could use Zig for TLEs, and we'll look into it, but I worry it might not be possible because Zig can't guarantee memory safety in the same way that Rust can.
- deleted 3y ago[deleted]
- zombodb 3y agoIf you build it, they'll come. The only limit is yourself!
- bear9628 3y agoNo TLE support is a not a goal. Languages/extensions in Postgres should only be marked as trusted if you can guarantee that they do not access Postgres internals or even the file filesystem. With Zig and in pgzx in particular we can not guarantee this to be the case. With pgzx we provide you a number of utilities on top of the C API. But depending on the extension you are working on you might heavily use the Postgres internals and their C APIs as is. Postgres has support for a number of languages. Some having their own strenghts or weaknesses. One strength of Zig is its closeness to C, while providing a many quality of live improvements over C (optional types instead of null pointer, generic data structures, ...). This makes it a formidable choice for extensions that want to integrate deeply with the Postgres internals.
- anarazel 3y agoIf I suddenly had unlimited resources and attention, I'd really like to provide infrastructure to write trusted functions by running them via a webassembly runtime, with a small injected safe surface for accessing postgres functionality. That'd make it a lot easier to write trusted PLs in a variety of languages. Still probably needs a bit of per-language sugar on top, but a lot less than doing it from scratch. Edit: Minor grammar fix
- bear9628 3y agoHehe :) I also have WASM in mind when thinking about future TLE support :)
- michelpp 3y agoI don't know much about it, but I think this has been tried? https://dylibso.com/blog/pg-extism/ https://dylibso.com/blog/pg-extism/ Looking at the repo though it looks like it hasn't been updated in a while.
- nilslice 3y agowe're actually working on a new version of this, which _does_ support TLE! if you're interested, please join https://extism.org/discord https://extism.org/discord and we can get you a preview.
- shay_ker 3y agoI have a dumb, and likely annoying, question: why isn't Rust the clear choice? I think it has to do with Zig's "interop with C" but this is very abstract for me. Is there a code example that shows something that's super easy in Zig but a PITA in Rust? I've heard that Rust's types and compiler can be frustrating to deal with as well. Is Zig a more "joyful" programming language?
- minimaxir 3y agoThere is a separate project to build Postgres extensions with Rust: https://github.com/pgcentralfoundation/pgrx https://github.com/pgcentralfoundation/pgrx
- nextaccountic 3y agoAnd lots of interesting extensions use it, like https://github.com/tembo-io/pgmq https://github.com/tembo-io/pgmq https://github.com/zombodb/zombodb https://github.com/zombodb/zombodb https://github.com/supabase/pg_jsonschema https://github.com/supabase/pg_jsonschema
- giovannibonetti 3y agoI've heard that when working with complex data structures with Rust you often need to write unsafe code, which is a lot more painful to work with in Rust. Once in you are in unsafe land anyway, some people say Zig is more convenient.
- ccleve 3y agoI haven't worked with pgzx, but it's possible that memory management is easier. Zig uses memory arenas, and so does Postgres. If one can map directly to the other, it would be a huge win. With Rust/pgrx (which I have used, extensively) memory integration is more difficult. There's pg memory, and there's Rust memory, and they're not the same, and you have to play games to pass data between them. Lifetime issues crop up. Rust might be able to solve the problem in the future with custom allocators, but it's just not there yet.
- 3y ago
- jedisct1 3y agoPostgreSQL is such an amazing piece of software, that keeps being more awesome release after release. It has almost become a full-featured operating system at that point. And the ability to write extensions in Zig is very nice.
- slt2021 3y agoPostgres creator Stonebraker is working on an actual operating system to run database workloads :) https://en.wikipedia.org/wiki/DBOS https://en.wikipedia.org/wiki/DBOS
- nop_slide 3y agoTangential, first time bumping into Xata. What is the difference between Xata and Neon? Their site says > Xata is the only serverless data platform for PostgreSQL. I thought Neon was serverless postgres too? Edit: This looks really dope actually, going to play around with it later. Really dig the free tier and the built in search from the get go.
- tudorg 3y agoWe're referring more to the "data platform" part. What we mean by a "Postgres data platform" (and arguably it's not an widely accepted term, so take it with a grain of salt) is that you get not only Postgres, but secondary data stores around it, more tightly integrated than in a typical cloud provider: - Elasticsearch, to which we replicate data automatically via CDC - Blob storage for file attachments, we use S3 to store the files and Cloudflare to cache them on CDN - (in the future) Caching - (in the future) Time-series databases, again with automatic replication from Postgres That said, you can use just the Postgres part, and we are very happy if you do.
- nop_slide 3y agoEdited my comment above, but this looks really rad. I made an account and going to try porting a side project DB to it later! Particularly we need to improve our search and I've been lazy about tweaking our ts_vector setup.
- nextaccountic 3y ago> Elasticsearch Do you use zombodb? https://github.com/zombodb/zombodb https://github.com/zombodb/zombodb
- bear9628 3y agoYou can get an overview of the platform in this Post: https://xata.io/blog/serverless-postgres-platform#the-platform https://xata.io/blog/serverless-postgres-platform#the-platfo... Zombodb is a really cool project, but no, we don't use it. We use logical replication and triggers to also capture schema changes with your records into an event stream. The event stream is used to send your data to Elasticsearch (and create/update the index). See: https://xata.io/blog/serverless-postgres-platform#logical-replication-events https://xata.io/blog/serverless-postgres-platform#logical-re... Stay tuned, we are planning to open source this component as well.
- anarazel 3y agoIt's completely besides the point, but part of the complexity of the char_count example is just due to the C version for some reason copying the input strings into newly allocated memory, which the zig version doesn't :)
- tudorg 3y agoThis is fair, and it's a bit embarrassing but I copied the C version from an older tutorial without looking at it critical enough. Still, the Zig version gets the arguments as normal function parameters and doesn't need to call `PG_GETARG_*`.
- e12e 3y agoI notice that the c version looks like it uses memcopy as a convoluted typecast from a pointer to text (presumably a postgres typedef?) to a pointer to char (null terminated?)? Could this actually have a purpose? Ed: I don't think PG uses c strings? Looks like memcopy might be needed for call-by-reference, return-by-value - but not really in the simple function that only walks the array? https://www.postgresql.org/docs/current/xfunc-c.html#XFUNC-C-BASETYPE https://www.postgresql.org/docs/current/xfunc-c.html#XFUNC-C...
- anarazel 3y agoI wonder what things you'd like to see exposed by postgres to make something like this easier. E.g. you have code to maintain the module magic and function info, for both of which there are some plans to change the contents in the next few major versions. If we were to provide zig consumable ways of creating the struct from C headers, would that help?
- bear9628 3y agoOne of the reasons why we are using Zig is that we can consume the C API mostly as is. When importing the C headers Zig translates the C headers into Zig declarations. Unfortunately this is not always possible for C macros, and the reason why we have to maintain those structs. And this is where we have to step in with Custom Zig code. But most of the time we actually consume the C APIs as is. > If we were to provide zig consumable ways of creating the struct from C headers, would that help? Yes, that would be awesome. I'm curious how they will look like in the future. We have had the most difficulties with the module magic, function info, and varatt/Datum macros. Fortunately you have to solve the module magic/function info "only once". The Datum conversion and VARATT macros are more troublesome. We have some conversion support for a number of common zig types. But ideally we would like users to be able to use the C APIs as is, while we provide some type directed default conversions for convenience. The main problems we've been facing with C macro translations are type conversions/casts in macros, especially if the underyling struct heavily uses union (for example VARATT macros). In some cases translating inline functions instead of macros might work better, due to the translator having more type information available. We fixed some of the translations manually. You can find them in varatt.zig and datum.zig (where we opted to implement the text to cstring translation ourselves). Data structures like lists, slist, dlist, hash tables are quite consumable as is. We have some typed wrappers for those and provide iterators. Macros with control flow can not be reused, but I think this is fair, especially as the foreach macros are a very common C patterns. All in all we have had no troubles with them.
- anarazel 3y agoHi, > > If we were to provide zig consumable ways of creating the struct from C headers, would that help? > Yes, that would be awesome. I don't know much about zig and won't have a whole lot of time to learn - but if you can outline what is required to make C macros [un]usable, it might be possible to improve something. Either on its own, or as part of future work. > I'm curious how they will look like in the future. There's quite a few things. For one, I'd like to introduce a faster function calling infrastructure for the simple cases (mainly small number of arguments, without SRF support). That'll need to be declared in the function info struct. For another, eventually I want to support a different encoding for variable length types. Including making it reasonably efficient to have variable-length integers. > We have had the most difficulties with the module magic, function info, and varatt/Datum macros. Fortunately you have to solve the module magic/function info "only once". The Datum conversion and VARATT macros are more troublesome. We have some conversion support for a number of common zig types. But ideally we would like users to be able to use the C APIs as is, while we provide some type directed default conversions for convenience. Ugh, the varatt stuff doesn't look easily maintainable long-term. It looks like you just need it for getDatumTextSliceZ()? At the same time, I don't really know why you need it? Most of this should be doable via C functions, and the parts that are not, you could easily wrap yourself - you already seem to have some C code as part of pgzx.
- ww520 3y agoThis is a fantastic news. I got into Zig in the last two weeks and it has be a very pleasant language. The comptime is an ingenious invention. It enables type safe meta-programming and easy generic support, in the same language syntax. The native support for sub-byte types like u2, u3, or u7 make packing data very easy. The native vector support on SIMD makes SIMD level parallel programming like child play. The language is looking very good.