3 ms·

Excited to see this - and excellent use case for libpg_query (I'm the original author and still help maintain it together with the rest of the team) and appreci

by lfittl 3y ago

Excited to see this - and excellent use case for libpg_query (I'm the original author and still help maintain it together with the rest of the team) and appreciate the shout out to pganalyze! If anyone else has a use case for using the Postgres parser outside the server, we have a healthy ecosystem of libraries that build on the core C library (we maintain bindings for Ruby, Go and Rust ourselves), as well as various projects using it (e.g. sqlc uses it for a type-safe way for using hand-written SQL in Go): https://github.com/pganalyze/libpg_query#resources https://github.com/pganalyze/libpg_query#resources

derefr 3y ago
With all the various systems making use of Postgres's parser, is there a reason that the Postgres team haven't "seen the writing on the wall" about the demand for their parser, and so extracted their parser into a standalone library? I imagine that Postgres itself would then consume the parser library as a — possibly regularly-snapshot-vendored — static library dependency; while other applications and wrapper libraries would be free to consume it as a dynamic shared library; and it could be freely independently distro-packaged; and so forth. In other words, it would become a de-facto "libxml for SQL" — the kind of venerable C lib that you expect everything else to just be a wrapper around. Is it just that doing this would involve making functions with internal linkage into functions with external linkage, and thereby preventing some WPO opportunities when compiling Postgres itself?
- lfittl 3y ago
  Generally I agree that this would be great to have, and Postgres does have a set of libraries it already maintains as part of the main source tree (i.e. libpq, etc), and there is a shared set of code between the backend and the "frontend" (https://github.com/postgres/postgres/tree/master/src/common https://github.com/postgres/postgres/tree/master/src/common). So theoretically you could imagine the parser moving into that shared code portion, sharing code but not necessarily requiring linking to a library from the backend. However, the challenge from what I've understood from past conversations with some folks working on Postgres core is that the parser is currently heavily tied into the backend - note the parser isn't just the scan.l/gram.y file, but also the raw parse node structs that it outputs. You can see how many files we pull in from the main tree that are prefixed with "src_backend": https://github.com/pganalyze/libpg_query/tree/15-latest/src/postgres https://github.com/pganalyze/libpg_query/tree/15-latest/src/... Further, there isn't a canonical way to output node trees into a text format today in core, besides the rather hard to work with output of debug_print_parse - there have been discussions on -hackers to potentially utilize JSON here, which may make this a bit easier. Note that in libpg_query we currently use Protobuf (but used to use JSON), which does have the benefit of getting auto-generated structs in the language bindings - but Protobuf is not used in core Postgres at all today. All in all, I think there is some upstream interest, but its not clear that this is a good idea from a maintainability perspective.

by lfittl 3y ago

derefr 3y ago

With all the various systems making use of Postgres's parser, is there a reason that the Postgres team haven't "seen the writing on the wall" about the demand for their parser, and so extracted their parser into a standalone library? I imagine that Postgres itself would then consume the parser library as a — possibly regularly-snapshot-vendored — static library dependency; while other applications and wrapper libraries would be free to consume it as a dynamic shared library; and it could be freely independently distro-packaged; and so forth. In other words, it would become a de-facto "libxml for SQL" — the kind of venerable C lib that you expect everything else to just be a wrapper around. Is it just that doing this would involve making functions with internal linkage into functions with external linkage, and thereby preventing some WPO opportunities when compiling Postgres itself?

lfittl 3y ago
Generally I agree that this would be great to have, and Postgres does have a set of libraries it already maintains as part of the main source tree (i.e. libpq, etc), and there is a shared set of code between the backend and the "frontend" (https://github.com/postgres/postgres/tree/master/src/common https://github.com/postgres/postgres/tree/master/src/common). So theoretically you could imagine the parser moving into that shared code portion, sharing code but not necessarily requiring linking to a library from the backend. However, the challenge from what I've understood from past conversations with some folks working on Postgres core is that the parser is currently heavily tied into the backend - note the parser isn't just the scan.l/gram.y file, but also the raw parse node structs that it outputs. You can see how many files we pull in from the main tree that are prefixed with "src_backend": https://github.com/pganalyze/libpg_query/tree/15-latest/src/postgres https://github.com/pganalyze/libpg_query/tree/15-latest/src/... Further, there isn't a canonical way to output node trees into a text format today in core, besides the rather hard to work with output of debug_print_parse - there have been discussions on -hackers to potentially utilize JSON here, which may make this a bit easier. Note that in libpg_query we currently use Protobuf (but used to use JSON), which does have the benefit of getting auto-generated structs in the language bindings - but Protobuf is not used in core Postgres at all today. All in all, I think there is some upstream interest, but its not clear that this is a good idea from a maintainability perspective.