4 ms·
It's a pity that the thing has to get reimplemented all over so many times. For example, you'd think you could reuse PostgreSQL's parser, but its code is autoge
by d33 9y ago
It's a pity that the thing has to get reimplemented all over so many times. For example, you'd think you could reuse PostgreSQL's parser, but its code is autogenerated and riddled with side effects:
https://github.com/postgres/postgres/blob/master/src/backend/parser/gram.y https://github.com/postgres/postgres/blob/master/src/backend...
We've a lot of got languages and usually their parsers are tightly coupled, basically forcing people to reimplement them if you want a flexible parser. I'd say it would be great if we had one parsing library for each language, ideally used at its core. I think that would greatly simplify implementing linters, static analysers and maybe even tools that generate code. Also, perhaps this would enable even more cooler hacks?
- sixdimensional 9y agoI agree, I wish every vendor would publish an official open source parser for their SQL dialects, or that the community would create a repository of standard parsers or grammars for them all. We are almost there with ANTLR grammars... https://github.com/antlr/grammars-v4 https://github.com/antlr/grammars-v4
- dllthomas 9y agoThe trickiest bit isn't parsing per-se, but disambiguating references, so grammar alone isn't enough. And unfortunately, disambiguating references doesn't work the same way everywhere - for instance, tables on Hive will shadow a CTE.
- manigandham 9y agoDefinitely agreed on the amount of duplicate effort. Apache Calcite seems to be a good solution but usually gets ignored: https://calcite.apache.org https://calcite.apache.org Interestingly, Uber does use Calcite themselves for AthenaX: https://github.com/uber/AthenaX/ https://github.com/uber/AthenaX/
- dllthomas 9y agoWhen I looked at Calcite, the branding was very focused on query optimization, which was sufficiently unrelated to what I was doing that I didn't dig in. Had I, it might've been the starting point instead of my personal project, although at the time it didn't have Vertica support (I think that was ultimately added by a different group at Uber).
- manigandham 9y agoYes, it's not marketed much, but then again it's a very specialized project that isn't relevant for many people. It is pretty good though and used by plenty of other Apache projects like Apex, Drill, Flink, Hive, Storm, etc: https://calcite.apache.org/docs/powered_by.html https://calcite.apache.org/docs/powered_by.html
- lfittl 9y agoJust in the off-chance this is useful to someone: If you find yourself needing the Postgres parser elsewhere, e.g. in some custom tooling to analyze SQL queries, I extracted it into a library some time ago: https://github.com/lfittl/libpg_query https://github.com/lfittl/libpg_query Your comment still applies though, as a lot of the parser output is very specific to the chosen Postgres version (10 at the moment) and can change unpredictably.
- teej 9y agoI am in the middle of a project similar to what uber is doing and it leans very heavily on this library. Thank you so much for this! I would not have been able to pull it off without libpg_query.
- arnon 9y agoThere's also HSSQLPPP for Postgres, and it's written in Haskell https://github.com/JakeWheat/hssqlppp https://github.com/JakeWheat/hssqlppp
- gilmi 9y agoOr simple-sql-parser by the same author which is just a parser. https://github.com/JakeWheat/simple-sql-parser https://github.com/JakeWheat/simple-sql-parser