4 ms·

Python doesn't already have parsers for all SQL variants. There are several important projects I can't do in my current job because I don't have Hive and Impala

Python doesn't already have parsers for all SQL variants. There are several important projects I can't do in my current job because I don't have Hive and Impala parsers. It would be ideal to have them in Python, but a binary that could output a digestible representation would be huge for me.

dllthomas 9y ago
This project can output JSON for Hive queries, FWIW
- daveFNbuck 9y ago
  Yeah, that's why I mentioned that a binary that can do that would be huge for me.
  - dllthomas 9y ago
    You're saying this is huge? Or you're missing that this can be wrapped into a binary in a handful of lines? :D
    - daveFNbuck 9y ago
      Yes, I'm saying this would be huge for someone who only needed Hive. This would be huge for me if it could also do Impala.
      - dllthomas 9y ago
        Got it! I'd be happy to serve as a resource for anyone taking a swing at adding an Impala front end (or any other, for that matter). I'm unlikely to get to it myself any time soon.
        daveFNbuck 9y ago
        There's something like 5k lines of code for the Hive dialect. How much of that is auto-generated? Is there any way to take advantage of the similarities between Hive and Impala to reduce the work needed?
        dllthomas 9y ago
        It's not auto-generated, but it's not difficult. There's three approaches likely to be reasonable. First, you could make a queryparser-impala package starting with a copy of queryparser-hive, make the changes reflecting the differences you're aware of. Second, you could start from a description of the grammar and build it fresh, using queryparser-hive only as additional reference. Third, if Hive and Impala are close enough (I'm not sufficiently familiar with Impala to say) you could extend queryparser-hive to also work with Impala. In any case, a large corpus of Impala queries would be hugely valuable. IIRC, Heli took the first approach when going from Vertica to Hive; Matt took the second when adding Presto.
        daveFNbuck 9y ago
        Thanks, I'll keep this in mind for when I have time to start a new project.

dllthomas 9y ago

This project can output JSON for Hive queries, FWIW

daveFNbuck 9y ago
Yeah, that's why I mentioned that a binary that can do that would be huge for me.
- dllthomas 9y ago
  You're saying this is huge? Or you're missing that this can be wrapped into a binary in a handful of lines? :D
  - daveFNbuck 9y ago
    Yes, I'm saying this would be huge for someone who only needed Hive. This would be huge for me if it could also do Impala.
    - dllthomas 9y ago
      Got it! I'd be happy to serve as a resource for anyone taking a swing at adding an Impala front end (or any other, for that matter). I'm unlikely to get to it myself any time soon.
      - daveFNbuck 9y ago
        There's something like 5k lines of code for the Hive dialect. How much of that is auto-generated? Is there any way to take advantage of the similarities between Hive and Impala to reduce the work needed?
        dllthomas 9y ago
        It's not auto-generated, but it's not difficult. There's three approaches likely to be reasonable. First, you could make a queryparser-impala package starting with a copy of queryparser-hive, make the changes reflecting the differences you're aware of. Second, you could start from a description of the grammar and build it fresh, using queryparser-hive only as additional reference. Third, if Hive and Impala are close enough (I'm not sufficiently familiar with Impala to say) you could extend queryparser-hive to also work with Impala. In any case, a large corpus of Impala queries would be hugely valuable. IIRC, Heli took the first approach when going from Vertica to Hive; Matt took the second when adding Presto.
        daveFNbuck 9y ago
        Thanks, I'll keep this in mind for when I have time to start a new project.