4 ms·

I think that talking about Ion, without talking about PartiQL, is not setting people up with proper context. PartiQL is AWS's specification for a parser/query

I think that talking about Ion, without talking about PartiQL, is not setting people up with proper context. PartiQL is AWS's specification for a parser/query language that is compatible with standard SQL, but can query semi-structured or unstructured data (think JSON, Parquet, CSV/TSV etc) https://aws.amazon.com/blogs/opensource/announcing-partiql-one-query-language-for-all-your-data/ https://aws.amazon.com/blogs/opensource/announcing-partiql-o... PartiQL uses Ion as it's backbone and data format: https://partiql.org/faqs.html#why-do-you-choose-ion-to-extend-SQLs-type-system-over https://partiql.org/faqs.html#why-do-you-choose-ion-to-exten... https://github.com/partiql/partiql-lang-kotlin/blob/master/examples/src/kotlin/org/partiql/examples/ParserExample.kt#L17-L18 https://github.com/partiql/partiql-lang-kotlin/blob/master/e... I looked pretty deeply into this, but failed a bit short of understanding what they meant when "if your query engine supports PartiQL." Does that mean writing a new DB that delegates incoming queries to PartiQL? Not sure. Anyways, they use it in Quantum Ledger DB, and a few other internal projects: https://docs.aws.amazon.com/qldb/latest/developerguide/ql-reference.query.html https://docs.aws.amazon.com/qldb/latest/developerguide/ql-re... So maybe that can give some more context around "what the hell is this, why does it exist, how would you use it?"

whoevercares 6y ago
Yeah It’s more like a spec, they expect you to write the query optimizer, analyzer basically a new query engine. They offered a document based “reference” implementation on Kotlin though so I think expert could follow it. Point is AWS might not make up its mind yet on whether this does more harm to their DB business or not
- gavinray 6y ago
  Ahh. I guess for SQL you could just pass the queries through, and for NoSQL engines try to convert the AST format. When I was researching this area, it seems like Apache Calcite is the way to go, and already does this though? It lets you use standard SQL and has adapters which translate from the abstract query AST to the particular implementation. https://calcite.apache.org/docs/adapter.html https://calcite.apache.org/docs/adapter.html You can query anything with this, exactly the same way. It's kind of wild I've never heard it talked about tbh. I went looking for solutions to multi-datastore querying when I fiddled with a business intelligence side project. Pretty useless to only be able to query one type of data, and too time consuming to implement individual mappings. Apache Calcite and Metabase Query Language (quite the exact usecase there for Metabase, haha) were the only things I could find.
svnpenn 6y ago
That sounds great, but is Kotlin really the only current implementation?
- gavinray 6y ago
  If you want a working implementation (see comment below) and not a reference for the parser, you could use Apache Calcite: https://calcite.apache.org/ https://calcite.apache.org/ But again, yeah it's a JVM thing so your options are that.
  - nautilus12 6y ago
    It amazes how everyone likes to hate on JVM and yet nearly every enterprise company uses it in the big data space....
    - dragonwriter 6y ago
      Everyone likes to hate on COBOL, and plenty of enterprises use it in all kinds of spaces, too.
      - ChiranSoogoor 6y ago
        The communication gap between Businesses using COBOL and Academic institutions teaching IT is the reason for the hatred for the language among young computer graduates. Whether you love or not you live with and support your family, so teach the youngsters to love COBOL, for their grandmother is not going to die any sooner!
ta1234567890 6y ago
Given Postgres already supports document types, do you know what is the main advantage of using AWS' instead?
- nine_k 6y ago
  Does Postgres support column-oriented formats, like Parquet?
  - dragonwriter 6y ago
    > Does Postgres support column-oriented formats, like Parquet? With an appropriate FDW, sure, and I'm pretty sure I've seen an FDW for parquet specifically, as well as other columnar formats.
- nautilus12 6y ago
  To give Amazon more money. Oh you mean advantage to you? haha...well..
revel 6y ago
ion precedes PartiQL by many years, maybe even a decade. The sole reason ion exists is to make parsing json faster and less ambiguous so that a few specific edge cases are handled efficiently. So far so good, right? The problem is that it spreads, like an infection, to surrounding services. Inside Amazon there are literally hundreds of libraries that duplicate standard json libraries in various languages but support ion instead of json. All of this is just to deal with interoperability. Ion is slower than protobufs and less universally understood than json. Honestly it's just an annoyance.

whoevercares 6y ago

Yeah It’s more like a spec, they expect you to write the query optimizer, analyzer basically a new query engine. They offered a document based “reference” implementation on Kotlin though so I think expert could follow it. Point is AWS might not make up its mind yet on whether this does more harm to their DB business or not

gavinray 6y ago
Ahh. I guess for SQL you could just pass the queries through, and for NoSQL engines try to convert the AST format. When I was researching this area, it seems like Apache Calcite is the way to go, and already does this though? It lets you use standard SQL and has adapters which translate from the abstract query AST to the particular implementation. https://calcite.apache.org/docs/adapter.html https://calcite.apache.org/docs/adapter.html You can query anything with this, exactly the same way. It's kind of wild I've never heard it talked about tbh. I went looking for solutions to multi-datastore querying when I fiddled with a business intelligence side project. Pretty useless to only be able to query one type of data, and too time consuming to implement individual mappings. Apache Calcite and Metabase Query Language (quite the exact usecase there for Metabase, haha) were the only things I could find.

svnpenn 6y ago

That sounds great, but is Kotlin really the only current implementation?

gavinray 6y ago
If you want a working implementation (see comment below) and not a reference for the parser, you could use Apache Calcite: https://calcite.apache.org/ https://calcite.apache.org/ But again, yeah it's a JVM thing so your options are that.
- nautilus12 6y ago
  It amazes how everyone likes to hate on JVM and yet nearly every enterprise company uses it in the big data space....
  - dragonwriter 6y ago
    Everyone likes to hate on COBOL, and plenty of enterprises use it in all kinds of spaces, too.
    - ChiranSoogoor 6y ago
      The communication gap between Businesses using COBOL and Academic institutions teaching IT is the reason for the hatred for the language among young computer graduates. Whether you love or not you live with and support your family, so teach the youngsters to love COBOL, for their grandmother is not going to die any sooner!

ta1234567890 6y ago

Given Postgres already supports document types, do you know what is the main advantage of using AWS' instead?

nine_k 6y ago
Does Postgres support column-oriented formats, like Parquet?
- dragonwriter 6y ago
  > Does Postgres support column-oriented formats, like Parquet? With an appropriate FDW, sure, and I'm pretty sure I've seen an FDW for parquet specifically, as well as other columnar formats.
nautilus12 6y ago
To give Amazon more money. Oh you mean advantage to you? haha...well..

revel 6y ago

ion precedes PartiQL by many years, maybe even a decade. The sole reason ion exists is to make parsing json faster and less ambiguous so that a few specific edge cases are handled efficiently. So far so good, right? The problem is that it spreads, like an infection, to surrounding services. Inside Amazon there are literally hundreds of libraries that duplicate standard json libraries in various languages but support ion instead of json. All of this is just to deal with interoperability. Ion is slower than protobufs and less universally understood than json. Honestly it's just an annoyance.