3 ms·

I'm curious what errors you were running into? Mind posting an issue in the repo, or here? Thanks!

noone_important 2y ago
Sorry for the late answer. I get the following errors if I try to execute queries on delta tables (accessing them with spark works fine) InvalidInputException: Invalid Input Error: Attempting to execute an unsuccessful or closed pending query result Error: IO Error: Hit DeltaKernel FFI error (from: kernel_scan_data_next in DeltaSnapshot GetFile): Hit error: 2 (ArrowError) with message (Invalid argument error: Incorrect datatype for StructArray field "partitionValues", expected Map(Field { name: "entries", data_type: Struct([Field { name: "keys", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "values", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, false) got Map(Field { name: "key_value", data_type: Struct([Field { name: "key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false))
- aleatorisch 2y ago
  Thanks, appreciate the reply. I believe this should have been fixed by https://github.com/delta-incubator/delta-kernel-rs/pull/210 https://github.com/delta-incubator/delta-kernel-rs/pull/210. I'm guessing you were reading from a table with a checkpoint :) DuckDB unfortunately hasn't pulled in the latest changes just yet, but as soon as they update to include that fix, I expect your queries will work. If the table is public, I'd be happy to check that kernel now supports reading it.
noone_important 2y ago
My problem is probably already covered by: https://github.com/duckdb/duckdb_delta/issues/18 https://github.com/duckdb/duckdb_delta/issues/18

Sorry for the late answer. I get the following errors if I try to execute queries on delta tables (accessing them with spark works fine) InvalidInputException: Invalid Input Error: Attempting to execute an unsuccessful or closed pending query result Error: IO Error: Hit DeltaKernel FFI error (from: kernel_scan_data_next in DeltaSnapshot GetFile): Hit error: 2 (ArrowError) with message (Invalid argument error: Incorrect datatype for StructArray field "partitionValues", expected Map(Field { name: "entries", data_type: Struct([Field { name: "keys", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "values", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, false) got Map(Field { name: "key_value", data_type: Struct([Field { name: "key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false))

aleatorisch 2y ago
Thanks, appreciate the reply. I believe this should have been fixed by https://github.com/delta-incubator/delta-kernel-rs/pull/210 https://github.com/delta-incubator/delta-kernel-rs/pull/210. I'm guessing you were reading from a table with a checkpoint :) DuckDB unfortunately hasn't pulled in the latest changes just yet, but as soon as they update to include that fix, I expect your queries will work. If the table is public, I'd be happy to check that kernel now supports reading it.

noone_important 2y ago

My problem is probably already covered by: https://github.com/duckdb/duckdb_delta/issues/18 https://github.com/duckdb/duckdb_delta/issues/18