3 ms·

Don't you lose the in-memory interop with other libraries by doing this? I'm thinking that duckdb will no longer be able to read polars data that has been load

Don't you lose the in-memory interop with other libraries by doing this? I'm thinking that duckdb will no longer be able to read polars data that has been loaded into memory, as it can currently do, due to duckdb supporting Arrow. Isn't the benefit of arrow that it's supported by many languages and libraries as a standard? Will there be an option to use the "compatible" string format?

emidln 2y ago
Wouldnt you just recompile your duckdb with the hacked Arrow?
- suremarc 2y ago
  Arrow is a cross-language standard. Polars is written in Rust but DuckDB is written in C++. So it's not that simple unfortunately.
  - emidln 2y ago
    If you're using Arrow from duckdb, you're using C or C++ bindings. Hack the arrow lib, recompile (and adjust if necessary) whatever bindings/programs using your hacked arrow. If you're using multiple implementations of arrow that arent just bindings, you might have to hack it in more than one place. You don't have to solve the entire world, you only need to hack what you actually use. If the issue with an optimized arrow impl is memory incompat in duckdb, recompile duckdb with your hacked arrow. This is where "overengineered" stuff like mono repos really shine. I've done exactly this type of thing at past jobs. Don't let perfect be the enemy of good enough.
mau 2y ago
Arrow supports this string format: https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout https://arrow.apache.org/docs/format/Columnar.html#variable-... From the article: > As luck would have it, the Arrow spec was also finally making progress with adding the long anticipated German Style string types to the specification. Which, spoiler alert, is the type we implemented.
ritchie46 2y ago
DuckDB has their own string type (quite similar to this) that deviates from Arrow (large)-string type, so it had to do a copy anyway. Nothing has changed on that front.

emidln 2y ago

Wouldnt you just recompile your duckdb with the hacked Arrow?

suremarc 2y ago
Arrow is a cross-language standard. Polars is written in Rust but DuckDB is written in C++. So it's not that simple unfortunately.
- emidln 2y ago
  If you're using Arrow from duckdb, you're using C or C++ bindings. Hack the arrow lib, recompile (and adjust if necessary) whatever bindings/programs using your hacked arrow. If you're using multiple implementations of arrow that arent just bindings, you might have to hack it in more than one place. You don't have to solve the entire world, you only need to hack what you actually use. If the issue with an optimized arrow impl is memory incompat in duckdb, recompile duckdb with your hacked arrow. This is where "overengineered" stuff like mono repos really shine. I've done exactly this type of thing at past jobs. Don't let perfect be the enemy of good enough.

mau 2y ago

Arrow supports this string format: https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout https://arrow.apache.org/docs/format/Columnar.html#variable-... From the article: > As luck would have it, the Arrow spec was also finally making progress with adding the long anticipated German Style string types to the specification. Which, spoiler alert, is the type we implemented.

ritchie46 2y ago

DuckDB has their own string type (quite similar to this) that deviates from Arrow (large)-string type, so it had to do a copy anyway. Nothing has changed on that front.