5 ms·

One of the authors here - portability of the storage is indeed one of our goals. We test that the same file can be loaded everywhere on different systems, inclu

One of the authors here - portability of the storage is indeed one of our goals. We test that the same file can be loaded everywhere on different systems, including between ARM and x86. I should mention the storage is still very much a work-in-progress. We are actively working on redesigning the storage to add compression and other extra features, meaning the storage format can change and be incompatible between different DuckDB versions. We plan to stabilize the storage format with V1.0, after which every subsequent DuckDB version should be able to read database files created by previous versions.

Abishek_Muthian 6y ago
Thanks for the confirmation on portability, I didn't see Go support in the home page; Is there an official Go driver support planned as well?
- mytherin 6y ago
  Neither Hannes nor me are very experienced with Go, so we are not the most suitable to do that. We do very much welcome external contributions, and someone did make a WIP go driver [1] in the past. We also have a sqlite3 compatible C API that can be used to use DuckDB. This is actually how we use the sqlite3 shell: we directly use the sqlite shell but bind it to the duckdb library using this API. It might be possible to use the Go SQLite driver in a similar manner to talk to DuckDB. [1] https://github.com/marcboeker/go-duckdb https://github.com/marcboeker/go-duckdb
- sgt 6y ago
  And Dart bindings? That could be really interesting for Flutter developers.
marton78 6y ago
Couldn't you have used an already existing format for storage, e.g. Apache ORC?
- hfmuehleisen 6y ago
  Another DuckDB developer here, we support reading table data from Parquet files directly.
- mytherin 6y ago
  Those formats have a different use case from a database system - they are mostly designed for write once, read many times workloads. As far as I’m aware you cannot update them in an ACID compliant manner, for example, which makes it difficult to use them as backend for an ACID database system.
- StreamBright 6y ago
  I am a big fan of those formats but decoupling the actual storage features from the ecosystem is not a trivial task. I haven't look at the C++ version of ORC for a while but it used to be incomplete. Other than that, the solutions ORC uses to compress data is pretty amazing.
StreamBright 6y ago
Do you consider (or maybe you already have) the advanced columnar features? Run length encoding, dictionary encoding, etc. It would be great to see how those perform under these workloads.
- mytherin 6y ago
  Definitely. We have quite detailed plans for compression :) more to follow when we get to it.
  - StreamBright 6y ago
    Excellent! Thanks for the update.
snthpy 6y ago
DuckDB looks very interesting and I'm quite excited to examine it more closely in the next few days! I just wanted to add to the discussion that an unchanging file format, or at least a backwards compatible one, is a key feature of sqlite. See for example Richard Hipp's comments here [1] (I think he also mentioned earlier in the talk that the file format has become a limiting factor now in terms of some of the refactoring that they can do). The file format therefore seems likely to be a major factor in the long term success of this project and I am glad to see that you are taking your time before settling on any architecture here. Given that you are targeting the data science and analytics space, what are your plans for integration with arrow and the feather file format? From a purely user/developer perspective, arrow's aim of shared memory data structures across different analytics tools, seems like a great goal. I know Wes McKinney and Ursa Labs have also spent quite some time at the file storage part of this, see for example the Feather V2 announcement [2]. What are your thoughts on the tradeoffs they considered and how do you see the requirements of DuckDB in relation to theirs? From the Carnegie Mellon DuckDB talk [3], I saw that you already have a zero-copy reader to the pandas memory data structures, so the vision I have is that DuckDB could be the universal SQL interface to arrow datasets which can then also be shared with more complex ML models. Is that something that we can hope for or are there obstacles to this? [1] https://youtu.be/Jib2AmRb_rk?t=3150 https://youtu.be/Jib2AmRb_rk?t=3150 [2] https://ursalabs.org/blog/2020-feather-v2/ https://ursalabs.org/blog/2020-feather-v2/ [3] https://www.youtube.com/watch?v=PFUZlNQIndo https://www.youtube.com/watch?v=PFUZlNQIndo
- mytherin 6y ago
  Thanks for your detailed reply! As you mentioned - lessons learned by SQLite here are indeed crucial. We are very carefully trying to craft a storage format before fixing it in-place, specifically to try to avoid these problems. Backwards compatibility is a must, but backwards compatibility to a sane format is massively preferable :) No doubt we will end up making some mistakes in hindsight, though. We already support reading from the arrow in-memory format [1], therefore it is already possible to use DuckDB to perform SQL-on-Arrow. More support there is definitely possible, and I think this is a very promising direction. [1] https://github.com/cwida/duckdb/pull/866 https://github.com/cwida/duckdb/pull/866

Abishek_Muthian 6y ago

Thanks for the confirmation on portability, I didn't see Go support in the home page; Is there an official Go driver support planned as well?

mytherin 6y ago
Neither Hannes nor me are very experienced with Go, so we are not the most suitable to do that. We do very much welcome external contributions, and someone did make a WIP go driver [1] in the past. We also have a sqlite3 compatible C API that can be used to use DuckDB. This is actually how we use the sqlite3 shell: we directly use the sqlite shell but bind it to the duckdb library using this API. It might be possible to use the Go SQLite driver in a similar manner to talk to DuckDB. [1] https://github.com/marcboeker/go-duckdb https://github.com/marcboeker/go-duckdb
sgt 6y ago
And Dart bindings? That could be really interesting for Flutter developers.

marton78 6y ago

Couldn't you have used an already existing format for storage, e.g. Apache ORC?

hfmuehleisen 6y ago
Another DuckDB developer here, we support reading table data from Parquet files directly.
mytherin 6y ago
Those formats have a different use case from a database system - they are mostly designed for write once, read many times workloads. As far as I’m aware you cannot update them in an ACID compliant manner, for example, which makes it difficult to use them as backend for an ACID database system.
StreamBright 6y ago
I am a big fan of those formats but decoupling the actual storage features from the ecosystem is not a trivial task. I haven't look at the C++ version of ORC for a while but it used to be incomplete. Other than that, the solutions ORC uses to compress data is pretty amazing.

StreamBright 6y ago

Do you consider (or maybe you already have) the advanced columnar features? Run length encoding, dictionary encoding, etc. It would be great to see how those perform under these workloads.

mytherin 6y ago
Definitely. We have quite detailed plans for compression :) more to follow when we get to it.
- StreamBright 6y ago
  Excellent! Thanks for the update.

snthpy 6y ago

DuckDB looks very interesting and I'm quite excited to examine it more closely in the next few days! I just wanted to add to the discussion that an unchanging file format, or at least a backwards compatible one, is a key feature of sqlite. See for example Richard Hipp's comments here [1] (I think he also mentioned earlier in the talk that the file format has become a limiting factor now in terms of some of the refactoring that they can do). The file format therefore seems likely to be a major factor in the long term success of this project and I am glad to see that you are taking your time before settling on any architecture here. Given that you are targeting the data science and analytics space, what are your plans for integration with arrow and the feather file format? From a purely user/developer perspective, arrow's aim of shared memory data structures across different analytics tools, seems like a great goal. I know Wes McKinney and Ursa Labs have also spent quite some time at the file storage part of this, see for example the Feather V2 announcement [2]. What are your thoughts on the tradeoffs they considered and how do you see the requirements of DuckDB in relation to theirs? From the Carnegie Mellon DuckDB talk [3], I saw that you already have a zero-copy reader to the pandas memory data structures, so the vision I have is that DuckDB could be the universal SQL interface to arrow datasets which can then also be shared with more complex ML models. Is that something that we can hope for or are there obstacles to this? [1] https://youtu.be/Jib2AmRb_rk?t=3150 https://youtu.be/Jib2AmRb_rk?t=3150 [2] https://ursalabs.org/blog/2020-feather-v2/ https://ursalabs.org/blog/2020-feather-v2/ [3] https://www.youtube.com/watch?v=PFUZlNQIndo https://www.youtube.com/watch?v=PFUZlNQIndo

mytherin 6y ago
Thanks for your detailed reply! As you mentioned - lessons learned by SQLite here are indeed crucial. We are very carefully trying to craft a storage format before fixing it in-place, specifically to try to avoid these problems. Backwards compatibility is a must, but backwards compatibility to a sane format is massively preferable :) No doubt we will end up making some mistakes in hindsight, though. We already support reading from the arrow in-memory format [1], therefore it is already possible to use DuckDB to perform SQL-on-Arrow. More support there is definitely possible, and I think this is a very promising direction. [1] https://github.com/cwida/duckdb/pull/866 https://github.com/cwida/duckdb/pull/866