3 ms·

What's duckdb Why is it needed over say postgresql or mysql

by noloblo 4y ago

RobinL 4y ago
DuckDB is a no-dependency SQL engine that's available from multiple programming languages. A bit like SQLite but optimised for analytics. For instance, in Python, it can be installed using `pip install duckdb`. For things like unit tests, it can be very valuable because no additional services need to be running.
simonw 4y ago
It's a really interesting new (emerged in the last few years) piece of technology. It's kind of a cross between SQLite and columnar analytical databases such as Snowflake or Cassandra. You can run it happily on a laptop, but because it's column oriented it can calculate aggregates (group by/count queries for example) incredibly fast - so it's better for a lot of analytical workloads than regular row-oriented databases. I have some initial notes from trying it out here: https://til.simonwillison.net/duckdb/parquet https://til.simonwillison.net/duckdb/parquet
- _benedict 4y ago
  Cassandra isn't columnar, it's row-oriented. The original authors coined a new term "column family" which is just a nonsense (and confusing) term, meant to describe its original relatively unstructured API, which has unfortunately stuck in all descriptions of the database, and leads to confusion like this. Cassandra is also not an analytics database, it is intended for "OLTP"-like workloads.
  - simonw 4y ago
    Thanks. I wanted another example other than Snowflake so I did a quick Google search and grabbed the first example from the top search summary box that I recognized.
  - BenoitP 4y ago
    I always considered Cassandra's 'column' designation a guarantee that OLTP transactions will apply atomically to column groups; or that you'll minimize IO writes for a row to the column group (as opposed to writing a new version of the entire row). But this is actually just a wild guess. Anyone want to chime in and confirm/deny?

3 ms·

What's duckdb Why is it needed over say postgresql or mysql

by noloblo 4y ago

What's duckdb Why is it needed over say postgresql or mysql

RobinL 4y ago
DuckDB is a no-dependency SQL engine that's available from multiple programming languages. A bit like SQLite but optimised for analytics. For instance, in Python, it can be installed using `pip install duckdb`. For things like unit tests, it can be very valuable because no additional services need to be running.
simonw 4y ago
It's a really interesting new (emerged in the last few years) piece of technology. It's kind of a cross between SQLite and columnar analytical databases such as Snowflake or Cassandra. You can run it happily on a laptop, but because it's column oriented it can calculate aggregates (group by/count queries for example) incredibly fast - so it's better for a lot of analytical workloads than regular row-oriented databases. I have some initial notes from trying it out here: https://til.simonwillison.net/duckdb/parquet https://til.simonwillison.net/duckdb/parquet
- _benedict 4y ago
  Cassandra isn't columnar, it's row-oriented. The original authors coined a new term "column family" which is just a nonsense (and confusing) term, meant to describe its original relatively unstructured API, which has unfortunately stuck in all descriptions of the database, and leads to confusion like this. Cassandra is also not an analytics database, it is intended for "OLTP"-like workloads.
  - simonw 4y ago
    Thanks. I wanted another example other than Snowflake so I did a quick Google search and grabbed the first example from the top search summary box that I recognized.
  - BenoitP 4y ago
    I always considered Cassandra's 'column' designation a guarantee that OLTP transactions will apply atomically to column groups; or that you'll minimize IO writes for a row to the column group (as opposed to writing a new version of the entire row). But this is actually just a wild guess. Anyone want to chime in and confirm/deny?