5 ms·

I am not 100% clear how this works... If you query a Parquet file from your lake via DuckDB-in-browser, does DuckDB run in WASM on the web client and pull the

by gregw2 2y ago

I am not 100% clear how this works... If you query a Parquet file from your lake via DuckDB-in-browser, does DuckDB run in WASM on the web client and pull the compressed parquet to your browser where it is decompressed? Or are you connecting some DuckDB on the web client to some DuckDB component on a server somewhere? I presume yes to the first and no to the second but just checking I have my mental model correct.

4b11b4 2y ago
The data would be sent to the browser
- throwaway-blaze 2y ago
  Um, my data lake is measured in Pb, not Tb. How is that going to work exactly?
  - undefuser 2y ago
    DuckDB has certain optimizations which allows it to read only parts of a parquet file. It can also read remote file in streaming fashion so it does not have to wait for the entire file to be downloaded or to store a large amount of data in memory. Relevant documentation: https://duckdb.org/2021/06/25/querying-parquet.html https://duckdb.org/2021/06/25/querying-parquet.html
  - 1egg0myegg0 2y ago
    Howdy! I work at MotherDuck and DuckDB Labs (part time as a blogger). At MotherDuck, we have both client side and server side compute! So the initial reduction from PB/TB to GB/MB can happen server side, and the results can be sliced and diced at top speed in your browser!
    - code_biologist 2y ago
      Please spend a sentence or two explaining the server side filtering mechanism and linking to documentation! I would like to know the conditions required for streaming queries! From the sibling comment and a search of the docs it seems like this is a Parquet only feature, which seems pretty important to note!
      - justincormack 2y ago
        Not specific to Ducks but S3 select https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/select... can filter Parquet server side on S3 and is supported by some other object stores.
      - FridgeSeal 2y ago
        Parquet is designed with predicate push-down in mind. Partitions are laid out on disk, and then blocks within files are laid out so that consumers can very, very easily narrow in on which files they need to read, before doing anymore IO than a list, or a small metadata read. Once you know what you are reading, many parquet/arrow libraries will support streaming reads/aggregations, so the client doesn’t need to load the whole working set in memory.
        LunaSea 2y ago
        This only covers very simple min / max / sum cases. For all others you'll need to download all columns you are filtering or selecting.
    - victor106 2y ago
      Does duckdb work with delta files?
  - simonw 2y ago
    DuckDB can use the HTTP range header trick agains parquet, which means a lot of analytical questions against multi-GB files can be answered by fetching only small portions of the overall data. Here's a post about applying this same trick to SQLite from few years ago: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database...
zX41ZdbW 2y ago
I have an example of a database hosted on GitHub: https://github.com/ClickHouse/web-tables-demo https://github.com/ClickHouse/web-tables-demo Which you can query anywhere using a ClickHouse local database engine. When you do a query, it downloads only the required ranges of required columns from the server and does computations on your machine. In the same way you can query any data sources with ClickHouse, either local or remote, which is a strict superset in power and usability than duckdb.

by gregw2 2y ago

4b11b4 2y ago

The data would be sent to the browser

throwaway-blaze 2y ago
Um, my data lake is measured in Pb, not Tb. How is that going to work exactly?
- undefuser 2y ago
  DuckDB has certain optimizations which allows it to read only parts of a parquet file. It can also read remote file in streaming fashion so it does not have to wait for the entire file to be downloaded or to store a large amount of data in memory. Relevant documentation: https://duckdb.org/2021/06/25/querying-parquet.html https://duckdb.org/2021/06/25/querying-parquet.html
- 1egg0myegg0 2y ago
  Howdy! I work at MotherDuck and DuckDB Labs (part time as a blogger). At MotherDuck, we have both client side and server side compute! So the initial reduction from PB/TB to GB/MB can happen server side, and the results can be sliced and diced at top speed in your browser!
  - code_biologist 2y ago
    Please spend a sentence or two explaining the server side filtering mechanism and linking to documentation! I would like to know the conditions required for streaming queries! From the sibling comment and a search of the docs it seems like this is a Parquet only feature, which seems pretty important to note!
    - justincormack 2y ago
      Not specific to Ducks but S3 select https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/select... can filter Parquet server side on S3 and is supported by some other object stores.
    - FridgeSeal 2y ago
      Parquet is designed with predicate push-down in mind. Partitions are laid out on disk, and then blocks within files are laid out so that consumers can very, very easily narrow in on which files they need to read, before doing anymore IO than a list, or a small metadata read. Once you know what you are reading, many parquet/arrow libraries will support streaming reads/aggregations, so the client doesn’t need to load the whole working set in memory.
      - LunaSea 2y ago
        This only covers very simple min / max / sum cases. For all others you'll need to download all columns you are filtering or selecting.
  - victor106 2y ago
    Does duckdb work with delta files?
- simonw 2y ago
  DuckDB can use the HTTP range header trick agains parquet, which means a lot of analytical questions against multi-GB files can be answered by fetching only small portions of the overall data. Here's a post about applying this same trick to SQLite from few years ago: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database...

zX41ZdbW 2y ago

I have an example of a database hosted on GitHub: https://github.com/ClickHouse/web-tables-demo https://github.com/ClickHouse/web-tables-demo Which you can query anywhere using a ClickHouse local database engine. When you do a query, it downloads only the required ranges of required columns from the server and does computations on your machine. In the same way you can query any data sources with ClickHouse, either local or remote, which is a strict superset in power and usability than duckdb.