1egg0myegg0

8 ms·

31.
by 1egg0myegg0 4y ago
DuckDB 0.5.0 "Pulchellus" released with persistent indexes, out-of-core hash join, new join order optimizer, and many other improvements!
32.
by 1egg0myegg0 4y ago
DuckDB can actually read SQLite or Postgres directly! In the SQLite case, something like Litestream plus DuckDB could work really well! Also, with Pyarrow's help, DuckDB can already do this with Delta tables! https://github.
33.
by 1egg0myegg0 4y ago
It's likely out of memory issues with many many distinct values in an aggregated column, or with a very large join intermediate. The joins are now able to handle larger than RAM, and larger than RAM intermediates in a group by is on th
34.
by 1egg0myegg0 4y ago
Hmm, I would recommend reading a sample of the code first!
35.
by 1egg0myegg0 4y ago
Short answer, no. Long answer, also no. Blockchain is inherently an inefficient way to compute. You should just trust a centralized authority and get waaaaay more utility for your compute. Your centralized authority could be the very cause
36.
by 1egg0myegg0 4y ago
Good question! As a disclaimer, I work for DuckDB Labs. There are 2 big benefits to working with Parquet files in DuckDB, and both relate to speed! DuckDB can query parquet right where it sits, so there is no need to insert it into the db f
37.
by 1egg0myegg0 4y ago
We have a node client if that would be helpful! https://duckdb.org/docs/api/nodejs
38.
by 1egg0myegg0 4y ago
Interesting thought! I have not tried this yet so I only have a guess as an answer. Could you export the data as SQL statements and then run those statements on DuckDB? That may be easier to set up, but may take longer to run... DuckDB also
39.
by 1egg0myegg0 4y ago
For compatibility, the original column in the data source takes precedence. That's how other DBs handle things so we wanted to stay standard where it made sense!
40.
by 1egg0myegg0 4y ago
If you'd like to work with DuckDB in a SQL IDE/GUI, we recommend DBeaver! It uses the DuckDB JDBC connector. A quick how to guide is here: https://duckdb.org/docs/guides/sql_editors/dbeaver
41.
by 1egg0myegg0 4y ago
Good catch! Fixed! Needed a group by all in there.
42.
by 1egg0myegg0 4y ago
It is a goal of ours to become a standard multi-table storage format! However, today we are still in beta and have made some breaking changes in the last few releases. (Exporting from the old version, then reimporting your DB in the new all
43.
by 1egg0myegg0 4y ago
I should add that we can read/write Apache Arrow as well!
44.
by 1egg0myegg0 4y ago
Here is a solved Github Issue related to CGO for the Go bindings! If you have another issue, please feel free to post it on their Github page! https://github.com/marcboeker/go-duckdb/issues/4
45.
by 1egg0myegg0 4y ago
Thanks! Can you tell that my SQL-fu is stronger than my HTML-fu? :-) Much appreciated!
46.
by 1egg0myegg0 4y ago
Ah, well, that was a risk that I took... Thank you for the feedback though! The Star Wars puns were too hard to resist... If you have a specific question, definitely post it here and I will clarify!
47.
by 1egg0myegg0 4y ago
Yes, that benchmark result is quite old in Duck years! :-) We actually run that benchmark as a part of our test suite now, so I am certain that there is improvement from that version. The biggest DuckDB I've used so far was about 400 G
48.
by 1egg0myegg0 4y ago
Fugue has a DuckDB back end and I believe they can actually use Dask and DuckDB in combination for what I believe is similar to what you are looking for! There is also a way to map Python functions in DuckDB using the relational (dataframe-
49.
by 1egg0myegg0 4y ago
Here is a paper that was used when building the DuckDB approach! https://db.in.tum.de/~leis/papers/ART.pdf There are some other papers and details about the architecture here: https://duckdb.org/wh
50.
by 1egg0myegg0 4y ago
Good questions! You are correct that it depends. We do have indexes to help with point queries, but they are not going to be quite as fast as SQLite because DuckDB stores data in a columnar format. (Soon they will be persistent - see commen
51.
by 1egg0myegg0 4y ago
Let me see if I can assuage some of your concerns! First off - DuckDB is MIT licensed, so you are welcome to use and enhance it essentially however you please! DuckDB Labs is a commercial entity that offers commercial support and custom int
52.
by 1egg0myegg0 4y ago
Yes, it is always difficult to use dates in SQLite... DuckDB makes dates easier - like they should be!
53.
by 1egg0myegg0 4y ago
Yes, good thought! That is listed at the bottom of the article as something we are looking at for the future.
54.
by 1egg0myegg0 4y ago
Persistent indexes are being actively worked on! Stay tuned. As for the crashes - DuckDB is very well tested and used in production in many places. The core functionality is very mature! Let us know if you test it out! Happy to help if I c
55.
by 1egg0myegg0 4y ago
Excellent question! I'll jump in - I am a part of the DuckDB team though, so if other users have thoughts it would be great to get other perspectives as well. First things first - we really like quite a lot about the SQLite approach. D
56.
by 1egg0myegg0 4y ago
Thank you for the feedback! I will check those Clickhouse features out. I totally agree on the trailing commas, and I use commas first syntax for that same reason! But maybe not anymore... :-)
57.
by 1egg0myegg0 4y ago
Close! DuckDB has very fast bulk insert speeds, but slower individual row insertion/update speeds. (Disclaimer: I write docs for DuckDB)
58.
by 1egg0myegg0 4y ago
Can it be done? Yes. Can I do it? No. Would I want to? Also no.
59.
by 1egg0myegg0 4y ago
I hope that DuckDB can help here! You can use the CLI or Python client to read a parquet file, but my favorite for quick exploration is popping open DBeaver (a SQL IDE) and using the DuckDB JDBC driver. That way you can see some column meta
60.
by 1egg0myegg0 4y ago
We have a Relational API in addition to SQL! Here are some examples for the Python Relational API client: https://github.com/duckdb/duckdb/blob/master/examples/python... Plus, if you are working in

More ›