4 ms·
Design and implementation of DuckDB internals
- viccis 6mo agoAm I missing something or is the content empty?
- esafak 6mo agohttps://github.com/DBatUTuebingen/DiDi https://github.com/DBatUTuebingen/DiDi
- viccis 6mo agoThank you, I didn't realize all of the course counted as "slides and auxiliary material" haha edit: Really great stuff in here. Every day at work I think about how much I love DuckDB
- esafak 6mo agoWhat do you use it for? What's the best part for you?
- goerch 6mo agoComputing page rank on par with NetworkX: https://github.com/idesis-gmbh/WikiExperiments https://github.com/idesis-gmbh/WikiExperiments Educational local DW from Github Archive events: https://github.com/idesis-gmbh/GitHubExperiments https://github.com/idesis-gmbh/GitHubExperiments It is quite fast for OLAP applications. It works on low cost hardware.
- viccis 6mo agoSorry I missed this reply. I'm currently using it to provide a SQL interface for doing data analytics for customers that don't have data large enough to justify a large platform like Snowflake or Databricks. Its syntax is close enough to the other two that, especially since the logic is already normalized through abstracted query definitions, it's a drop in replacement. Given that it's so lightweight, I can use it run searches in an AWS Lambda function, which is crazy useful.
- fg137 6mo agoUnfortunately it does not seem that there are lecture videos.
- mrtimo 6mo agoIf you are a data scientist or do anything with data... duckdb is like a swiss army knife. So many great ways it can help your workflow. The original video from CMU in 2020 [1] is a classic. Minutes 3-8 present a good argument for adding duckdb to your data cleaning/processing workflow. And if you want to add a semantic layer on top of data, Malloy [2] is my favorite so far (it has duckdb built in): [1]: https://www.youtube.com/watch?v=PFUZlNQIndo https://www.youtube.com/watch?v=PFUZlNQIndo [2]: https://docs.malloydata.dev/documentation/ https://docs.malloydata.dev/documentation/
- anitil 6mo agoThank you for the recommendation on that video! I've already adopted to using DuckDB for my ad-hoc analytics work but I didn't know the background
- buryat 6mo agothank you! Learned why DuckDB is named this way
- owlstuffing 6mo agoAnalytics with type-safe raw SQL (including DuckDb’s awesome extensions) is pure gold: https://github.com/manifold-systems/manifold/blob/master/docs/articles/duckdb_info.md https://github.com/manifold-systems/manifold/blob/master/doc...
- mpweiher 6mo agoThe actual slides are linked from the intro-text: https://github.com/DBatUTuebingen/DiDi https://github.com/DBatUTuebingen/DiDi
- password4321 6mo agoOver the years I've seen anecdotes here on HN that DuckDB crashes often for several people. Is this still an issue for anyone?
- xtracto 6mo agoWe use it heavily at my workplace. It doesn't crash at all if you use it as OLAP. But if you use it incorrectly, it will crash. It's pretty solid.
- wenc 6mo agoI use DuckDB daily. In short — It doesn’t crash often at all. What you may be remembering were reports of exceptional cases where it didn’t handle out of memory errors well. I was one of the people affected. I was running complex analytic queries on 400 GB parquets and I only had 128GB memory. It used jemalloc which didn’t gracefully degrade. They fixed a lot of the OOM issues so it’s more robust now. I haven’t had a crash for a long time. On normal sized datasets it never crashes.
- jazzpush2 6mo agoNever seen this and have several products that use it...
- chkrishnatej 6mo agoThat was a good start for understanding DuckDB internals!!!