legg0myegg0

5 ms·

1.
by legg0myegg0 5y ago
Does anyone happen to know which country the new company is incorporated in? I'm still looking for a chance to use ClickHouse because it sounds so excellent!
2.
by legg0myegg0 5y ago
I am not affiliated, just a happy user! I have spoken with one of the developers, but that's it! I've just come up through the SQL side of analytics and I'm moving into Data Science and I feel like DuckDB is a superpower for
3.
by legg0myegg0 5y ago
This is so fast!! If anybody is using Pandas to keep rows in order and has hesitated to use DuckDB for that reason, hesitate no more! Give it a shot!
4.
by legg0myegg0 5y ago
How would you compare the goals, vision, and current status of DataFusion with DuckDB? (www.duckdb.org) Could DuckDB be an execution engine for Ballista?
5.
Python and SQL: Better Together(alex-monahan.github.io)
1 points by legg0myegg0 5y ago|0 comments
6.
by legg0myegg0 5y ago
I'm not sure if it would help in your case, but could you process all categories at once with a larger SQL query? If so, DuckDB can process bulk queries about 20x faster than SQLite per CPU core because it is vectorized and column orie
7.
by legg0myegg0 5y ago
Came here just to recommend DuckDB! :-) Huge fan. It's unreasonably fast for how easy it is to use.
8.
by legg0myegg0 5y ago
Odd that DuckDB didn't work for you on Windows! I only use it on Windows and love it!
9.
by legg0myegg0 5y ago
Here is a DuckDB FDW for Postgres! I have not used it, but it sounds like what you need! https://github.com/alitrack/duckdb_fdw
10.
by legg0myegg0 5y ago
The best part of this is just how easy it is! Just a pip install and you're up and running using industry standard Postgres SQL!
11.
by legg0myegg0 5y ago
The DuckDB folks are migrating from pull to push and put together this interesting documentation of their reasoning! They use a vectorized model instead of a compiled one, so it's another interesting comparison point. It seems like pus
12.
by legg0myegg0 5y ago
I completely agree!! We see 20-100x performance from DuckDB over SQLite for OLAP style queries.
13.
by legg0myegg0 6y ago
Dremio also appears to take a similar approach, but with more advanced caching features / query pushdown. Plus it has Apache Arrow at its heart. I think that would be my choice of solution in this space
14.
by legg0myegg0 6y ago
Check out DuckDB! It is designed for OLAP instead of OLTP, but it uses Postgres syntax and types! It's columnar and lightning fast for big queries.
15.
by legg0myegg0 6y ago
Try DuckDB! I've been getting 20x SQLite performance on one thread, and it usually scales linearly with threads!
16.
by legg0myegg0 6y ago
Well then maybe don't design killer robots...??
17.
by legg0myegg0 6y ago
Go Diggerdoos!! Go Tech Go!
18.
by legg0myegg0 6y ago
We use plotly.js! It is a layer built on top of D3 and has some great looking charts. It also has some statistical and 3D plots that come in handy for us.
19.
by legg0myegg0 6y ago
I absolutely love this concept! I remember the Popular Science article years ago that described this for use on ships! What are the current challenges to scaling this more broadly? Can kites be made larger for more power, or what is the lim
20.
by legg0myegg0 6y ago
DuxkDB's query engine is inspired by the same paper! You'll be surprised what you can process on a single node with DuckDB - it takes 33 Spark nodes to match performance! That still puts it ahead of Photon, and it is open source.
21.
by legg0myegg0 6y ago
It's also possible to return a result set as an Arrow table, so round trip SQL on Arrow queries is possible (Arrow to DuckDB to Arrow)! It's not 100% zero-copy for strings, but it should work pretty well!
22.
by legg0myegg0 6y ago
I work at a Fortune 100 company and we have this in production for our self-service analytics platform as a part of our data transformation web service. Each web request can do multiple pandas transformations, or spin up it's own DuckD
23.
by legg0myegg0 6y ago
Before the latest optimization, and only using 1 core, vs. SQLite we were seeing 133x performance on a basic group by or join, and about 4x for a pretty complex query. It was roughly even to Pandas in performance, but it can scale to larger
24.
by legg0myegg0 6y ago
I think DuckDB checks a number of these boxes! It is embedded and written in C so compilable to WASM. It is also 10x faster than SQLite and interoperable with Apache Arrow! It might be a good place to start anyway!