marsupialtail_2

7 ms·

31.
by marsupialtail_2 4y ago
OK I'll admit: https://news.ycombinator.com/item?id=34189422 is not a real pure Python SQL engine, this one is.
32.
by marsupialtail_2 4y ago
Just did.
33.
by marsupialtail_2 4y ago
I am writing a Python based SQL query engine: https://github.com/marsupialtail/quokka . My personal goal is to get Andy Pavlo to mention it in his year-end blogs. I agree with many of the points made in the blog by Andy
34.
by marsupialtail_2 4y ago
Wait this is really cool. How do you parse and optimize SQL?
35.
by marsupialtail_2 4y ago
I sponsor OSS software I depend on and you should too!
36.
by marsupialtail_2 4y ago
Let's make this top-rated comment :-P
37.
by marsupialtail_2 4y ago
That's right. My background is mostly in quantitative finance, where we would use models like linear regression on expert-engineered features based on market data, instead of throwing a deep neural network at raw price data like what s
38.
by marsupialtail_2 4y ago
None of the benchmarks involved any UDFs.
39.
by marsupialtail_2 4y ago
Polars is adding that feature itself. You probably just have to wait a couple weeks
40.
by marsupialtail_2 4y ago
I have. I am about 2x faster than trino with fault tolerance. But I didn't put the numbers on that plot because this trino feature is still really new and I might not be benchmarking it in the best way.
41.
by marsupialtail_2 4y ago
I only wrote Python. I am using C++ and Rust libraries
42.
by marsupialtail_2 4y ago
Yeah it's just that the UDFs I see people need require Python packages like sklearn and PyTorch that really don't have good alternatives in other libraries.
43.
by marsupialtail_2 4y ago
that's certainly one way to look at it
44.
by marsupialtail_2 4y ago
Yes -- we are building a sql compiler to the dataframe API. It currently passes half of TPC-H, but is not really ready. It will be open-sourced soon. The SQL optimizations like predicate pushdown and early projection are all there already i
45.
by marsupialtail_2 4y ago
Name?
46.
by marsupialtail_2 4y ago
The focus is olap and etl not oltp. In fact I assume data doesn't change :-)
47.
by marsupialtail_2 4y ago
SQLGlot is really great, check it out!
48.
by marsupialtail_2 4y ago
I don't really compile python functions to sql -- I just execute sql. It's just that the execution engine is written in python
49.
by marsupialtail_2 4y ago
hmm I wasn't aware of https://github.com/apache/arrow-datafusion-python ... thanks for the pointer. time series target release by April this year. main challenge is supporting them in the SQL API -- execution engin
50.
by marsupialtail_2 4y ago
Firstly quokka is pure Python, which I believe is good for interoperability with Python based UDFs. Secondly quokka tries to be fault tolerant. I.e. it can handle worker failures intra query and not have to start over. This is quite importa
51.
by marsupialtail_2 4y ago
Checkpointing is actually a bit difficult for fault tolerance for long running batch computations and state tends to grow linearly as the computation progresses, unlike streaming applications where you typically use windows. This means peri
52.
by marsupialtail_2 4y ago
Perhaps you can write the underlying query engine :-) in something that's not Python lol
53.
by marsupialtail_2 4y ago
I just find it funny how my post on why I wrote a SQL engine in pure Python is right next to this one on the front page
54.
I wrote a SQL engine in Python(github.com)
282 points by marsupialtail_2 4y ago|73 comments
55.
by marsupialtail_2 4y ago
Ask me anything :-)
56.
Show HN: Rewriting SparkSQL in Python(github.com)
4 points by marsupialtail_2 4y ago|1 comments