3 ms·
Firstly quokka is pure Python, which I believe is good for interoperability with Python based UDFs. Secondly quokka tries to be fault tolerant. I.e. it can han
by marsupialtail_2 4y ago
Firstly quokka is pure Python, which I believe is good for interoperability with Python based UDFs.
Secondly quokka tries to be fault tolerant. I.e. it can handle worker failures intra query and not have to start over. This is quite important in real world spark deployments with thousands of nodes running many hours. AFAIK this is not well supported by most spark alternatives.
Finally quokka has a much stronger focus on time series data analytics. It is meant to excel at workloads like range joins and asof/PIT joins used for feature engineering. (This part isn't too stable yet so is not open source)
This means quokka optimizes on a different point in the UDF/performance/fault tolerance tradeoff space than something like arrow ballista or starrocks, which I think are pure performance plays
- theLiminator 4y agoInteresting, I was wondering if you considered building on top of https://github.com/apache/arrow-datafusion-python https://github.com/apache/arrow-datafusion-python I really do think a distributed db with compute/storage separation and optimized for feature engineering/dataloading (for training NNs) is underserved. I'd be very interested in the time series aspects of what you're building.
- marsupialtail_2 4y agohmm I wasn't aware of https://github.com/apache/arrow-datafusion-python https://github.com/apache/arrow-datafusion-python... thanks for the pointer. time series target release by April this year. main challenge is supporting them in the SQL API -- execution engine support is already done