houqp

7 ms·

31.
by houqp 5y ago
Neat! I have also built a similar project in Rust https://github.com/roapi/roapi/tree/main/columnq-cli :)
32.
Embedded OLAP engine Apache Arrow Datafusion 6.0.0 release(arrow.apache.org)
8 points by houqp 5y ago|0 comments
33.
Apache Arrow DataFusion 6.0.0 Release(arrow.apache.org)
1 points by houqp 5y ago|0 comments
34.
Apache Arrow Datafusion 6.0.0 release(arrow.apache.org)
2 points by houqp 5y ago|0 comments
35.
by houqp 5y ago
I think these two systems explore different design spaces, the biggest difference I would say is Roapi can apply more read optimizations by exploiting the fact that it doesn't need to support frequent online updates from the client. Mo
36.
by houqp 5y ago
In its current form, the main use-case is to load data into memory first then serve them through query apis. Thomas has made some effort to support querying data directly from remote source without loading them into memory: https:/&#x
37.
by houqp 5y ago
Thanks, nice work on qocache and qframe too :)
38.
by houqp 5y ago
yeah, that's a good idea. thanks for the suggestion :)
39.
by houqp 5y ago
Yes, I am aiming for production grade online serving + many more query frontends and data types.
40.
by houqp 5y ago
I looked into Datasette before starting ROAPI. From a product/use-case point of view, to me Datasette focuses more on quick and easy ad-hoc data exploration type of work. ROAPI focuses more production ready online serving of static dat
41.
by houqp 5y ago
This is true, the core of it is Apache arrow datafusion query engine, which is also a project I help maintain. I doubt you will be able to beat it with PHP though ;) The VM overhead alone will cause a big hit to your performance even if we
42.
by houqp 5y ago
That's right, it's intended to be more lightweight since it's built with only Rust from the ground up. Apache Drill also only focuses on serving SQL as the user interface while ROAPI wants to provide a pluggable interface to
43.
by houqp 5y ago
Author of the project here, thanks for writing about ropai! Happy to answer any question.
44.
by houqp 5y ago
Thanks! It's using Datafusion as the query engine: https://github.com/apache/arrow-datafusion
45.
Exactly once delivery from Kafka to Delta Lake with Rust(github.com)
1 points by houqp 5y ago|0 comments
46.
Show HN: Columnq brings OLAP to Unix pipes(github.com)
32 points by houqp 5y ago|2 comments
47.
Show HN: Query small data with columnq CLI(github.com)
2 points by houqp 5y ago|0 comments
48.
by houqp 5y ago
I didn't dive into Vaex's implementation, but based on the example code, I would say they are similar in the sense that they all provide a Dataframe interface for end users to perform compute on relational data. It looks like Vaex
49.
by houqp 5y ago
Datafusion, and Ballista by definition, also provides a Dataframe API that let's you construct queries programmatically. It also has preliminary support for UDFs. We also have community members implementing Spark native executors using
50.
by houqp 5y ago
ETL pipeline is a perfect fit for Datafusion and its distributed version Ballista. Personally, this is the main reason I am investing my time into Datafusion.
51.
by houqp 5y ago
Indeed, big shout out to the InfluxDB team!
52.
by houqp 5y ago
> - Is it possible to handle data larger than fits into RAM? Not at the moment, but the community has plans to add support for disk spill. > - Any benchmark? like: https://h2oai.github.io/db-benchmark/ ( see 50GB
53.
by houqp 5y ago
You beat me to it, was about to post the github link :) Readme is a good starting place to learn more about the project.
54.
by houqp 5y ago
One of the Arrow Datafusion committers here. Happy to help answer any question.
55.
Apache Arrow Datafusion 5.0.0 release(arrow.apache.org)
78 points by houqp 5y ago|44 comments
56.
Datafusion 5.0.0 release with major new features and performance improvements(arrow.apache.org)
9 points by houqp 5y ago|1 comments
57.
by houqp 5y ago
Scribd | REMOTE or onsite | full-time Scribd has opening for lots of teams in all levels, please check them out out at https://www.scribd.com/careers ! Special shout out to the data platform positions. See https:/&#x2F
58.
Show HN: Parse and Visualize Brainwaves with Rust(github.com)
3 points by houqp 5y ago|0 comments
59.
by houqp 5y ago
Shameless plug, I also built a tool in Rust to provide SQL/GraphQL query access to CSV and many other tabular file formats: https://github.com/roapi/roapi .
60.
Automatically recycling EKS worker nodes(tech.scribd.com)
3 points by houqp 6y ago|1 comments

More ›