4 ms·
I looked into Datasette before starting ROAPI. From a product/use-case point of view, to me Datasette focuses more on quick and easy ad-hoc data exploration typ
by houqp 5y ago
I looked into Datasette before starting ROAPI. From a product/use-case point of view, to me Datasette focuses more on quick and easy ad-hoc data exploration type of work. ROAPI focuses more production ready online serving of static datasets. So I would expect users to use ROAPI to power micro-services in production with high QPS.
From a technical design point of view, ROAPI authors owns the full stack end to end from query parsing, data format parsing to query execution because I am also a maintainer of Apache arrow and it's sub-project datafusion. The whole project is built with Rust end to end from scratch. Datasette is mostly a wrapper around sqlite. It translates user actions into SQL queries, then execute them on sqlite. In ROAPI, we work at a lower level. We translate REST APIs, GraphQL and SQLs into datafusion logical plans and execute them. Datafusion is also a analytical compute engine optimized for columnar data, so it will be a lot faster for OLAP workload, while sqlite is optimized for OLTP. I also plan to add other type of query capabilities like nearest neighbor vector search for ML applications, etc.
- gavinray 5y agoThank you for a genuine, well-thought answer. Much appreciated. I'm not in the Data Science space so I only know of Datasette, but maybe worth copy-pasting that on a FAQ page under "How does ROAPI compare to X?" to avoid repeating it.
- houqp 5y agoyeah, that's a good idea. thanks for the suggestion :)
- stereosteve 5y agoSimilar question: how does it compare to using ClickHouse with http interface (which supports a number of output formats including Arrow and ArrowStream). It does take a few more commands to start a ClickHouse server, create table and load data in... a friendly loader program could probably get pretty close to Roapi interface. Both Roapi and Datafusion look very cool! Excited to learn about them.
- houqp 5y agoI think these two systems explore different design spaces, the biggest difference I would say is Roapi can apply more read optimizations by exploiting the fact that it doesn't need to support frequent online updates from the client. Most of the datasets it serves will be static. For data-sources that supports streaming updates like delta tables, the update frequency will be much lower than what clickhouse supports.
- zX41ZdbW 5y agoThis is not a concern for ClickHouse. You can load data into MergeTree table that support streaming data ingestion or into Memory table (that is read/write as well). Also you can run queries on the data without any preprocessing (with `file`, `url` table functions or similar). See also https://clickhouse.com/docs/en/operations/utilities/clickhouse-local/ https://clickhouse.com/docs/en/operations/utilities/clickhou...