5 ms·

What's the difference between this, and Datasette which has been in development multiple years by the Django co-creator? Sorry if it's a dumb question but I sk

What's the difference between this, and Datasette which has been in development multiple years by the Django co-creator? Sorry if it's a dumb question but I skimmed the docs and sample and it looks similar + didn't see any mention: https://github.com/simonw/datasette https://github.com/simonw/datasette

FraaJad 5y ago
datasette is only sqlite, ROAPI serves up CSV, Parquet storage etc.,
- gavinray 5y ago
  Datasette serves JSON, CSV, SQLite, and a bunch of others as well. Don't think it handles Parquet tho https://docs.datasette.io/en/stable/ecosystem.html#sqlite-utils "Insert data into a SQLite database from JSON, CSV or TSV, automatically creating tables with the correct schema or altering existing tables to add missing columns." There's a bunch of plugins (mix of official and user-developed) that might add other formats too, I'm not 100% sure: https://datasette.io/plugins?sort=downloads-this-week https://datasette.io/plugins?sort=downloads-this-week
houqp 5y ago
I looked into Datasette before starting ROAPI. From a product/use-case point of view, to me Datasette focuses more on quick and easy ad-hoc data exploration type of work. ROAPI focuses more production ready online serving of static datasets. So I would expect users to use ROAPI to power micro-services in production with high QPS. From a technical design point of view, ROAPI authors owns the full stack end to end from query parsing, data format parsing to query execution because I am also a maintainer of Apache arrow and it's sub-project datafusion. The whole project is built with Rust end to end from scratch. Datasette is mostly a wrapper around sqlite. It translates user actions into SQL queries, then execute them on sqlite. In ROAPI, we work at a lower level. We translate REST APIs, GraphQL and SQLs into datafusion logical plans and execute them. Datafusion is also a analytical compute engine optimized for columnar data, so it will be a lot faster for OLAP workload, while sqlite is optimized for OLTP. I also plan to add other type of query capabilities like nearest neighbor vector search for ML applications, etc.
- gavinray 5y ago
  Thank you for a genuine, well-thought answer. Much appreciated. I'm not in the Data Science space so I only know of Datasette, but maybe worth copy-pasting that on a FAQ page under "How does ROAPI compare to X?" to avoid repeating it.
  - houqp 5y ago
    yeah, that's a good idea. thanks for the suggestion :)
- stereosteve 5y ago
  Similar question: how does it compare to using ClickHouse with http interface (which supports a number of output formats including Arrow and ArrowStream). It does take a few more commands to start a ClickHouse server, create table and load data in... a friendly loader program could probably get pretty close to Roapi interface. Both Roapi and Datafusion look very cool! Excited to learn about them.
  - houqp 5y ago
    I think these two systems explore different design spaces, the biggest difference I would say is Roapi can apply more read optimizations by exploiting the fact that it doesn't need to support frequent online updates from the client. Most of the datasets it serves will be static. For data-sources that supports streaming updates like delta tables, the update frequency will be much lower than what clickhouse supports.
    - zX41ZdbW 5y ago
      This is not a concern for ClickHouse. You can load data into MergeTree table that support streaming data ingestion or into Memory table (that is read/write as well). Also you can run queries on the data without any preprocessing (with `file`, `url` table functions or similar). See also https://clickhouse.com/docs/en/operations/utilities/clickhouse-local/ https://clickhouse.com/docs/en/operations/utilities/clickhou...

FraaJad 5y ago

datasette is only sqlite, ROAPI serves up CSV, Parquet storage etc.,

gavinray 5y ago
Datasette serves JSON, CSV, SQLite, and a bunch of others as well. Don't think it handles Parquet tho https://docs.datasette.io/en/stable/ecosystem.html#sqlite-utils "Insert data into a SQLite database from JSON, CSV or TSV, automatically creating tables with the correct schema or altering existing tables to add missing columns." There's a bunch of plugins (mix of official and user-developed) that might add other formats too, I'm not 100% sure: https://datasette.io/plugins?sort=downloads-this-week https://datasette.io/plugins?sort=downloads-this-week

houqp 5y ago

I looked into Datasette before starting ROAPI. From a product/use-case point of view, to me Datasette focuses more on quick and easy ad-hoc data exploration type of work. ROAPI focuses more production ready online serving of static datasets. So I would expect users to use ROAPI to power micro-services in production with high QPS. From a technical design point of view, ROAPI authors owns the full stack end to end from query parsing, data format parsing to query execution because I am also a maintainer of Apache arrow and it's sub-project datafusion. The whole project is built with Rust end to end from scratch. Datasette is mostly a wrapper around sqlite. It translates user actions into SQL queries, then execute them on sqlite. In ROAPI, we work at a lower level. We translate REST APIs, GraphQL and SQLs into datafusion logical plans and execute them. Datafusion is also a analytical compute engine optimized for columnar data, so it will be a lot faster for OLAP workload, while sqlite is optimized for OLTP. I also plan to add other type of query capabilities like nearest neighbor vector search for ML applications, etc.

gavinray 5y ago
Thank you for a genuine, well-thought answer. Much appreciated. I'm not in the Data Science space so I only know of Datasette, but maybe worth copy-pasting that on a FAQ page under "How does ROAPI compare to X?" to avoid repeating it.
- houqp 5y ago
  yeah, that's a good idea. thanks for the suggestion :)
stereosteve 5y ago
Similar question: how does it compare to using ClickHouse with http interface (which supports a number of output formats including Arrow and ArrowStream). It does take a few more commands to start a ClickHouse server, create table and load data in... a friendly loader program could probably get pretty close to Roapi interface. Both Roapi and Datafusion look very cool! Excited to learn about them.
- houqp 5y ago
  I think these two systems explore different design spaces, the biggest difference I would say is Roapi can apply more read optimizations by exploiting the fact that it doesn't need to support frequent online updates from the client. Most of the datasets it serves will be static. For data-sources that supports streaming updates like delta tables, the update frequency will be much lower than what clickhouse supports.
  - zX41ZdbW 5y ago
    This is not a concern for ClickHouse. You can load data into MergeTree table that support streaming data ingestion or into Memory table (that is read/write as well). Also you can run queries on the data without any preprocessing (with `file`, `url` table functions or similar). See also https://clickhouse.com/docs/en/operations/utilities/clickhouse-local/ https://clickhouse.com/docs/en/operations/utilities/clickhou...