3 ms·

Nice! Congrats on shipping. Any particular reason it's tied closely together with sqlite rather sqlalchemy which could potentially enable a range of backends? I

by ptype 9y ago

Nice! Congrats on shipping. Any particular reason it's tied closely together with sqlite rather sqlalchemy which could potentially enable a range of backends? I guess to ensure immutability?

simonw 9y ago
The "REST API for your database" thing isn't actually the part of this I'm most excited about - there are quite a few entries in that category already, such as https://github.com/jeffknupp/sandman2 https://github.com/jeffknupp/sandman2 The thing that excites me is being able to take almost any dataset, package it up into a simple, well-understood format and then deploy that in a read-only, highly scalable way, taking advantage of modern containerization systems. Read-only SQLite is /perfect/ for this use-case. It's fast, widely supported, has a great feature-set and can be distributed as a single file. My favourite feature of Datasette isn't the web UI, it's the publish command: datasette publish now ~/path-to-my/sqlite.db Combined with my https://github.com/simonw/csvs-to-sqlite https://github.com/simonw/csvs-to-sqlite tool it means you can take a CSV file and turn it into a live-on-the-internet REST SQL API in literally minutes. Here are a bunch of datasets that I've done that to just today: https://github.com/simonw/datasette/wiki/Datasettes https://github.com/simonw/datasette/wiki/Datasettes
- stdbrouw 9y ago
  I'm absolutely, unequivocally not trying to be snarky, but isn't CSV itself perfect for this use-case?
  - lukechilds 9y ago
    Only if you want the entire dataset. Datasette allows you to query for a subset of data.
  - _wmd 9y ago
    CSV files have a single order and no support for indexes. Even if you sort your CSV on one or more interesting columns and use binary search to locate an interesting range, you can still only do this along a single dimension, any remaining queries must scan the entire file. You'd also have to write all the code for this yourself. SQLite is a much better solution
  - simonw 9y ago
    CSV is a pretty terrible format for storing and transmitting data: it only supports string types, the escaping rules are poorly understood, and it can only model a single flat table.

by ptype 9y ago

Nice! Congrats on shipping. Any particular reason it's tied closely together with sqlite rather sqlalchemy which could potentially enable a range of backends? I guess to ensure immutability?

simonw 9y ago

The "REST API for your database" thing isn't actually the part of this I'm most excited about - there are quite a few entries in that category already, such as https://github.com/jeffknupp/sandman2 https://github.com/jeffknupp/sandman2 The thing that excites me is being able to take almost any dataset, package it up into a simple, well-understood format and then deploy that in a read-only, highly scalable way, taking advantage of modern containerization systems. Read-only SQLite is /perfect/ for this use-case. It's fast, widely supported, has a great feature-set and can be distributed as a single file. My favourite feature of Datasette isn't the web UI, it's the publish command: datasette publish now ~/path-to-my/sqlite.db Combined with my https://github.com/simonw/csvs-to-sqlite https://github.com/simonw/csvs-to-sqlite tool it means you can take a CSV file and turn it into a live-on-the-internet REST SQL API in literally minutes. Here are a bunch of datasets that I've done that to just today: https://github.com/simonw/datasette/wiki/Datasettes https://github.com/simonw/datasette/wiki/Datasettes

stdbrouw 9y ago
I'm absolutely, unequivocally not trying to be snarky, but isn't CSV itself perfect for this use-case?
- lukechilds 9y ago
  Only if you want the entire dataset. Datasette allows you to query for a subset of data.
- _wmd 9y ago
  CSV files have a single order and no support for indexes. Even if you sort your CSV on one or more interesting columns and use binary search to locate an interesting range, you can still only do this along a single dimension, any remaining queries must scan the entire file. You'd also have to write all the code for this yourself. SQLite is a much better solution
- simonw 9y ago
  CSV is a pretty terrible format for storing and transmitting data: it only supports string types, the escaping rules are poorly understood, and it can only model a single flat table.