3 ms·
This is a perfectly reasonable question. I've been thinking about this for five years and I still don't have a great, snappy answer to this. I can answer it in
by simonw 3y ago
This is a perfectly reasonable question. I've been thinking about this for five years and I still don't have a great, snappy answer to this.
I can answer it in several paragraphs, but I really would like to be able to answer it in a single sentence some day!
So here goes with the several paragraph version...
The problem that Datasette solves better than anything else (a big claim, which I'm happy to be challenged on) is publishing structured data online.
Let's say you have data you want to share online - every global power plant, or the full history of the US congress. How do you do it?
Some options:
- Publish some CSV files somewhere - on a website, on GitHub, to S3
- Build a custom application for it, often using Django or Rails or similar.
- Put it in a Google Sheet - that's how we solved this problem at the Guardian years ago, see https://simonwillison.net/2018/Aug/19/instantly-publish-datasette/ https://simonwillison.net/2018/Aug/19/instantly-publish-data...
Datasette was originally created to take on this problem. I realized that SQLite is the perfect platform for this: it's fast, robust and crucially can be deployed anywhere that can host a dynamic web application (if you're publishing read-only data you don't need to worry about backups and replication and suchlike).
Here's every global power plant: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants https://global-power-plants.datasettes.com/global-power-plan...
And US congressional legislators: https://congress-legislators.datasettes.com/legislators https://congress-legislators.datasettes.com/legislators - I use that one in the Datasette tutorial: https://datasette.io/tutorials/explore https://datasette.io/tutorials/explore
If you're comfortable with the command-line, I challenge you to find a quicker way to publish data online than this:
sqlite-utils insert manatees.db locations \
Manatee_Carcass_Recovery_Locations_in_Florida.csv --csv -d
sqlite-utils transform manatees.db locations \
--rename LAT latitude \
--rename LONG_ longitude \
--drop created_user \
--drop last_edited_user \
--drop X \
--drop Y \
--drop STATE \
--drop OBJECTID \
--pk FIELDID
datasette publish vercel manatees.db \
--project datasette-manatees --install datasette-cluster-map
That's using the datasette-publish-vercel plugin, but Datasette can also publish to Fly, Google Cloud Run, Heroku and more using additional plugins: https://docs.datasette.io/en/stable/publish.html https://docs.datasette.io/en/stable/publish.html
For more on the sqlite-utils bits see https://datasette.io/tutorials/clean-data https://datasette.io/tutorials/clean-data
So that's publishing. But Datasette has grown far beyond that in the five years I've been working on it.
I find myself turning to it any time I have any data I want to poke at and start exploring. That's the data journalism angle - "find stories in data".
In terms of commercial applications, I have a strong hunch that if I can help journalists find stories in their data, I can help everyone else find stories in their data as well.
Another key detail here is the plugins.
WordPress is a good CMS... with 10,000+ plugins that mean you can point it at any content publishing problem you can think of. As a result, it runs a double-digit percentage of the web now.
The most ambitious version of Datasette looks like that.
I want to build an open source EDA (Exploratory Data Analysis) and publishing tool that has thousands of plugins that mean you can use it to solve any data exploration, analysis, visualization or publishing problem.
It's at 127 plugins so far, so there's still a long way to go - but it's a great start! https://datasette.io/plugins https://datasette.io/plugins
Attempting to turn the above into single sentences is hard, because there are a lot of different angles to it - but here are a few attempts:
Datasette is the fastest way to publish data online as an interactive, searchable database.
Datasette is WordPress for data: an extensible open source platform with plugins for exploring, analyzing, visualizing, and publishing data.
- virtuous_sloth 3y agoHeh... "I can answer it in several paragraphs, but I really would like to be able to answer it in a single sentence some day!" ... "Datasette is WordPress for data: an extensible open source platform with plugins for exploring, analyzing, visualizing, and publishing data."
- simonw 3y agoConfession: I posted my multi-paragraph comment into Claude and asked it for some ideas. It didn't come up with exactly that, but what it said helped me get there: https://gist.github.com/simonw/a160a49d39446aa53870ed1ca43d7e39 https://gist.github.com/simonw/a160a49d39446aa53870ed1ca43d7...
- DandyDev 3y agoThis is a great explanation. And to me, what sets Datasette apart from a generic SQL UI, is that Datasette excels at publishing _specific_ and curated datasets and allowing interactive exploration in a way that plain CSVs just don't offer.