3 ms·

been seeing a lot of these convert X into JSON api with query ...but what would be the use cases for this type of tool? who needs this type of solution?

by pwaai 9y ago

simonw 9y ago
Datasette is specifically designed for publishing static data - it doesn't handle UPDATE/INSERT traffic at all. This is a good fit for any time you want to share some static data with the world - instead of posting a CSV file, you can share your data as a Datasette instance with a browseable web UI and a JSON API instead. The three use-cases I'm personally most interested in for this are: Data journalism: newspapers collect data about e.g. police shootings https://vice-police-shootings.now.sh/ https://vice-police-shootings.now.sh/ or actions under the antiquities act https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9/antiquities-act%2Factions_under_antiquities_act https://fivethirtyeight.datasettes.com/fivethirtyeight-2628d... - I want to make it as easy as possible to publish the data beneath the stories. Government open data: cities like San Francisco publish huge amounts if interesting data. I used San Francisco's 190,000 line CSV of trees in the city to build https://sf-tree-search.now.sh/ https://sf-tree-search.now.sh/ Cultural institutions: museums, libraries and art galleries have the collection and sharing of data as part of their mandates. I want to make it as easy as possible to build things like the London Science Museum's collection API: https://group.sciencemuseum.org.uk/about-us/collection/using-our-collection-api/ https://group.sciencemuseum.org.uk/about-us/collection/using...
- toomuchtodo 9y ago
  Why not convert the data from csv to json and serve the json blob from your object store or CDN (if this solution is immutable)? No need for the API interface, just gzip the json blob and have the client retrieve it entirely. No need for an app, app server, and the json can be cached with the rest of your content. Unless I’m missing something?
  - bdcravens 9y ago
    If you're serving the entire JSON dataset, sure, but this solution allows you to do server-side filtering.
  - simonw 9y ago
    Size. The San Francisco trees database is around 50MB, and I've successfully tested it with multiple GB sized SQLite databases.
    - lozzo 9y ago
      this was a key answer to a good question. I think you should emphasize that part (handling large data sets) as a key aspect of datasette
chrsstrm 9y ago
I'm just finishing up a tool that lets you use Google Sheets as a data API resource that returns JSON. In my particular case I have many clients who want the ease of use of WordPress but don't have the resources to keep it properly maintained in terms of update security. Many of these clients do not actually need to update most of their site on any sort of frequent basis. The case that prompted my new project is a restaurant who updates their daily specials on a daily basis, their seasonal specials on a monthly basis, and other menu items maybe 2-3 times a year in addition to semi-frequent homepage background image changes and an occasional flash message. Instead of giving them WordPress, this is what I did: The site is built in Jekyll so it is 100% static. The dynamic portions of the site like the food menu makes use of Jekyll's data files feature. The site owner makes changes to their menu in a Google Sheet (which is where they typically store this anyway) and then chooses a deploy target and submits a custom Google Form. The form submit pings a listening Go server on the deploy target which then kicks off a build script. The build script requests the Sheet data one sheet at a time and stores the results in JSON files which Jekyll can use. A couple of additional steps to get images into place and then the site is compiled by Jekyll and linked into the nginx webroot. The script even writes back to a log sheet to let the user know how the build went. With selectable deploy targets the site owner can preview her changes on a staging site first before deploying to prod. And just like that, the user has a site with all the speed and security of a static site and all the flexibility of a dynamic site.

been seeing a lot of these convert X into JSON api with query ...but what would be the use cases for this type of tool? who needs this type of solution?

by pwaai 9y ago

been seeing a lot of these convert X into JSON api with query ...but what would be the use cases for this type of tool? who needs this type of solution?

simonw 9y ago

Datasette is specifically designed for publishing static data - it doesn't handle UPDATE/INSERT traffic at all. This is a good fit for any time you want to share some static data with the world - instead of posting a CSV file, you can share your data as a Datasette instance with a browseable web UI and a JSON API instead. The three use-cases I'm personally most interested in for this are: Data journalism: newspapers collect data about e.g. police shootings https://vice-police-shootings.now.sh/ https://vice-police-shootings.now.sh/ or actions under the antiquities act https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9/antiquities-act%2Factions_under_antiquities_act https://fivethirtyeight.datasettes.com/fivethirtyeight-2628d... - I want to make it as easy as possible to publish the data beneath the stories. Government open data: cities like San Francisco publish huge amounts if interesting data. I used San Francisco's 190,000 line CSV of trees in the city to build https://sf-tree-search.now.sh/ https://sf-tree-search.now.sh/ Cultural institutions: museums, libraries and art galleries have the collection and sharing of data as part of their mandates. I want to make it as easy as possible to build things like the London Science Museum's collection API: https://group.sciencemuseum.org.uk/about-us/collection/using-our-collection-api/ https://group.sciencemuseum.org.uk/about-us/collection/using...

toomuchtodo 9y ago
Why not convert the data from csv to json and serve the json blob from your object store or CDN (if this solution is immutable)? No need for the API interface, just gzip the json blob and have the client retrieve it entirely. No need for an app, app server, and the json can be cached with the rest of your content. Unless I’m missing something?
- bdcravens 9y ago
  If you're serving the entire JSON dataset, sure, but this solution allows you to do server-side filtering.
- simonw 9y ago
  Size. The San Francisco trees database is around 50MB, and I've successfully tested it with multiple GB sized SQLite databases.
  - lozzo 9y ago
    this was a key answer to a good question. I think you should emphasize that part (handling large data sets) as a key aspect of datasette

chrsstrm 9y ago

I'm just finishing up a tool that lets you use Google Sheets as a data API resource that returns JSON. In my particular case I have many clients who want the ease of use of WordPress but don't have the resources to keep it properly maintained in terms of update security. Many of these clients do not actually need to update most of their site on any sort of frequent basis. The case that prompted my new project is a restaurant who updates their daily specials on a daily basis, their seasonal specials on a monthly basis, and other menu items maybe 2-3 times a year in addition to semi-frequent homepage background image changes and an occasional flash message. Instead of giving them WordPress, this is what I did: The site is built in Jekyll so it is 100% static. The dynamic portions of the site like the food menu makes use of Jekyll's data files feature. The site owner makes changes to their menu in a Google Sheet (which is where they typically store this anyway) and then chooses a deploy target and submits a custom Google Form. The form submit pings a listening Go server on the deploy target which then kicks off a build script. The build script requests the Sheet data one sheet at a time and stores the results in JSON files which Jekyll can use. A couple of additional steps to get images into place and then the site is compiled by Jekyll and linked into the nginx webroot. The script even writes back to a log sheet to let the user know how the build went. With selectable deploy targets the site owner can preview her changes on a staging site first before deploying to prod. And just like that, the user has a site with all the speed and security of a static site and all the flexibility of a dynamic site.