4 ms·

Absolutely, having ability to download the actual data and keep it is always going to be important. We want to facilitate access to data and think it should be

Absolutely, having ability to download the actual data and keep it is always going to be important. We want to facilitate access to data and think it should be available from the source. But, there will inevitably be fragmentation, so it's valuable to have a service available to catalog and aggregate it and make it available over a single protocol. For what it's worth, we run PostgREST [0] on top of the DDN, so you can get your query results in JSON and CSV files. For example: $ curl -sSH "Content-Type: text/csv" https://data.splitgraph.com/cityofchicago/covid19-daily-cases-deaths-and-hospitalizations-naz8-j4nc/latest/-/rest/covid19_daily_cases_deaths_and_hospitalizations | wc -l 172 It's limited to 10k rows but we might have an ability in the future to "order" a CSV dump asynchronously and place in a destination (like an S3 bucket) of choice. [0] https://postgrest.org/en/latest/ https://postgrest.org/en/latest/

atombender 6y ago
Did you mean to use the "Accept" header? "Content-Type" describes the format of the request body.
- mildbyte 6y ago
  Yeah, sorry, got too excited! It will return JSON by default and CSV with Accept: text/csv: $ curl -sH "Accept: text/csv" https://data.splitgraph.com/cityofchicago/covid19-daily-cases-deaths-and-hospitalizations-naz8-j4nc/latest/-/rest/covid19_daily_cases_deaths_and_hospitalizations?select=lab_report_date,cases_total,deaths_total | head -n5 lab_report_date,cases_total,deaths_total 2020-03-01,0,0 2020-03-02,0,0 2020-03-03,0,0 2020-03-04,0,0
  - atombender 6y ago
    Very nice!
mbostock 6y ago
Any chance you’re willing to enable CORS on these endpoints? (Access-Control-Allow-Origin: * response header.)
- chatmasta 6y ago
  Hey! Big fan of your work. For the REST API endpoints (e.g. [0] [1]), we do set the Access-Control-Allow-Origin header to *, but for GET requests we do not send it at all. I'm not sure if this is the same as setting it to star. We can set it explicitly though; we do want to enable this use case (eventually we'll have quotas). Soon we're releasing a Web UI for writing SQL queries. Part of that will include an HTTP (or websocket maybe) endpoint that takes a SQL query and returns the rows, so you'll be able to execute SQL queries directly from the browser. As an aside, we'd love to get this working with Observable! Let us know anything we can do to help. You can email me at miles@splitgraph.com or join the Discord and we can figure it out :) [0] Docs: https://www.splitgraph.com/cambridgema-gov/covid19-cumulative-cases-by-date-tdt9-vq5y/latest/-/api-schema https://www.splitgraph.com/cambridgema-gov/covid19-cumulativ... [1] Endpoint (powered by PostgREST): https://data.splitgraph.com:443/cambridgema-gov/covid19-cumulative-cases-by-date-tdt9-vq5y/latest/-/rest/covid19_cumulative_cases_by_date https://data.splitgraph.com:443/cambridgema-gov/covid19-cumu...
  - dylz 6y ago
    Not setting it is equivalent to a deny
- chatmasta 6y ago
  Ok, so this should be working now, for all types of requests. Let us know if it's not. If you're testing with the same endpoints, you may need to give the cache a few minutes to evict the old values missing the header.
  - mbostock 6y ago
    It works, thanks! https://observablehq.com/@mbostock/hello-splitgraph https://observablehq.com/@mbostock/hello-splitgraph

atombender 6y ago

Did you mean to use the "Accept" header? "Content-Type" describes the format of the request body.

mildbyte 6y ago
Yeah, sorry, got too excited! It will return JSON by default and CSV with Accept: text/csv: $ curl -sH "Accept: text/csv" https://data.splitgraph.com/cityofchicago/covid19-daily-cases-deaths-and-hospitalizations-naz8-j4nc/latest/-/rest/covid19_daily_cases_deaths_and_hospitalizations?select=lab_report_date,cases_total,deaths_total | head -n5 lab_report_date,cases_total,deaths_total 2020-03-01,0,0 2020-03-02,0,0 2020-03-03,0,0 2020-03-04,0,0
- atombender 6y ago
  Very nice!

mbostock 6y ago

Any chance you’re willing to enable CORS on these endpoints? (Access-Control-Allow-Origin: * response header.)

chatmasta 6y ago
Hey! Big fan of your work. For the REST API endpoints (e.g. [0] [1]), we do set the Access-Control-Allow-Origin header to *, but for GET requests we do not send it at all. I'm not sure if this is the same as setting it to star. We can set it explicitly though; we do want to enable this use case (eventually we'll have quotas). Soon we're releasing a Web UI for writing SQL queries. Part of that will include an HTTP (or websocket maybe) endpoint that takes a SQL query and returns the rows, so you'll be able to execute SQL queries directly from the browser. As an aside, we'd love to get this working with Observable! Let us know anything we can do to help. You can email me at miles@splitgraph.com or join the Discord and we can figure it out :) [0] Docs: https://www.splitgraph.com/cambridgema-gov/covid19-cumulative-cases-by-date-tdt9-vq5y/latest/-/api-schema https://www.splitgraph.com/cambridgema-gov/covid19-cumulativ... [1] Endpoint (powered by PostgREST): https://data.splitgraph.com:443/cambridgema-gov/covid19-cumulative-cases-by-date-tdt9-vq5y/latest/-/rest/covid19_cumulative_cases_by_date https://data.splitgraph.com:443/cambridgema-gov/covid19-cumu...
- dylz 6y ago
  Not setting it is equivalent to a deny
chatmasta 6y ago
Ok, so this should be working now, for all types of requests. Let us know if it's not. If you're testing with the same endpoints, you may need to give the cache a few minutes to evict the old values missing the header.
- mbostock 6y ago
  It works, thanks! https://observablehq.com/@mbostock/hello-splitgraph https://observablehq.com/@mbostock/hello-splitgraph