Hey! Socrata is awesome :) The basic idea here is that you can write a Splitgraph plugin for any type of data source [0], so we like to use Socrata as a conveni

Hey! Socrata is awesome :) The basic idea here is that you can write a Splitgraph plugin for any type of data source [0], so we like to use Socrata as a convenient example. Each plugin includes a Postgres Foreign Data Wrapper (FDW) to translate from SQL to whatever upstream query language (e.g. Socrata FDW [1]). By default we don't copy any data into Splitgraph (although we do cache query results keyed by AST). When a query arrives, we parse it for any table identifiers, and "mount" a temporary foreign table for each, using the appropriate FDW. Foreign tables are mostly transparent to the Postgres query planner, so a JOIN will work (example [2]), and an FDW can even optionally implement aggregate pushdown. But yes, especially in the case of cross-dataset JOINS, there are some pathological cases. When your use case outgrows live querying (e.g. due to pathological JOIN queries), you can ingest any data source into a Splitgraph image, a versioned snapshot of data similar to a Docker image. [3] Our focus right now is on the internal enterprise use case. We use the Socrata plugin as a demo, but our customers are more interested in plugins for data sources they use internally, e.g. Postgres, Snowflake, Google Search Console, CSV in S3, etc. We support all those too; you can mount them all locally, and you can also mount them on Splitgraph.com (or an internal deployment), but still thin docs / no web UI yet. [0] https://www.splitgraph.com/blog/foreign-data-wrappers https://www.splitgraph.com/blog/foreign-data-wrappers [1] https://github.com/splitgraph/splitgraph/blob/master/splitgraph/ingestion/socrata/fdw.py https://github.com/splitgraph/splitgraph/blob/master/splitgr... [2] https://www.splitgraph.com/workspace/ddn?layout=hsplit&query=SELECT%0A++++cambridge_cases.date+AS+date%2C%0A++++chicago_cases.cases_total+AS+chicago_daily_cases%2C%0A++++cambridge_cases.new_positive_cases+AS+cambridge_daily_cases%0AFROM%0A++++%22cityofchicago%2Fcovid19-daily-cases-deaths-and-hospitalizations-naz8-j4nc%22.covid19_daily_cases_deaths_and_hospitalizations+chicago_cases%0AFULL+OUTER+JOIN%0A++++%22cambridgema-gov%2Fcovid19-case-count-by-date-axxk-jvk8%22.covid19_case_count_by_date+cambridge_cases%0AON%0A++++date_trunc%28%27day%27%2C+chicago_cases.lab_report_date%3A%3Atimestamp%29+%3D+cambridge_cases.date%3A%3Atimestamp%0AWHERE+date+IS+NOT+NULL%0AORDER+BY+date+DESC%3B%0A%0A https://www.splitgraph.com/workspace/ddn?layout=hsplit&query... [3] https://www.splitgraph.com/docs/concepts/images https://www.splitgraph.com/docs/concepts/images