5 ms·

you just took a feature of postgresql and mentioned it in your article to sell your product. you didnt answer your own question. is it a secret weapon?

by omani 6y ago

chatmasta 6y ago
Splitgraph co-founder here. Splitgraph is based on PostgreSQL, so naturally all of its features are part of Splitgraph as well. We want to add value to a battle-tested RDBMS rather than inventing a database from the ground up. In particular, we really like the feature of foreign data wrappers, and it forms the basis of one of our core abstractions ("mounting" upstream data) [0]. We've added a lot of scaffolding to make writing and using FDWs easier with a Splitgraph engine (which is just Postgres with the Splitgraph library loaded into it). You can also IMPORT from upstream sources using Splitfiles, to pack the data into a versioned “image” (analogous to Dockerfiles and docker images). As for whether they're a secret weapon? Well, they're not so secret, but it does seem like they're underutilized considering the power they grant. We certainly like them. [0] I wrote a post today about how we use a Socrata FDW to "mount" 40k+ government datasets, making them all available via SQL: https://www.splitgraph.com/blog/40k-sql-datasets https://www.splitgraph.com/blog/40k-sql-datasets
- sradman 6y ago
  > As for whether they're a secret weapon? Well, they're not so secret, but it does seem like they're underutilized considering the power they grant. We certainly like them. They are indeed powerful and underutilized. They also go by different names in other RDBMSes (proxy tables, linked servers, remote tables, virtual tables, etc.). If I remember correctly, SQL Anywhere 6 released in 1998 was the first rich implementation [1] of "proxy tables" to "remote servers" and Microsoft soon followed with "Linked Servers". The embedded database libraries (Apache Derby and SQLite) added rich APIs/SPIs to create virtual table plugins like SQLite's FTS. Using remote tables for federated queries across heterogenous data sources, and for access to non-relational data sources, is a well established technique and a powerful "weapon" for those who know how to wield it. It might not be PostgreSQL's secret weapon but it may very well be Splitgraph's. [1] http://dcx.sap.com/index.html#1001/en/dbwnen10/wn-remote-new601.html http://dcx.sap.com/index.html#1001/en/dbwnen10/wn-remote-new...
  - jungturk 6y ago
    Right. In AWS Redshift-land, this feature is called Spectrum, and allows you to use the redshift query engine & its SQL support to access external data (say, parquet files on S3).
  - ako 6y ago
    I think oracle had database links before 1998 which enables you to have proxy tables for tables in other oracle databases.
    - sradman 6y ago
      It looks like Oracle had CREATE DATABASE LINK since at least 7.3 [1] in 1996 (probably earlier too but I couldn't find when it was introduced). It wasn't Oracle-only either: > However, to access non-Oracle systems you must use Oracle Heterogeneous Services. [1] https://docs.oracle.com/cd/A57673_01/DOC/server/doc/SQL73/ch4a.htm#toc087 https://docs.oracle.com/cd/A57673_01/DOC/server/doc/SQL73/ch...

you just took a feature of postgresql and mentioned it in your article to sell your product. you didnt answer your own question. is it a secret weapon?

by omani 6y ago

you just took a feature of postgresql and mentioned it in your article to sell your product. you didnt answer your own question. is it a secret weapon?

chatmasta 6y ago

Splitgraph co-founder here. Splitgraph is based on PostgreSQL, so naturally all of its features are part of Splitgraph as well. We want to add value to a battle-tested RDBMS rather than inventing a database from the ground up. In particular, we really like the feature of foreign data wrappers, and it forms the basis of one of our core abstractions ("mounting" upstream data) [0]. We've added a lot of scaffolding to make writing and using FDWs easier with a Splitgraph engine (which is just Postgres with the Splitgraph library loaded into it). You can also IMPORT from upstream sources using Splitfiles, to pack the data into a versioned “image” (analogous to Dockerfiles and docker images). As for whether they're a secret weapon? Well, they're not so secret, but it does seem like they're underutilized considering the power they grant. We certainly like them. [0] I wrote a post today about how we use a Socrata FDW to "mount" 40k+ government datasets, making them all available via SQL: https://www.splitgraph.com/blog/40k-sql-datasets https://www.splitgraph.com/blog/40k-sql-datasets

sradman 6y ago
> As for whether they're a secret weapon? Well, they're not so secret, but it does seem like they're underutilized considering the power they grant. We certainly like them. They are indeed powerful and underutilized. They also go by different names in other RDBMSes (proxy tables, linked servers, remote tables, virtual tables, etc.). If I remember correctly, SQL Anywhere 6 released in 1998 was the first rich implementation [1] of "proxy tables" to "remote servers" and Microsoft soon followed with "Linked Servers". The embedded database libraries (Apache Derby and SQLite) added rich APIs/SPIs to create virtual table plugins like SQLite's FTS. Using remote tables for federated queries across heterogenous data sources, and for access to non-relational data sources, is a well established technique and a powerful "weapon" for those who know how to wield it. It might not be PostgreSQL's secret weapon but it may very well be Splitgraph's. [1] http://dcx.sap.com/index.html#1001/en/dbwnen10/wn-remote-new601.html http://dcx.sap.com/index.html#1001/en/dbwnen10/wn-remote-new...
- jungturk 6y ago
  Right. In AWS Redshift-land, this feature is called Spectrum, and allows you to use the redshift query engine & its SQL support to access external data (say, parquet files on S3).
- ako 6y ago
  I think oracle had database links before 1998 which enables you to have proxy tables for tables in other oracle databases.
  - sradman 6y ago
    It looks like Oracle had CREATE DATABASE LINK since at least 7.3 [1] in 1996 (probably earlier too but I couldn't find when it was introduced). It wasn't Oracle-only either: > However, to access non-Oracle systems you must use Oracle Heterogeneous Services. [1] https://docs.oracle.com/cd/A57673_01/DOC/server/doc/SQL73/ch4a.htm#toc087 https://docs.oracle.com/cd/A57673_01/DOC/server/doc/SQL73/ch...