4 ms·

Thought it was some groundbreaking new database technology.... Speaking of, What would an interesting version of "database of databases" look like?

tyingq 6y ago
The closest I can think of would be the "Data Governance" space. Tools like Collibra: https://www.collibra.com/ https://www.collibra.com/ Screenshot that makes it easier to understand: https://www.collibra.com/wp-content/uploads/Blog-DataLineageDiagram-820x350.jpg https://www.collibra.com/wp-content/uploads/Blog-DataLineage...
indymike 6y ago
Apparently SQLite. Who would have thought?
LukeEF 6y ago
TerminusHub is a database filled with databases (for collaboration): https://terminusdb.com/hub/ https://terminusdb.com/hub/
ralusek 6y ago
Probably CosmosDB or MarkLogic.
chatmasta 6y ago
This is basically what we're building at Splitgraph [0]. We're calling it a "data delivery network." You connect to one SQL endpoint and can query (and join across) 40k+ different datasets. It's built on Postgres, and as far as your SQL client is concerned, it's talking to a Postgres database with 40k tables in it. Right now we forward queries to public data portals, but eventually you'll be able to connect live data sources to the DDN without writing any code. We want it to be as easy as configuring Cloudflare; you just upload a set of read-only credentials in the web UI and we take care of the rest. For more private use cases, we're planning to offer private deployments to AWS/GCP/Azure. Technically, this is database virtualization, which isn't really a new concept. We're implementing it as a database proxy, using PgBouncer instances to intercept queries and route them to Splitgraph engines. Within a Splitgraph engine (which is Postgres + some custom code), each "table" is either a "mounted" live database via a foreign database wrapper (FDW), or part of a point-in-time, versioned database snapshot called a "data image" that you can build with sgr. [0] https://www.splitgraph.com https://www.splitgraph.com

tyingq 6y ago

The closest I can think of would be the "Data Governance" space. Tools like Collibra: https://www.collibra.com/ https://www.collibra.com/ Screenshot that makes it easier to understand: https://www.collibra.com/wp-content/uploads/Blog-DataLineageDiagram-820x350.jpg https://www.collibra.com/wp-content/uploads/Blog-DataLineage...

indymike 6y ago

Apparently SQLite. Who would have thought?

LukeEF 6y ago

TerminusHub is a database filled with databases (for collaboration): https://terminusdb.com/hub/ https://terminusdb.com/hub/

ralusek 6y ago

Probably CosmosDB or MarkLogic.

chatmasta 6y ago

This is basically what we're building at Splitgraph [0]. We're calling it a "data delivery network." You connect to one SQL endpoint and can query (and join across) 40k+ different datasets. It's built on Postgres, and as far as your SQL client is concerned, it's talking to a Postgres database with 40k tables in it. Right now we forward queries to public data portals, but eventually you'll be able to connect live data sources to the DDN without writing any code. We want it to be as easy as configuring Cloudflare; you just upload a set of read-only credentials in the web UI and we take care of the rest. For more private use cases, we're planning to offer private deployments to AWS/GCP/Azure. Technically, this is database virtualization, which isn't really a new concept. We're implementing it as a database proxy, using PgBouncer instances to intercept queries and route them to Splitgraph engines. Within a Splitgraph engine (which is Postgres + some custom code), each "table" is either a "mounted" live database via a foreign database wrapper (FDW), or part of a point-in-time, versioned database snapshot called a "data image" that you can build with sgr. [0] https://www.splitgraph.com https://www.splitgraph.com