6 ms·

This is every data scientists’ dream.

by fabcomm 5y ago

villasv 5y ago
This is not new, though. So it may be a dream in the sense of people have been asleep?
- sneilan1 5y ago
  How long until Google shuts this service down?
- puzzledobserver 5y ago
  Why the condescension? Do you mean that Google has been offering this service for a while? Or do you mean that similar services have previously been offered by other organizations? In which case, perhaps you could link to them?
  - villasv 5y ago
    No condescension intended. Google has been offering this service for a while, at least since 2018 iirc.
prepend 5y ago
It’s ok, but surprisingly feature poor since they only index datasets with structured metadata. I kind of wish they would compile all their metadata into a structured mega-catalog and allowed searching by api. Or just dumped it out as a dataset itself.
- chatmasta 5y ago
  Then you'll love what we're doing at Splitgraph: https://www.splitgraph.com/connect https://www.splitgraph.com/connect As far as your SQL client is concerned, data.splitgraph.com:5432 is a giant Postgres database with ~40,000 tables in it. You can query and join across them using your existing tools. Behind the curtain, we'll forward your query to the upstream data source, translating it from SQL to whatever language it expects. (We can also ingest delta-compressed versioned snapshots).
  - prepend 5y ago
    That’s pretty cool, I’ll test it out when I get to a machine with a Postgres client. Do you translate to api calls and query in real-time? Or are you ingesting and archiving?
    - chatmasta 5y ago
      We support both! A Splitgraph repository can "mount" an external data source, and we'll proxy queries to it using a system based on Postgres Foreign Data Wrappers (FDW). But a repository can also contain any number of "data images," which are versioned snapshots of data roughly inspired by Docker images. You can define them with a declarative, Dockerfile-like syntax called a Splitfile, and you can rebuild them against upstream sources with caching semantics similar to "docker build." Our core philosophy has always been that it makes sense to start with data federation (live data), and then selectively warehouse/ingest only what you need (versioned data). We're shipping some upcoming features to support this workflow. You start by providing us (or your private deployment) a set of read-only credentials to any supported data source, which we then "mount" as a repository, making it discoverable in the catalog, and instantly queryable with all the other data on Splitgraph. If or when you decide that you want to warehouse this data, we'll make it easy for you to schedule a loading job to ingest it as a Splitgraph image. This way, you can query the live or versioned data in any repository, by simply changing the tag you use to address it. You can do all this stuff locally, btw – a decentralized workflow is fully supported, and you can push data between peers. The public Splitgraph.com happens to be a "super peer" with a data catalog, scalability features, etc. But if you just want to experiment on your own, you can try it in five minutes! I don't want to spam this thread too much, so I'll limit it to one link – maybe take a look at the Splitfile docs: https://www.splitgraph.com/docs/working-with-data/using-splitfiles https://www.splitgraph.com/docs/working-with-data/using-spli...
  - zX41ZdbW 5y ago
    What are the largest datasets in Splitgraph? Can I list the datasets sorted by size? We have the need for large public datasets for testing ClickHouse: https://clickhouse.tech/docs/en/getting-started/example-datasets/ https://clickhouse.tech/docs/en/getting-started/example-data...

This is every data scientists’ dream.

villasv 5y ago

This is not new, though. So it may be a dream in the sense of people have been asleep?

sneilan1 5y ago
How long until Google shuts this service down?
puzzledobserver 5y ago
Why the condescension? Do you mean that Google has been offering this service for a while? Or do you mean that similar services have previously been offered by other organizations? In which case, perhaps you could link to them?
- villasv 5y ago
  No condescension intended. Google has been offering this service for a while, at least since 2018 iirc.

prepend 5y ago

It’s ok, but surprisingly feature poor since they only index datasets with structured metadata. I kind of wish they would compile all their metadata into a structured mega-catalog and allowed searching by api. Or just dumped it out as a dataset itself.

chatmasta 5y ago
Then you'll love what we're doing at Splitgraph: https://www.splitgraph.com/connect https://www.splitgraph.com/connect As far as your SQL client is concerned, data.splitgraph.com:5432 is a giant Postgres database with ~40,000 tables in it. You can query and join across them using your existing tools. Behind the curtain, we'll forward your query to the upstream data source, translating it from SQL to whatever language it expects. (We can also ingest delta-compressed versioned snapshots).
- prepend 5y ago
  That’s pretty cool, I’ll test it out when I get to a machine with a Postgres client. Do you translate to api calls and query in real-time? Or are you ingesting and archiving?
  - chatmasta 5y ago
    We support both! A Splitgraph repository can "mount" an external data source, and we'll proxy queries to it using a system based on Postgres Foreign Data Wrappers (FDW). But a repository can also contain any number of "data images," which are versioned snapshots of data roughly inspired by Docker images. You can define them with a declarative, Dockerfile-like syntax called a Splitfile, and you can rebuild them against upstream sources with caching semantics similar to "docker build." Our core philosophy has always been that it makes sense to start with data federation (live data), and then selectively warehouse/ingest only what you need (versioned data). We're shipping some upcoming features to support this workflow. You start by providing us (or your private deployment) a set of read-only credentials to any supported data source, which we then "mount" as a repository, making it discoverable in the catalog, and instantly queryable with all the other data on Splitgraph. If or when you decide that you want to warehouse this data, we'll make it easy for you to schedule a loading job to ingest it as a Splitgraph image. This way, you can query the live or versioned data in any repository, by simply changing the tag you use to address it. You can do all this stuff locally, btw – a decentralized workflow is fully supported, and you can push data between peers. The public Splitgraph.com happens to be a "super peer" with a data catalog, scalability features, etc. But if you just want to experiment on your own, you can try it in five minutes! I don't want to spam this thread too much, so I'll limit it to one link – maybe take a look at the Splitfile docs: https://www.splitgraph.com/docs/working-with-data/using-splitfiles https://www.splitgraph.com/docs/working-with-data/using-spli...
- zX41ZdbW 5y ago
  What are the largest datasets in Splitgraph? Can I list the datasets sorted by size? We have the need for large public datasets for testing ClickHouse: https://clickhouse.tech/docs/en/getting-started/example-datasets/ https://clickhouse.tech/docs/en/getting-started/example-data...