3 ms·

> decentralization; it's the only real way to deal with increasing complexity and sprawl Decentralization can help with sprawl (aka scale) but it makes softwar

by sa46 6y ago

> decentralization; it's the only real way to deal with increasing complexity and sprawl Decentralization can help with sprawl (aka scale) but it makes software more complex so it’s not a way to tackle complexity. Decentralization is only tolerated because, at some point, it’s the only method left to scale. As an example, distributed databases are significantly more complex to build than a single node database. You’ve also ignored all of the downsides of using your OLTP database for OLAP queries. The most important is that adhoc queries can impact production. In my experience, that cost dwarfs the costs of maintaining an ETL pipeline as soon as you have your first outage. Another downside is that with a federated, pull based model you need to keep the mapping (FDW definition) up to date. In essence, the maintenance cost moves from an ETL pipeline to an FDW definition.

chatmasta 6y ago
Re: ad-hoc queries impacting production, you can mitigate this by pointing your data mesh to a read replica of the prod DB (which you can colocate with it to avoid bandwidth costs). Admittedly at that point you might be approaching a warehousing use case. Re: maintaining the mapping. Sure that's a tradeoff. But I'd rather manage declarative FDW definitions than a programmatic ETL pipeline. Yes, handling schema changes is non-trivial (a naive starting point is periodic introspection of the remote table), but that's true in an ETL pipeline too. As you say, you're moving the cost from the ETL pipeline to the FDW definition. But what if we can lower the cost of the "FDW definition" (a.k.a. setting up and maintaining the data mesh)? From our perspective, at least with the product we're building, we think we can make a data mesh that's low maintenance enough that it makes sense as a first default before ETL. But the two aren't mutually exclusive. A data mesh can connect multiple warehouses just as easily as it can your production databases. For example, you could move data (ETL) from your data sources to your Snowflake and BigQuery warehouses, and then connect them both to your data mesh. Also, a mesh can add value as a unified access layer that proxies queries and applies data governance rules (e.g. sharing data, access control, column masking, query whitelisting, rewriting, rate limiting, auditing, firewalling, etc.). It would be hard to do this in your warehouse, because as soon as you have another one, or there's some data that you can't ETL into it, you lose any benefits of its access layer. (e.g. Maybe you have two warehouses because you haven't fully migrated to one, or for cost reasons like storing FireBase clickstream data in BigTable that you can't move to Snowflake with the rest of your data. Now you need to maintain two sets of accounts and ACLs for each warehouse.). (And for what it's worth -- this is a bit off topic of the current thread -- we do recognize there is often a use case for warehousing when the tradeoffs of federation aren't worth it. We can actually act as a warehouse too, which makes it easy to start with a data mesh but have the option to bail out to warehousing for the specific data assets that need it. Ideally, you can just click a button that says something like "snapshot data" or "schedule query" or "replicate data source." To ingest and store the data, we can leverage the concept we introduced of "data images" for storing versioned snapshots of data in delta-compressed, content-addressable chunks of hot-swappable cstore_fdw files. You can create data images with a declarative, Dockerfile-like syntax called Splitfiles. They look something like FROM upstream:tag IMPORT { SELECT x, y FROM ... }). You get full provenance and can `sgr rebuild` an updated image only when its upstream changes, just like Docker.)
- xyzzy_plugh 6y ago
  > Also, a mesh can add value as a unified access layer that proxies queries and applies data governance rules (e.g. sharing data, access control, column masking, query whitelisting, rewriting, rate limiting, auditing, firewalling, etc.). It would be hard to do this in your warehouse, because as soon as you have another one, or there's some data that you can't ETL into it, you lose any benefits of its access layer. (e.g. Maybe you have two warehouses because you haven't fully migrated to one, or for cost reasons like storing FireBase clickstream data in BigTable that you can't move to Snowflake with the rest of your data. Now you need to maintain two sets of accounts and ACLs for each warehouse.). I have been actively championing this point of view for years but... I'm curious, are any paying customers actually interested in this? I don't know how you can market this successfully.
  - chatmasta 6y ago
    We're really early and have done very little marketing outside of blog posts this summer, but we do have some paying customers. Their use cases are varying, since Splitgraph can serve a number of different functions (data catalog, data lake/mesh, query proxying/caching, versioned data snapshots, etc.). The common selling point across customers seems to be the integration of the access layer (Postgres proxy) and discovery layer (data catalog). Their data storage/access patterns are different. e.g. We have one who uses it as a data lake for periodically querying CSV files in a 20TB S3 bucket. But then another is more interested in the versioning features; they're building data images in Prefect workflows, tagging them with Semantic Versioning syntax, and pushing them to their Splitgraph deployment. For all of them, the discovery features of the catalog are important. We still have a lot of work to do on the ACL / governance features, but we've had prospects specifically ask about them as priorities. IMO, the best argument for our thesis is similar to GitLab's (unify the DevOps lifecycle). Even though GitLab didn't have the best-in-class version control or CI on their own, by offering them together they made it worth using GitLab. We want to unify the data stack by integrating the discovery, access and (optionally) storage layers. Even if each layer has some tradeoffs, the benefit of integrating them has a multiplicative effect on the platform itself.

by sa46 6y ago

chatmasta 6y ago

Re: ad-hoc queries impacting production, you can mitigate this by pointing your data mesh to a read replica of the prod DB (which you can colocate with it to avoid bandwidth costs). Admittedly at that point you might be approaching a warehousing use case. Re: maintaining the mapping. Sure that's a tradeoff. But I'd rather manage declarative FDW definitions than a programmatic ETL pipeline. Yes, handling schema changes is non-trivial (a naive starting point is periodic introspection of the remote table), but that's true in an ETL pipeline too. As you say, you're moving the cost from the ETL pipeline to the FDW definition. But what if we can lower the cost of the "FDW definition" (a.k.a. setting up and maintaining the data mesh)? From our perspective, at least with the product we're building, we think we can make a data mesh that's low maintenance enough that it makes sense as a first default before ETL. But the two aren't mutually exclusive. A data mesh can connect multiple warehouses just as easily as it can your production databases. For example, you could move data (ETL) from your data sources to your Snowflake and BigQuery warehouses, and then connect them both to your data mesh. Also, a mesh can add value as a unified access layer that proxies queries and applies data governance rules (e.g. sharing data, access control, column masking, query whitelisting, rewriting, rate limiting, auditing, firewalling, etc.). It would be hard to do this in your warehouse, because as soon as you have another one, or there's some data that you can't ETL into it, you lose any benefits of its access layer. (e.g. Maybe you have two warehouses because you haven't fully migrated to one, or for cost reasons like storing FireBase clickstream data in BigTable that you can't move to Snowflake with the rest of your data. Now you need to maintain two sets of accounts and ACLs for each warehouse.). (And for what it's worth -- this is a bit off topic of the current thread -- we do recognize there is often a use case for warehousing when the tradeoffs of federation aren't worth it. We can actually act as a warehouse too, which makes it easy to start with a data mesh but have the option to bail out to warehousing for the specific data assets that need it. Ideally, you can just click a button that says something like "snapshot data" or "schedule query" or "replicate data source." To ingest and store the data, we can leverage the concept we introduced of "data images" for storing versioned snapshots of data in delta-compressed, content-addressable chunks of hot-swappable cstore_fdw files. You can create data images with a declarative, Dockerfile-like syntax called Splitfiles. They look something like FROM upstream:tag IMPORT { SELECT x, y FROM ... }). You get full provenance and can `sgr rebuild` an updated image only when its upstream changes, just like Docker.)

xyzzy_plugh 6y ago
> Also, a mesh can add value as a unified access layer that proxies queries and applies data governance rules (e.g. sharing data, access control, column masking, query whitelisting, rewriting, rate limiting, auditing, firewalling, etc.). It would be hard to do this in your warehouse, because as soon as you have another one, or there's some data that you can't ETL into it, you lose any benefits of its access layer. (e.g. Maybe you have two warehouses because you haven't fully migrated to one, or for cost reasons like storing FireBase clickstream data in BigTable that you can't move to Snowflake with the rest of your data. Now you need to maintain two sets of accounts and ACLs for each warehouse.). I have been actively championing this point of view for years but... I'm curious, are any paying customers actually interested in this? I don't know how you can market this successfully.
- chatmasta 6y ago
  We're really early and have done very little marketing outside of blog posts this summer, but we do have some paying customers. Their use cases are varying, since Splitgraph can serve a number of different functions (data catalog, data lake/mesh, query proxying/caching, versioned data snapshots, etc.). The common selling point across customers seems to be the integration of the access layer (Postgres proxy) and discovery layer (data catalog). Their data storage/access patterns are different. e.g. We have one who uses it as a data lake for periodically querying CSV files in a 20TB S3 bucket. But then another is more interested in the versioning features; they're building data images in Prefect workflows, tagging them with Semantic Versioning syntax, and pushing them to their Splitgraph deployment. For all of them, the discovery features of the catalog are important. We still have a lot of work to do on the ACL / governance features, but we've had prospects specifically ask about them as priorities. IMO, the best argument for our thesis is similar to GitLab's (unify the DevOps lifecycle). Even though GitLab didn't have the best-in-class version control or CI on their own, by offering them together they made it worth using GitLab. We want to unify the data stack by integrating the discovery, access and (optionally) storage layers. Even if each layer has some tradeoffs, the benefit of integrating them has a multiplicative effect on the platform itself.