4 ms·

> Taxi shifts the obligation of describing how things stitch together from Consumers to Producers. Traditionally, it falls to consumers to work out how to do th

> Taxi shifts the obligation of describing how things stitch together from Consumers to Producers. Traditionally, it falls to consumers to work out how to do this. And, it’s an expensive question to answer… it involves tracking down API specs, reading docs, and building a mental model of how things hang together. I work on enterprise data platforms and this project sounds very much like the "data mesh" concept where responsibility for defining metadata/usability of datasets is supposed to rest with the teams producing the datasets rather than the teams consuming datasets. The same criticism probably applies: * Even though it is expensive/slow for a consumer to figure out how to use data, it is even more expensive/slow to have producers try to continuously anticipate the needs of all possible current and future consumers, rather than focus on their system and have actual consumers figure out how to use the subset of data they need as it changes. * Above a Dunbar-esque-sized organization you can no longer rely on empathy to encourage producers to expend extra effort on consumer's needs, so either monetary or disciplinary encouragement is needed, both of which raise costs even further. Is there a name for this kind of paradox? When an approach will work only in a company below a certain size/complexity/age, but that kind of company probably doesn't feel the pain enough to need the approach?

martypitt 3y ago
> it is even more expensive/slow to have producers try to continuously anticipate the needs of all possible current and future consumers I agree, but I don't think Taxi encourages that. (I sure hope it doesn't). All that's really happening is defining a system-agnostic set of terms, embedding those in producer schemas, and then letting consumers using those same terms to ask for data. Those terms can be used to automate integration, and in many (but not all) cases, alleviate the need for low-value plumbing code. Semantic Metadata is a growing trend in the data ecosystem (cube.dev is doing a nice job here too). The Agile folks have had Ubiquitous Language as a first class concept since day dot. > an approach will work only in a company below a certain size/complexity/age I think that GraphQL federation is an example of mid-to-large companies trying to solve the federation problem, suggesting they both feel the pain, and are looking for solutions.
- mst 3y ago
  The goals of Taxi feel at least somewhat like 'creating a domain driven design style Ubiquitous Language to describe data and relationships' and the deliberate not-forcing-a-single-global-thing seems like it'll at least make it harder for enterprise dysfunction to bend it into having the failure modes the grandparent describes.
- cgio 3y ago
  What happens when a consumer does not understand the producer’s domain and misuses the data? As we load metadata with semantics of their own, we just kick the can down the road. Metadata is itself semantics about data. In many regulated environments, as producer you are responsible for downstream aggregations etc. I think the idea works great in prototyping, but similar to data mesh, ownership is poorly defined in the presence of aggregate domains and it’s more tuned to operational point to point interfaces.
  - martypitt 3y ago
    > What happens when a consumer does not understand the producer’s domain and misuses the data? That's a problem with or without Taxi/Orbital. Today, it's generally left to consumers to pick the fields that look right, and hope they get it right. That's a risky approach. And, by leaving it to consumers, you run the risk on every new integration. I think by asking producers to annotate their attributes with a strongly defined semantic contract, you reduce the risk of consumers getting it wrong. Producers understand their data much better than consumers, so are better informed on how to map their attributes to a set of semantic contracts that consumers can leverage.
seanhunter 3y ago
> Even though it is expensive/slow for a consumer to figure out how to use data, it is even more expensive/slow to have producers try to continuously anticipate the needs of all possible current and future consumers, rather than focus on their system and have actual consumers figure out how to use the subset of data they need as it changes. I agree completely on the speed impact but I would go further and say this is basically impossible. I have worked on data everywhere from small to massive enterprises and the only model I have ever see work at a large enterprise is for source systems to just produce data in a raw-ish format that comes out of their system and leave it to the people who need to consume that data to figure out how to massage it into whatever format they need. The reasons I think it's not possible for upstream to anticipate are: 1) Analytic usecases multiply as systems do and as users start consuming data. As soon as you start using data to address problems in the org, you see more and more questions you would like to use data to address. There is no way for a producer system to forsee all the possible analytic questions people would like to answer using their data, especially as data analysis is a creative endeavour and peoples' imagination is pretty amazing (especially taken collectively). 2) As data sources multiply, the analytical possibilities of combining these data sources explode quadratically. There is no way for any single upstream system to anticipate how this will go. What upstream systems need to do is produce all the data they can from their system and let users do what they do when combining. You can make some sensible guidelines available (eg "document what the columns mean", "keep up to date estimates of volumes to expect", "have a data sample/staging system/test instance for people to integrate against", "put your goddam timestamps into UTC wherever possible" etc) that aren't too onerous for upstream maintainers to follow if they want to be good citizens.

martypitt 3y ago

> it is even more expensive/slow to have producers try to continuously anticipate the needs of all possible current and future consumers I agree, but I don't think Taxi encourages that. (I sure hope it doesn't). All that's really happening is defining a system-agnostic set of terms, embedding those in producer schemas, and then letting consumers using those same terms to ask for data. Those terms can be used to automate integration, and in many (but not all) cases, alleviate the need for low-value plumbing code. Semantic Metadata is a growing trend in the data ecosystem (cube.dev is doing a nice job here too). The Agile folks have had Ubiquitous Language as a first class concept since day dot. > an approach will work only in a company below a certain size/complexity/age I think that GraphQL federation is an example of mid-to-large companies trying to solve the federation problem, suggesting they both feel the pain, and are looking for solutions.

mst 3y ago
The goals of Taxi feel at least somewhat like 'creating a domain driven design style Ubiquitous Language to describe data and relationships' and the deliberate not-forcing-a-single-global-thing seems like it'll at least make it harder for enterprise dysfunction to bend it into having the failure modes the grandparent describes.
cgio 3y ago
What happens when a consumer does not understand the producer’s domain and misuses the data? As we load metadata with semantics of their own, we just kick the can down the road. Metadata is itself semantics about data. In many regulated environments, as producer you are responsible for downstream aggregations etc. I think the idea works great in prototyping, but similar to data mesh, ownership is poorly defined in the presence of aggregate domains and it’s more tuned to operational point to point interfaces.
- martypitt 3y ago
  > What happens when a consumer does not understand the producer’s domain and misuses the data? That's a problem with or without Taxi/Orbital. Today, it's generally left to consumers to pick the fields that look right, and hope they get it right. That's a risky approach. And, by leaving it to consumers, you run the risk on every new integration. I think by asking producers to annotate their attributes with a strongly defined semantic contract, you reduce the risk of consumers getting it wrong. Producers understand their data much better than consumers, so are better informed on how to map their attributes to a set of semantic contracts that consumers can leverage.

seanhunter 3y ago

> Even though it is expensive/slow for a consumer to figure out how to use data, it is even more expensive/slow to have producers try to continuously anticipate the needs of all possible current and future consumers, rather than focus on their system and have actual consumers figure out how to use the subset of data they need as it changes. I agree completely on the speed impact but I would go further and say this is basically impossible. I have worked on data everywhere from small to massive enterprises and the only model I have ever see work at a large enterprise is for source systems to just produce data in a raw-ish format that comes out of their system and leave it to the people who need to consume that data to figure out how to massage it into whatever format they need. The reasons I think it's not possible for upstream to anticipate are: 1) Analytic usecases multiply as systems do and as users start consuming data. As soon as you start using data to address problems in the org, you see more and more questions you would like to use data to address. There is no way for a producer system to forsee all the possible analytic questions people would like to answer using their data, especially as data analysis is a creative endeavour and peoples' imagination is pretty amazing (especially taken collectively). 2) As data sources multiply, the analytical possibilities of combining these data sources explode quadratically. There is no way for any single upstream system to anticipate how this will go. What upstream systems need to do is produce all the data they can from their system and let users do what they do when combining. You can make some sensible guidelines available (eg "document what the columns mean", "keep up to date estimates of volumes to expect", "have a data sample/staging system/test instance for people to integrate against", "put your goddam timestamps into UTC wherever possible" etc) that aren't too onerous for upstream maintainers to follow if they want to be good citizens.