3 ms·

Your final example is very like the data mesh ideas coming out of thought works and elsewhere (https://martinfowler.com/articles/data-mesh-principles.html https

by LukeEF 6y ago

Your final example is very like the data mesh ideas coming out of thought works and elsewhere (https://martinfowler.com/articles/data-mesh-principles.html https://martinfowler.com/articles/data-mesh-principles.html). Data products being owned by data producers and common data models ONLY when required. It is as much organizational model as technology, but I don't really think this maps to SQL tables. You probably want to look at a versioned knowledge graph of some kind. Downside is no SQL, upside is flexibility and speed. (disclaimer - I work over at TerminusDB, the graph version of Dolt)

sixdimensional 6y ago
You're absolutely right that what I'm saying is like data mesh. The data mesh ideas aren't necessarily new - as this idea that data has been managed/owned by what some would call "federated teams" (optimistic view) and others might call "silos" (pessimistic view) isn't really new - especially in certain industries. Certainly, the current energy and focus on building collaborative organizational service models, process, standards, etc. and standardizing this around the concept of data mesh is "fresh" and very much on the tip of many people's minds. I worked at a company that had built a tool which was a data federation engine, and we dabbled with the concepts of "data fabric", "mesh", "network" etc. but we were more talking about tech/implementation vs. architecture and organizational structure. Still, we thought the right tech could enable this architecture/organizational structure and that it wasn't something totally figured out. Thanks for sharing and mentioning it, and also Terminus. Certainly, I agree that knowledge graph is another approach and one that can also fit very well with the idea of linking data that is both common and specific to domains. However, I've rarely seen graph databases that had the kind of versioning/time travel that a temporal database (such as relational ones) had built in natively (or anything like "diff"). I don't agree this can't map to SQL tables.. tables and relations (tuples) are just a form of data. People do collaborative data management using relational-based master data management systems already today and have been for many, many years. That is not to say that graphs aren't a good match for storing common/reference datasets or linking data together. Given that graphs can be considered a superset of relational model, I don't see any reason that you can't do the same thing in graph databases, just that doing diffs and enabling the same kind of workflow might be more difficult in a graph data model - but definitely not impossible. Lest we forget things like "Linked Data" [1], triple stores, etc. also that have been trying to get at linked data across the entire Internet. However, I never saw collaborative data management as part of that vision, it was mostly for querying. [1] https://en.wikipedia.org/wiki/Linked_data https://en.wikipedia.org/wiki/Linked_data

by LukeEF 6y ago

sixdimensional 6y ago

You're absolutely right that what I'm saying is like data mesh. The data mesh ideas aren't necessarily new - as this idea that data has been managed/owned by what some would call "federated teams" (optimistic view) and others might call "silos" (pessimistic view) isn't really new - especially in certain industries. Certainly, the current energy and focus on building collaborative organizational service models, process, standards, etc. and standardizing this around the concept of data mesh is "fresh" and very much on the tip of many people's minds. I worked at a company that had built a tool which was a data federation engine, and we dabbled with the concepts of "data fabric", "mesh", "network" etc. but we were more talking about tech/implementation vs. architecture and organizational structure. Still, we thought the right tech could enable this architecture/organizational structure and that it wasn't something totally figured out. Thanks for sharing and mentioning it, and also Terminus. Certainly, I agree that knowledge graph is another approach and one that can also fit very well with the idea of linking data that is both common and specific to domains. However, I've rarely seen graph databases that had the kind of versioning/time travel that a temporal database (such as relational ones) had built in natively (or anything like "diff"). I don't agree this can't map to SQL tables.. tables and relations (tuples) are just a form of data. People do collaborative data management using relational-based master data management systems already today and have been for many, many years. That is not to say that graphs aren't a good match for storing common/reference datasets or linking data together. Given that graphs can be considered a superset of relational model, I don't see any reason that you can't do the same thing in graph databases, just that doing diffs and enabling the same kind of workflow might be more difficult in a graph data model - but definitely not impossible. Lest we forget things like "Linked Data" [1], triple stores, etc. also that have been trying to get at linked data across the entire Internet. However, I never saw collaborative data management as part of that vision, it was mostly for querying. [1] https://en.wikipedia.org/wiki/Linked_data https://en.wikipedia.org/wiki/Linked_data