3 ms·

That + a central data platform team that provides infra, quality monitors, data lineage and catalogue capabilities + a central team that provides guidelines on

by skrrr 5y ago

That + a central data platform team that provides infra, quality monitors, data lineage and catalogue capabilities + a central team that provides guidelines on SLAs, metadata standards etc. Sounds good in theory, I am eager to see how it fails in practice

shoo 5y ago
many points of internet karma (and perhaps a profitable career as a consultant) awaits anyone who spills the beans on how their grand data mesh rearchitecture actually turned out a few years down the track, and if the exciting new problems caused by the data mesh were easier or harder to deal with than the boring old problems caused by the organisational and IT architecture it replaced.
- Foobar8568 5y ago
  You mean like the shit show that is Data Vault? https://danlinstedt.com/allposts/datavaultcat/datavault-issues-resolved/ https://danlinstedt.com/allposts/datavaultcat/datavault-issu... >>data vault 2.0 brings with it methodology, architecture, modeling, and implementation – best practices, standards, automation and more. the ability to encompass and leverage disciplined agile delivery, and sei/cmmi, six sigma, lean initiatives, cycle time reduction, and proper build practices lead us to one day sprint cycles. And let's not forget that shit show of a "book" https://www.amazon.com/Data-Architecture-Primer-Scientist-Warehouse-ebook/dp/B00QQ6B8X0/ https://www.amazon.com/Data-Architecture-Primer-Scientist-Wa...
KmVFIz 5y ago
I can chime in as part of the central team for SLAs, etc. We offer a platform to produce datasets given some inputs, SQL, and pushes to downstream systems. Standardized jobs are ran after the user’s SQL to produce standardized outputs. It works well, but has many issues too. User’s SQLs and input data can differ, often in unpredictable ways, because they bring their own and expect the central team to handle the rest. Those edge cases break the standardization rules, fails the workflow, confuses the user because the platform is a black box, and they ask about changing it or adding a new feature. Now your standardization asks are bottle-necked by this central team, and the options are: - to wait for the central team to fix/improve it - find some hack around the platform - don’t use the platform and its associated toolings, so you build it yourself and have another disjoint system for a specific use case - central team might build a feature that one team asked for 1 years ago, but now nobody needs it anymore and nobody knows why it’s in code. Repeat many many times for various asks over the years and your code base is likely a foreign mess. - give your resources/funding to the central team to prioritize your ask. When built and a few years later, the central team owns something they themselves never wanted.

by skrrr 5y ago

shoo 5y ago

many points of internet karma (and perhaps a profitable career as a consultant) awaits anyone who spills the beans on how their grand data mesh rearchitecture actually turned out a few years down the track, and if the exciting new problems caused by the data mesh were easier or harder to deal with than the boring old problems caused by the organisational and IT architecture it replaced.

Foobar8568 5y ago
You mean like the shit show that is Data Vault? https://danlinstedt.com/allposts/datavaultcat/datavault-issues-resolved/ https://danlinstedt.com/allposts/datavaultcat/datavault-issu... >>data vault 2.0 brings with it methodology, architecture, modeling, and implementation – best practices, standards, automation and more. the ability to encompass and leverage disciplined agile delivery, and sei/cmmi, six sigma, lean initiatives, cycle time reduction, and proper build practices lead us to one day sprint cycles. And let's not forget that shit show of a "book" https://www.amazon.com/Data-Architecture-Primer-Scientist-Warehouse-ebook/dp/B00QQ6B8X0/ https://www.amazon.com/Data-Architecture-Primer-Scientist-Wa...

KmVFIz 5y ago

I can chime in as part of the central team for SLAs, etc. We offer a platform to produce datasets given some inputs, SQL, and pushes to downstream systems. Standardized jobs are ran after the user’s SQL to produce standardized outputs. It works well, but has many issues too. User’s SQLs and input data can differ, often in unpredictable ways, because they bring their own and expect the central team to handle the rest. Those edge cases break the standardization rules, fails the workflow, confuses the user because the platform is a black box, and they ask about changing it or adding a new feature. Now your standardization asks are bottle-necked by this central team, and the options are: - to wait for the central team to fix/improve it - find some hack around the platform - don’t use the platform and its associated toolings, so you build it yourself and have another disjoint system for a specific use case - central team might build a feature that one team asked for 1 years ago, but now nobody needs it anymore and nobody knows why it’s in code. Repeat many many times for various asks over the years and your code base is likely a foreign mess. - give your resources/funding to the central team to prioritize your ask. When built and a few years later, the central team owns something they themselves never wanted.