3 ms·

Disclaimer: I am the co-founder of a competitor, Bruin (https://getbruin.com https://getbruin.com). We are exactly the kind of integrated platform Tristan is ta

Disclaimer: I am the co-founder of a competitor, Bruin (https://getbruin.com https://getbruin.com). We are exactly the kind of integrated platform Tristan is talking about. The article resonates with me a lot, and it is because we called this out months ago. I find the idea of Tristan walking back on the premise of MDS and claiming it to be not useful anymore funny because they were one of the main drivers of the term and the whole hype around it. He even acknowledges that they played ball with other companies in the space: > There was a lot of valuable co-marketing, partnership deals, co-sponsored events, and co-selling. This had real value for everyone involved—customers and vendors alike. Companies voluntarily integrated their products together, cross-promoted each other publicly, and built partnerships that made owning and operating these technologies far easier for customers. Sorry, but no, this didn't have value for the customers, only for the vendors. The customers were left alone by themselves to deal with all of this complexity, and the vendors made a lot of money off of that. They convinced companies that they needed a bunch of different tools to build a simple pipeline, and rode on the wave of huge valuations based on these same ideas that they are walking back on. The companies prefer integrated solutions now because they woke up. Instead of paying 150k/y each to Fivetran, dbt and whatnot, they realize that they are better of just hiring an engineer or two in the worst case. It is 2024, and none of these tools still properly talk to each other. Do you want to get an end-to-end lineage of your data? Good luck with that. How about quality? How about governance? Companies are left alone with this hype cycle. I'd claim that a significant part of the blame lies on the executives and leaders in the companies, who just jumped on the ship for the sake of building their CVs and skipped the critical thinking step. None of them seriously asked themselves the question of whether or not it makes sense. To be honest, I feel sorry for all the money spent on building solutions around all of these hype-driven products.

neeleshs 3y ago
Disclaimer : I'm the cofounder of Syncari. I agree 100%. The whole notion of modern data stack is a myopic , neither here nor there philosophy to address data in my opinion. Just take a look at the sheer number of tools one has to run - ETL(or more commonly, data dumpers), Wearhouse, transform tool,"reverse ETL", data quality tools, BI. And if you want some ML, that's whole another game. We also believe strongly in an integrated approach to this space, with a data model at the center of it all.
- gigatexal 3y ago
  Deleted
tomnipotent 3y ago
He's not "walking back on the premise of MDS", but acknowledges that the term "MDS" had been co-opted and lost its descriptive fidelity. > Sorry, but no, this didn't have value for the customers, only for the vendors Absolutely untrue. I've discovered multiple vendors over the last two decades because of co-marketing and partnership deals, and I know many others that have. My success rate with these vendors is no different from vendors I found myself or were referred to me by others. > The customers were left alone by themselves This is true with every vendor integration with every company on the globe. You either 1) have internal expertise to push through the pain, 2) hire support from the vendor itself (often X hours come baked into the deal), or 3) hire a consultant. This is as true for the MDS vendors as it is for Salesforce, AWS, or an ERP. It's true for your company, too. > They convinced companies that they needed a bunch of different tools Except you do? You need a database, you need something to extract data and move it locally, you need something to transform that data into something useful, and you need the ability to deliver that data to end users. This was true twenty years ago, true ten years ago, and still true today. The all-in-one vendors prior to MDS were horrible, and I'd rather cut off my arm then use something like Pentaho again. I'm also not convinced the current all-in-one vendors are any better. One of the advantages of the pick-your-stack MDS was that you could tailor the tools to you organizations specific needs, rather than have to fight against whatever your mega-vendor offers. I loved the fact that I could use open source for 90% of my stack, and then use PowerBI for the last 10%. Or I could be 100% FOSS and deliver data with Metabase. Or I could use Tableau for 50%, and FOSS for the other 50%. > they realize that they are better of just hiring an engineer or two in the worst case This simply isn't true for every size or class of company. Many businesses are better off with vendor-supported platforms than trying to build something bespoke, which can quickly become a liability with staff turnover or lack of strong internal technical leadership. > It is 2024, and none of these tools still properly talk to each other What does this even mean? How does a data extract tool "talk" to a transformation tool? Or "talk" to the database? I can see from the screenshot on your website that it offers a GUI with drag-and-drop - is that your definition of "talking"? Or are you referring to a tight integration that doesn't require additional work to glue them together? > How about quality? How about governance? These are management and organization issues, not tooling. I've seen amazing quality and governance with teams working in MSSQL/SSIS, and I've seen horrible quality and governance from teams using Hadoop/Spark. A tool can make a competent practitioners job easier, but it's not going to make an incompetent competent.
cgio 3y ago
As someone who has worked in this space, I would intuitively go with the decoupled tools every time. The toughest thing is matching a monolith of opinions with your requirements. Individual tools leave design space in their integration and thus allow proper engineering. That they allow it, does not mean it happens. If you go with a shopping bag architecture then you might be better off results wise with an integrated solution, but still, when the real engineers come along they’ll have an uphill battle to uproot an integrated solution vs redesigning the combination of individually good tools.
- CRConrad 3y ago
  > I would intuitively go with the decoupled tools every time. Seems to me that the problem is, each of these "decoupled tools" you mention isn't some specialised little awl or hacksaw ("the Unix philosophy..."), nor even a do-anything Swiss Army knife -- but a whole stack of cloud services, "integrations", and administration tools in its own right. A whole machine shop of machine shops. Building your own custom setup out of that is hard. You'll end up with something at least as unwieldy as a really big do-anything single-vendor mega-machine shop stack of cloud services, "integrations", and administration tools.
itsoktocry 3y ago
I don't even understand your argument. This is all just buy vs build. Pay Fivetran, or pay an engineer to build and maintain API connections. Pay dbt or use dbt core and orchestrate it yourself. There are tons of observability tools, lineage tools etc that you can buy. Or build them yourself. Or use AWS or GCP and their broad tooling. You'll never convince me that having more options is bad.

neeleshs 3y ago

Disclaimer : I'm the cofounder of Syncari. I agree 100%. The whole notion of modern data stack is a myopic , neither here nor there philosophy to address data in my opinion. Just take a look at the sheer number of tools one has to run - ETL(or more commonly, data dumpers), Wearhouse, transform tool,"reverse ETL", data quality tools, BI. And if you want some ML, that's whole another game. We also believe strongly in an integrated approach to this space, with a data model at the center of it all.

gigatexal 3y ago
Deleted

tomnipotent 3y ago

He's not "walking back on the premise of MDS", but acknowledges that the term "MDS" had been co-opted and lost its descriptive fidelity. > Sorry, but no, this didn't have value for the customers, only for the vendors Absolutely untrue. I've discovered multiple vendors over the last two decades because of co-marketing and partnership deals, and I know many others that have. My success rate with these vendors is no different from vendors I found myself or were referred to me by others. > The customers were left alone by themselves This is true with every vendor integration with every company on the globe. You either 1) have internal expertise to push through the pain, 2) hire support from the vendor itself (often X hours come baked into the deal), or 3) hire a consultant. This is as true for the MDS vendors as it is for Salesforce, AWS, or an ERP. It's true for your company, too. > They convinced companies that they needed a bunch of different tools Except you do? You need a database, you need something to extract data and move it locally, you need something to transform that data into something useful, and you need the ability to deliver that data to end users. This was true twenty years ago, true ten years ago, and still true today. The all-in-one vendors prior to MDS were horrible, and I'd rather cut off my arm then use something like Pentaho again. I'm also not convinced the current all-in-one vendors are any better. One of the advantages of the pick-your-stack MDS was that you could tailor the tools to you organizations specific needs, rather than have to fight against whatever your mega-vendor offers. I loved the fact that I could use open source for 90% of my stack, and then use PowerBI for the last 10%. Or I could be 100% FOSS and deliver data with Metabase. Or I could use Tableau for 50%, and FOSS for the other 50%. > they realize that they are better of just hiring an engineer or two in the worst case This simply isn't true for every size or class of company. Many businesses are better off with vendor-supported platforms than trying to build something bespoke, which can quickly become a liability with staff turnover or lack of strong internal technical leadership. > It is 2024, and none of these tools still properly talk to each other What does this even mean? How does a data extract tool "talk" to a transformation tool? Or "talk" to the database? I can see from the screenshot on your website that it offers a GUI with drag-and-drop - is that your definition of "talking"? Or are you referring to a tight integration that doesn't require additional work to glue them together? > How about quality? How about governance? These are management and organization issues, not tooling. I've seen amazing quality and governance with teams working in MSSQL/SSIS, and I've seen horrible quality and governance from teams using Hadoop/Spark. A tool can make a competent practitioners job easier, but it's not going to make an incompetent competent.

cgio 3y ago

As someone who has worked in this space, I would intuitively go with the decoupled tools every time. The toughest thing is matching a monolith of opinions with your requirements. Individual tools leave design space in their integration and thus allow proper engineering. That they allow it, does not mean it happens. If you go with a shopping bag architecture then you might be better off results wise with an integrated solution, but still, when the real engineers come along they’ll have an uphill battle to uproot an integrated solution vs redesigning the combination of individually good tools.

CRConrad 3y ago
> I would intuitively go with the decoupled tools every time. Seems to me that the problem is, each of these "decoupled tools" you mention isn't some specialised little awl or hacksaw ("the Unix philosophy..."), nor even a do-anything Swiss Army knife -- but a whole stack of cloud services, "integrations", and administration tools in its own right. A whole machine shop of machine shops. Building your own custom setup out of that is hard. You'll end up with something at least as unwieldy as a really big do-anything single-vendor mega-machine shop stack of cloud services, "integrations", and administration tools.

itsoktocry 3y ago

I don't even understand your argument. This is all just buy vs build. Pay Fivetran, or pay an engineer to build and maintain API connections. Pay dbt or use dbt core and orchestrate it yourself. There are tons of observability tools, lineage tools etc that you can buy. Or build them yourself. Or use AWS or GCP and their broad tooling. You'll never convince me that having more options is bad.