3 ms·
You have cut straight to the heart of a pretty interesting problem that we are still thinking through. You are exactly right, right now this is pure front-end.
by amcaskill 5y ago
You have cut straight to the heart of a pretty interesting problem that we are still thinking through.
You are exactly right, right now this is pure front-end. That's intentional, and there's a lot that we like about that approach, especially in light of the success of dbt.
We think dbt fundamentally changes what is needed from a BI tool and that vendors who are maintaining really heavy built-in data transformation layers will basically be wasting resources over the coming years. The approach of modelling in your data warehouse is just so much more sensible, that we think it's a really a good thing to bet on.
That said, having some form of metric modelling in your BI tool is really nice -- it helps you keep your queries dry, and makes it simpler to roll out changes. If we were to build something here, I think it would be very lightweight -- a config that basically let you define re-usable sql snippets, and maybe some constraints on them.
On the other hand, there are A LOT of startups building metrics layers, which look great. Usually these expose an API endpoint, and some sort SQL interface. We'd be just as happy to plug into one of those SQL interfaces and call it a day. I just wish one of those was open source, since the metrics layer is such a choke-hold on your data operation.
Maybe someone will build the 'dbt of metrics layers'. That would be great for the ecosystem. Maybe dbt will do it themselves. I think there's probably something interesting they could do there by treating stored procedures as materialization targets.
- buremba 5y agoI believe that we’re working on a similar product that you described. Our metrics layer, metriql, extends dbt for metrics definitions and provides an open-source CLI that serves your metrics to the data tools. We have REST API and use Trino protocol so in case you support Trino (formerly Presto) you already support metriql. :) Here is the link: https://metriql.com https://metriql.com
- amcaskill 5y agoLook at that! Yeah, I love it. That's exactly what I was thinking of. Trino is on the to-do list!
- mariushn 5y agoThanks for posting! Is metriql entirely open source, or just the CLI? How do you plan to make revenue?
- buremba 5y agoWe use BSL license and metriql is free with a single database target. If you want to connect multiple dbt projects in a single deployment, you need to go through the sales cycle. We work with ETL vendors that use metriql to make revenue with our BI tool integrations so we picked BSL license to be able to structure our business model in a way that you should be required to pay only if you're reselling metriql to your customers. We have plans to build a managed Cloud version as well. You can find the license here: https://github.com/metriql/metriql https://github.com/metriql/metriql
- sails 5y ago>I think it would be very lightweight Very interesting, that sounds great. I like the idea of keeping minimal business logic in the BI tool, but it is necessary. And yes, lots of metric layer startups, taking advantage of that does make sense. Will take a closer look, thanks!
- amitkb24 5y agoWe are one of the startups working on the metrics layer 'Transform': https://transform.co/ https://transform.co/ The founding team behind Transform worked on the early implementation of Airbnb's metric store (now popularly known as Minerva!). You can also check out this podcast on Data Engineering that dives into the Metrics layer: https://www.dataengineeringpodcast.com/transform-co-metrics-layer-episode-206/ https://www.dataengineeringpodcast.com/transform-co-metrics-...
- sails 5y agoGreat, will have a look, I've been meaning to do so.
- innerzeal 5y agoHave you looked at umetric from Uber? Does that serve the purpose?
- amcaskill 5y agoJust checked it out briefly, that looks like it's aiming to solve the same problem -- do you know if it's open source?