5 ms·

"We model data according to rigorous frameworks like Kimball or Inmon because we must regularly construct OLAP cubes for our analyses." Actually, no. We model

"We model data according to rigorous frameworks like Kimball or Inmon because we must regularly construct OLAP cubes for our analyses." Actually, no. We model our data this way so it can be used for business decisions. It doesn't take long for any entity of any scale to discover that the logging and eventing done for heartbeat status, debugging, and scaling is just different than what you need to make a variety of business decisions. You can solve with more events at different scales (button clicks nested in screen views) or pick events or event rolkups that appear to be clean business stages ("completed checkout") but still, your finance team, marketing group, all have different needs. So, you decide to have some core shared metrics, derived and defined, and make them usable by everyone. Folks agree on the defns, and due to ease and trust, you see more data supporting more decisions. You discover that some folks are doing 10 table joins to get an answer; it's fast but difficult to extend for new questions. You decide to build a view that solves some of these pains, and refactoring to allow a better time dimension. Your version links with the metrics you created, and the resulting queries shed tons of CTEs while becoming readable to the average user. And now, you have some ELT pipelines, some event transforms that result in counts and filters that map nicely to your business needs but still allow you to get atomic raws, and you and your teams start to trust in consistent results. Your metrics are mostly clearly summable, and ones that aren't are in table views that precalc the "daily uniques" or other metrics that may need a bit special handling. You've started modeling your data. No, we don't need olap cubes. But we do need some type of rigor around analytic data. Otherwise, why go to all the trouble to collect it, count it, and predict from it if it may be wrong with no measure of that uncertainty? And yeah, Kimball et al are from a world where olap was the answr, but it turns out they solved a broader set of sql problems. So, worth learning the good, toss the dated, and see what good data modeling can do for your analysis and predictions.

mwexler 5y ago
BTW, their followup piece at https://www.holistics.io/blog/olap-is-not-olap-cube/ https://www.holistics.io/blog/olap-is-not-olap-cube/ is a nicer drill down into some of these issues. Also a good read.
tresil 5y ago
Could not have written this any better. I’ve been in the data business for over 10 years. A number of technologies have come and gone during that time, but dimensional modeling and solid SQL skills keep delivering remarkable value.
chrstr 5y ago
Indeed, it is generally good for analytics to have a conceptual data model that is reasonably efficient both on the technical side (to quickly get answers for queries), and also on the logical side (to quickly translate questions to queries). This model doesn't need to be OLAP cubes, but it's also not that easy to find something better.
glogla 5y ago
> Actually, no. We model our data this way so it can be used for business decisions Yeah, that would be my disagreement as well. Sure, some of Kimball might be obsoleted by modern technology. But I don't think the point of Kimball is cubes, and even if it was, what better is there? I'd be really interested in what else is there, that is more modern and suited to the modern world. Data Vault? That probably isn't it, most of all because it's all about the "platform to build you Kimball on" but not the actual business modelling. (But the parts about Metrics Vault and Quality Vault and the "service stuff" are really good.) Anchor Modelling? That one seems, looking by 2021 eyes, like a way to implement columnar storage and schema-on-read in MS SQL Server using store procedures ... which is probably not actually a good idea. Puppini Bridge aka. Unified Star Schema seems like interesting concept, especially in self-service space. But even it's proponents warn you it doesn't scale performance-wise and also it is kind of incremental change on Kimball. (But the way the bridge table is kinda adjacency matrix of graph of your objects tickles my inner computer scientist fancy) So really, what else is there?
- cedricd 5y ago
  This isn't widely known, but activity schema[1] is aiming for this space (disclosure - I co-founded the company supporting it). It's a modeling approach that separates data modeling from data querying -- meaning that once data is modeled, it can answer any number of questions. The base building blocks of the data model can be combined at query time without having to figure out explicit foreign key joins. [1] https://www.narrator.ai/activity-schema/ https://www.narrator.ai/activity-schema/
- impjohn 5y ago
  Have you looked into the VKG (Virtual Knowledge Graph) approach?
hodgesrm 5y ago
> No, we don't need olap cubes. But we do need some type of rigor around analytic data. Otherwise, why go to all the trouble to collect it, count it, and predict from it if it may be wrong with no measure of that uncertainty? Modern data warehouses don't need to build cubes for performance reasons. Low latency data warehouses like ClickHouse or Druid can aggregate directly off source data. The biggest driver for modeling is allowing non-coders to access data and perform their own analyses. I don't see that problem ever going away. Cube modeling with dimensions and measures solves it well.

mwexler 5y ago

BTW, their followup piece at https://www.holistics.io/blog/olap-is-not-olap-cube/ https://www.holistics.io/blog/olap-is-not-olap-cube/ is a nicer drill down into some of these issues. Also a good read.

tresil 5y ago

Could not have written this any better. I’ve been in the data business for over 10 years. A number of technologies have come and gone during that time, but dimensional modeling and solid SQL skills keep delivering remarkable value.

chrstr 5y ago

Indeed, it is generally good for analytics to have a conceptual data model that is reasonably efficient both on the technical side (to quickly get answers for queries), and also on the logical side (to quickly translate questions to queries). This model doesn't need to be OLAP cubes, but it's also not that easy to find something better.

glogla 5y ago

> Actually, no. We model our data this way so it can be used for business decisions Yeah, that would be my disagreement as well. Sure, some of Kimball might be obsoleted by modern technology. But I don't think the point of Kimball is cubes, and even if it was, what better is there? I'd be really interested in what else is there, that is more modern and suited to the modern world. Data Vault? That probably isn't it, most of all because it's all about the "platform to build you Kimball on" but not the actual business modelling. (But the parts about Metrics Vault and Quality Vault and the "service stuff" are really good.) Anchor Modelling? That one seems, looking by 2021 eyes, like a way to implement columnar storage and schema-on-read in MS SQL Server using store procedures ... which is probably not actually a good idea. Puppini Bridge aka. Unified Star Schema seems like interesting concept, especially in self-service space. But even it's proponents warn you it doesn't scale performance-wise and also it is kind of incremental change on Kimball. (But the way the bridge table is kinda adjacency matrix of graph of your objects tickles my inner computer scientist fancy) So really, what else is there?

cedricd 5y ago
This isn't widely known, but activity schema[1] is aiming for this space (disclosure - I co-founded the company supporting it). It's a modeling approach that separates data modeling from data querying -- meaning that once data is modeled, it can answer any number of questions. The base building blocks of the data model can be combined at query time without having to figure out explicit foreign key joins. [1] https://www.narrator.ai/activity-schema/ https://www.narrator.ai/activity-schema/
impjohn 5y ago
Have you looked into the VKG (Virtual Knowledge Graph) approach?

hodgesrm 5y ago

> No, we don't need olap cubes. But we do need some type of rigor around analytic data. Otherwise, why go to all the trouble to collect it, count it, and predict from it if it may be wrong with no measure of that uncertainty? Modern data warehouses don't need to build cubes for performance reasons. Low latency data warehouses like ClickHouse or Druid can aggregate directly off source data. The biggest driver for modeling is allowing non-coders to access data and perform their own analyses. I don't see that problem ever going away. Cube modeling with dimensions and measures solves it well.