4 ms·

I've been in this field for more than a decade and I couldn't agree more. The tools, and even the underlying theory is stuck in the past, and every project I wo

I've been in this field for more than a decade and I couldn't agree more. The tools, and even the underlying theory is stuck in the past, and every project I work on is hampered by the technology not meeting the expectations of (business) users who demand a much more intuitive way of working - because that's what the experience every day on consumer devices. There are issues though with getting consistent and accurate measures that are not easily solved by just looking at the data - albeit at a very high level of detail) for example when customers move or change details, it's easy to lose track of where you should be reporting the facts. And depending on the volume of data it's not going to be a long term solution to recalculate everything every day - which is the mistake a lot of companies make when trying to cope with this. The problem is in your question. The Cube. This is a construct that assumes that the underlying data is typed allowing it to be re-arranged into facts, characteristics and attributes, and unless you have the foresight to think of your db like this before you build your web app, it's hard going to squeeze it into the dimensional model. In fact I'd go as far as saying that the dimensional model is no longer a suitable model for dealing with event based data generation, and that may be why you ask the question. It's one of my personal goals to solve this, I have some thoughts but it's not quite my priority at the moment, but I'd love to hear what others have to say on this.

siganakis 14y ago
I too have spent quite a few years working in this area. I have primarily been involved in building tools to make this kind of analysis easier - but it is very difficult. As you mention, the primary issue is that almost all business data is very complex. There are very few areas (in my experience), where simple queries offer much value, because there are so many niggling business rules that lead to exceptions and counter exceptions to rules. This means that "self service" analytics are pretty much impossible. There are often very few people in an organization who actually know how to define even seemingly simple measures like profit or revenue. This means you need consultants who can learn the business rules, and who also understand the data platform. This makes these projects expensive, and makes it hard for platforms to win in the market. For the most part in my experience, the best way forward is to choose a database engine (MySQL, Postgress, Oracle, DB2, SQL Server), get the data into it (itself difficult!), then write a bunch of SQL to incorporate the business rules. Then if you are feeling fancy (and still have budget!), built a nice HTML front end on it. Its not very sexy, but it does work. I would love it if there was an easier way, but I am yet to find it.
- viandante 14y ago
  Agree. But usually business rules and exceptions boil down to filters and look ups on the dataset. It is difficult to keep up with that mess from outside, but if you deliver a way to make look ups and filters efficiently, maybe you got a solution.

siganakis 14y ago

I too have spent quite a few years working in this area. I have primarily been involved in building tools to make this kind of analysis easier - but it is very difficult. As you mention, the primary issue is that almost all business data is very complex. There are very few areas (in my experience), where simple queries offer much value, because there are so many niggling business rules that lead to exceptions and counter exceptions to rules. This means that "self service" analytics are pretty much impossible. There are often very few people in an organization who actually know how to define even seemingly simple measures like profit or revenue. This means you need consultants who can learn the business rules, and who also understand the data platform. This makes these projects expensive, and makes it hard for platforms to win in the market. For the most part in my experience, the best way forward is to choose a database engine (MySQL, Postgress, Oracle, DB2, SQL Server), get the data into it (itself difficult!), then write a bunch of SQL to incorporate the business rules. Then if you are feeling fancy (and still have budget!), built a nice HTML front end on it. Its not very sexy, but it does work. I would love it if there was an easier way, but I am yet to find it.

viandante 14y ago
Agree. But usually business rules and exceptions boil down to filters and look ups on the dataset. It is difficult to keep up with that mess from outside, but if you deliver a way to make look ups and filters efficiently, maybe you got a solution.