4 ms·

Hard to imagine a worse idea. The problem in data warehousing is not the structure, it's the definitions. It doesn't matter one tiny little bit whether you sto

Hard to imagine a worse idea. The problem in data warehousing is not the structure, it's the definitions. It doesn't matter one tiny little bit whether you store your data in a fact/dimension star schema, a normalized OLTP-style schema, arbitrary aggregate/rollup tables, or this one-table monstrosity you've constructed. You still have to do the work of determining what the data means. This isn't 50% of the work of building a data warehouse. It's 99% of the work. This "Activity Stream" concept has some merit but most data sources can't support it. Much to my eternal frustration, most systems only store current state, and have no concept of history or events. You can't extract "activities" from a data system that doesn't record them. Still, maybe I'm wrong and your idea is great. The market will reject it. Star schema modeling is woven into the very fabric of the data warehousing profession. We use it everywhere, whether it's appropriate or not (and it is often not appropriate). It's what all the BI tools support. It's what analysts know how to write queries against. Your thing may be cool and all but no one knows how to use it. One last thing: "Numbers would always match (single source of truth)..." Not even close. This belies just massive ignorance of how analytics actually work. Let's say I have a data point that says the total amount of order #123 is $42.00. Ok... does that include shipping? Sales tax? Oh it does, and finance cares about that, so now I need to decompose that into 34.95 + 5.00 + 2.05 and label appropriately. Now Bob's report can sum the item total (no shipping, no taxes) and Alice's report can include shipping but not taxes, and Jimmy's report can sum the total, and now your wonderful "single source of truth" story goes in the toilet. Because "$42.00" isn't truth it's just a value. What turns it into truth is definition and consensus, which are social problems, not technical problems.

dang 6y ago
Can you please make your substantive points thoughtfully, rather than in the flamewar style? This is particularly important when people are sharing their work. We don't want a culture in which people get flamed and belittled for doing that. This is important because the effects that comments like this one produce are much stronger than the people making them assume they are. Worse, they compound. Then, unintentionally, we end up with an asshole culture which no one would want to subject themselves to. On HN, we want the incentives to go exactly the other way. I'm sure you can make your substantive criticisms without putdowns and name-calling if you want to—and that would actually be a quite valuable contribution. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
cedricd 6y ago
Appreciate the skepticism, since yes, we're a totally different approach. I'll try to address your points in order. Yes, we agree that 99% of the work is determining what the data means. Our structure doesn't magically make things better because of its structure. It's that once you have it analysis / aggregation on top of it becomes substantially less work because you don't have to constantly redo models to answer new questions. Yeah, we've all been there -- production DBs aren't typically architected to store historical data. But we've found in practice that the data sources you most care about do have it. Page views, emails sent / received, completed orders, etc. all have timestamps. And for some things you don't need it. If you wanted to do a query with all customers who are VIPs, you wouldn't need a 'became VIP' activity. Adding is_VIP as a feature to the customer in the activity stream works too. Generally if you can do an analysis the more traditional way then you should already have the data to do it in Narrator too. Sure, star schemas are the way of doing things and this is a new approach. But the efficiency gains realized by our own data scientists are enough to where they wouldn't go back -- it warrants the investment in learning it. Our challenge as a business will be how to convince others of that as well. What we mean by single source of truth is that data is internally consistent - each term is defined once. In your scenario you'll have a single 'completed order' activity with the total order amount. If you want to add shipping cost that's fine -- add a 'added shipping to order' activity with the cost in it. Do the same with sales tax. Bob, Alice, and Jimmy can create reports with whatever activities they want. The crucial point is by making those reports they're not defining a new model. They're just combining activities. Since all tables are generated straight from the activity stream a future analyst won't use a materialized view based on Bob's data to build a new report -- they'll build it straight from the original activities.
- modoc 6y ago
  If I am understanding correctly, if my BI folks want to report on things like order total or order sub total, or order sub total + shipping, or order sub total + tax, or presumably anything about the contents of the order (SKUs, quantities, per item pricing, price adjustments, coupons and promotions, etc...) instead of capturing a "order submitted" event, we'd have to capture every add to cart, every cart pricing recalc operation, shipping address added, shipping method selected, shipping cost added to order, shipping method changed, new shipping cost added to order, etc... as separate events? And have the smarts to generate reports using the final selected shipping method (for example) for calculations? For an average-ish B2C order that could be dozens of events, and for a complex B2B order that might easily be hundreds of events.
ahmedelsama 6y ago
Definitions are super important. Getting agreement on definitions is really hard and can easily get us stuck in a loop. In my last job, we spent months talking about "What is a Sale?" is it when someone signs a contract or when they pay or when they move in, etc... Then you add your sale metric into a table and as that table is used in different places the sales numbers till don't match, people don't remember what a sale is and the conversation starts up again. Why is Narrator different? In Narrator, you don't define what a sale is, you break up your data into customer actions. So "Signed Contract", "Paid Invoice", "Moved In" and thus as people ask different questions we can alway clearly see what they are referring to. Step 1 to alignment is guaranteeing consistency and transparency. How do you deal with states? You are right that does make it really hard. We see customers leveraging the incremental aspect of the activity stream to diff the stream with the updated_at to pull out changes as the activity stream updates. (every 15 minutes so yes you will loose changes in between that time). This doesn't solve the problem but does take you much closer. And then when you do add proper timestamps then you can have historical data in the activity stream and cleaned data from you new tables merged. All the users using that activity are NOT affected. Not perfect but allows us to have as accurate data as possible. What about the market? Yes, I agree the market is a challenge since the world has been only using star schema. We hope that standardization, speed, reusability aspect of Narrator is so compelling that people slowly switch. How do we get numbers to match? Consistency and Transparency. Every one who uses the "Completed Order" uses the same revenue. SO it is consistent! Then you can add the "Shipped Order" activity which has the shipping amount. By having clear CONSISTENT definitions and clear activities you end up in a world where your numbers match. The only ways for numbers not to match is if some is being deliberate about getting the data not to match which thanks to dataset, is always transparent. Definitions are a social problem but technology, limitations and consistency can help a lot.

dang 6y ago

Can you please make your substantive points thoughtfully, rather than in the flamewar style? This is particularly important when people are sharing their work. We don't want a culture in which people get flamed and belittled for doing that. This is important because the effects that comments like this one produce are much stronger than the people making them assume they are. Worse, they compound. Then, unintentionally, we end up with an asshole culture which no one would want to subject themselves to. On HN, we want the incentives to go exactly the other way. I'm sure you can make your substantive criticisms without putdowns and name-calling if you want to—and that would actually be a quite valuable contribution. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

cedricd 6y ago

Appreciate the skepticism, since yes, we're a totally different approach. I'll try to address your points in order. Yes, we agree that 99% of the work is determining what the data means. Our structure doesn't magically make things better because of its structure. It's that once you have it analysis / aggregation on top of it becomes substantially less work because you don't have to constantly redo models to answer new questions. Yeah, we've all been there -- production DBs aren't typically architected to store historical data. But we've found in practice that the data sources you most care about do have it. Page views, emails sent / received, completed orders, etc. all have timestamps. And for some things you don't need it. If you wanted to do a query with all customers who are VIPs, you wouldn't need a 'became VIP' activity. Adding is_VIP as a feature to the customer in the activity stream works too. Generally if you can do an analysis the more traditional way then you should already have the data to do it in Narrator too. Sure, star schemas are the way of doing things and this is a new approach. But the efficiency gains realized by our own data scientists are enough to where they wouldn't go back -- it warrants the investment in learning it. Our challenge as a business will be how to convince others of that as well. What we mean by single source of truth is that data is internally consistent - each term is defined once. In your scenario you'll have a single 'completed order' activity with the total order amount. If you want to add shipping cost that's fine -- add a 'added shipping to order' activity with the cost in it. Do the same with sales tax. Bob, Alice, and Jimmy can create reports with whatever activities they want. The crucial point is by making those reports they're not defining a new model. They're just combining activities. Since all tables are generated straight from the activity stream a future analyst won't use a materialized view based on Bob's data to build a new report -- they'll build it straight from the original activities.

modoc 6y ago
If I am understanding correctly, if my BI folks want to report on things like order total or order sub total, or order sub total + shipping, or order sub total + tax, or presumably anything about the contents of the order (SKUs, quantities, per item pricing, price adjustments, coupons and promotions, etc...) instead of capturing a "order submitted" event, we'd have to capture every add to cart, every cart pricing recalc operation, shipping address added, shipping method selected, shipping cost added to order, shipping method changed, new shipping cost added to order, etc... as separate events? And have the smarts to generate reports using the final selected shipping method (for example) for calculations? For an average-ish B2C order that could be dozens of events, and for a complex B2B order that might easily be hundreds of events.

ahmedelsama 6y ago

Definitions are super important. Getting agreement on definitions is really hard and can easily get us stuck in a loop. In my last job, we spent months talking about "What is a Sale?" is it when someone signs a contract or when they pay or when they move in, etc... Then you add your sale metric into a table and as that table is used in different places the sales numbers till don't match, people don't remember what a sale is and the conversation starts up again. Why is Narrator different? In Narrator, you don't define what a sale is, you break up your data into customer actions. So "Signed Contract", "Paid Invoice", "Moved In" and thus as people ask different questions we can alway clearly see what they are referring to. Step 1 to alignment is guaranteeing consistency and transparency. How do you deal with states? You are right that does make it really hard. We see customers leveraging the incremental aspect of the activity stream to diff the stream with the updated_at to pull out changes as the activity stream updates. (every 15 minutes so yes you will loose changes in between that time). This doesn't solve the problem but does take you much closer. And then when you do add proper timestamps then you can have historical data in the activity stream and cleaned data from you new tables merged. All the users using that activity are NOT affected. Not perfect but allows us to have as accurate data as possible. What about the market? Yes, I agree the market is a challenge since the world has been only using star schema. We hope that standardization, speed, reusability aspect of Narrator is so compelling that people slowly switch. How do we get numbers to match? Consistency and Transparency. Every one who uses the "Completed Order" uses the same revenue. SO it is consistent! Then you can add the "Shipped Order" activity which has the shipping amount. By having clear CONSISTENT definitions and clear activities you end up in a world where your numbers match. The only ways for numbers not to match is if some is being deliberate about getting the data not to match which thanks to dataset, is always transparent. Definitions are a social problem but technology, limitations and consistency can help a lot.