3 ms·
Appreciate the skepticism, since yes, we're a totally different approach. I'll try to address your points in order. Yes, we agree that 99% of the work is dete
by cedricd 6y ago
Appreciate the skepticism, since yes, we're a totally different approach.
I'll try to address your points in order.
Yes, we agree that 99% of the work is determining what the data means. Our structure doesn't magically make things better because of its structure. It's that once you have it analysis / aggregation on top of it becomes substantially less work because you don't have to constantly redo models to answer new questions.
Yeah, we've all been there -- production DBs aren't typically architected to store historical data. But we've found in practice that the data sources you most care about do have it. Page views, emails sent / received, completed orders, etc. all have timestamps. And for some things you don't need it. If you wanted to do a query with all customers who are VIPs, you wouldn't need a 'became VIP' activity. Adding is_VIP as a feature to the customer in the activity stream works too. Generally if you can do an analysis the more traditional way then you should already have the data to do it in Narrator too.
Sure, star schemas are the way of doing things and this is a new approach. But the efficiency gains realized by our own data scientists are enough to where they wouldn't go back -- it warrants the investment in learning it. Our challenge as a business will be how to convince others of that as well.
What we mean by single source of truth is that data is internally consistent - each term is defined once. In your scenario you'll have a single 'completed order' activity with the total order amount. If you want to add shipping cost that's fine -- add a 'added shipping to order' activity with the cost in it. Do the same with sales tax. Bob, Alice, and Jimmy can create reports with whatever activities they want. The crucial point is by making those reports they're not defining a new model. They're just combining activities. Since all tables are generated straight from the activity stream a future analyst won't use a materialized view based on Bob's data to build a new report -- they'll build it straight from the original activities.
- modoc 6y agoIf I am understanding correctly, if my BI folks want to report on things like order total or order sub total, or order sub total + shipping, or order sub total + tax, or presumably anything about the contents of the order (SKUs, quantities, per item pricing, price adjustments, coupons and promotions, etc...) instead of capturing a "order submitted" event, we'd have to capture every add to cart, every cart pricing recalc operation, shipping address added, shipping method selected, shipping cost added to order, shipping method changed, new shipping cost added to order, etc... as separate events? And have the smarts to generate reports using the final selected shipping method (for example) for calculations? For an average-ish B2C order that could be dozens of events, and for a complex B2B order that might easily be hundreds of events.
- cedricd 6y agoIt's not quite that granular. I was more responding to the idea that there can't be a source of truth for these activities. We actually have several e-commerce companies using our platform (with decently high volume). In practice we tend to see events like 'completed order' 'shipped order' 'product added to cart' 'order delivered'. I.e. they're all very discrete differentiated steps in the process. There's a bit of an art between when to make a new activity and when to add it as metadata on an existing one. A completed order will more likely have 'discount code' as a feature than 'discount code applied' as an activity for example. Your order completed event could have total amount along with tax, shipping, etc costs that add up to the total. It depends on the analyses you want to generate. We do see things like an order submitted event with the total, num products purchased, discount code on it, and a separate 'purchased product' event with individual product price, sku, etc. Once can do things like MRR and another could let you identify best selling skus or product categories. Happy to chat more offline if you want to dive into the specifics for your use case. We love digging into what sorts of analysis someone wants to do and figuring out which activities make sense https://calendly.com/ahmed-narrator/30min-1 https://calendly.com/ahmed-narrator/30min-1
- mst 6y agoI think the point is that you can choose your own granularity, and that it tries to make things fast no matter that choice.