4 ms·

Speaking as an analytics architect ... You'll be a lot better off spending your mental energy thinking about the outcomes you want to achieve (user engagement,

Speaking as an analytics architect ... You'll be a lot better off spending your mental energy thinking about the outcomes you want to achieve (user engagement, upselling, growth, etc) and the types of analysis you'll need to understand what changes you need to make to produce those outcomes. Protip: this is actually really hard, and people underestimate it by orders of magnitude. A blog post by Roger Peng (with indirect commentary from John Tukey) ... https://simplystatistics.org/2019/04/17/tukey-design-thinking-and-better-questions/ https://simplystatistics.org/2019/04/17/tukey-design-thinkin... One other immediate tip is to start thinking about correlating your telemetry with user surveys - again, strongly focusing on outcomes and the controllable aspects of those outcomes. Don't let the data lead the discuisson; decide on the question you're asking, and the implications of all of the possible answers to that question (clearly yes, clearly no, mixed, etc) before you ask it. Then engineer the lightest weight system possible to ingest, process, store, analyze, and visualize that data. For me, that would just be: 1. Log data in whatever logging tool you like. Persist the raw stuff forever in a cheap data lake. 2. Batch at some fixed interval into a staging area of a relational DB. 3. Transform it with stored procedures for now (while you figure out what the right transforms are) into a flat fact table. 4. Visualize in Superset or PowerBI or even plain old Excel. Once you've got the patterns of analysis at least fundamentally right you can consider stream processing (Flink or Kafka Streams are fine) to replace 2 and 3.

coderintherye 7y ago
Yes! Getting and transforming your data these days is easy and cheap to store. Use existing tooling for that (e.g. Snowplow, etc.) The hard part is producing actionable analysis.
DanHulton 7y ago
One small point about "persist the raw stuff forever": ...maybe don't? I wouldn't go so far as to call user data a toxic asset, but I'd definitely refer to it in the same breath as "technical debt". Sometimes you have to take on some to get anything done, but you need to recognize that it's a potential liability, as well. Persist it as long as you need it to obtain value from it, but after that, you're just holding on to something you can lose some day in a data breach, or build up enough of that people start to get mistrustful (Facebook, Google, etc.) Data expiration policies should definitely be something you think about and have justification for, even if you do decide that your expiration is "Never".

coderintherye 7y ago

Yes! Getting and transforming your data these days is easy and cheap to store. Use existing tooling for that (e.g. Snowplow, etc.) The hard part is producing actionable analysis.

DanHulton 7y ago

One small point about "persist the raw stuff forever": ...maybe don't? I wouldn't go so far as to call user data a toxic asset, but I'd definitely refer to it in the same breath as "technical debt". Sometimes you have to take on some to get anything done, but you need to recognize that it's a potential liability, as well. Persist it as long as you need it to obtain value from it, but after that, you're just holding on to something you can lose some day in a data breach, or build up enough of that people start to get mistrustful (Facebook, Google, etc.) Data expiration policies should definitely be something you think about and have justification for, even if you do decide that your expiration is "Never".