3 ms·

I'm surprised that there isn't a mention of Google Dataflow aka Apache Beam. The Beam programming model is specifically designed to solve nearly all of the prob

by teej 9y ago

I'm surprised that there isn't a mention of Google Dataflow aka Apache Beam. The Beam programming model is specifically designed to solve nearly all of the problems this post is addressing. > It is likely that one day, I'll need to shard the data and distribute the processing amongst multiple servers. But no company I've used this with currently has enough data flowing through its analytics system, or intense amounts of real-time processing, to warrant such a complexity. This solution is so over-engineered for low data volume. You could capture all of the business value for 10% of the engineering effort by just dumping this data into a database meant for analytics. And then you'd at least have an answer for things like fixing broken data, making full-history business logic changes, merging events, etc. If you're in AWS, sending your events from snowplow into S3 and then into Redshift/Athena/Presto/PostgreSQL is the way to go.

pragmacoders 9y ago
I am aggregating events on a user-level and on a session-level so as to gain a higher level understanding of user behavior. Like: - "What is the median time that users spend on a given page?" - "How many events do users generally fire in their first 3 sessions?" -"Who visited Page A, then Page B, THEN Page C?" -"How many users who used this feature also used this feature within the same day?" This is difficult to do with purely event-level metrics. It also takes a really long time and isn't suited to exploration. We're already using a data warehouse (AWS Redshift) to analyze event-level metrics. But more complex queries are much easier, and much faster to do, when the data is stored in ways that are more suited to such calculations. So, that's what this article is hopefully helping people to do! For a related topic, "Entity-Centric Indexing" is the ElasticSearch equivalent of what I'm working on!

by teej 9y ago

pragmacoders 9y ago

I am aggregating events on a user-level and on a session-level so as to gain a higher level understanding of user behavior. Like: - "What is the median time that users spend on a given page?" - "How many events do users generally fire in their first 3 sessions?" -"Who visited Page A, then Page B, THEN Page C?" -"How many users who used this feature also used this feature within the same day?" This is difficult to do with purely event-level metrics. It also takes a really long time and isn't suited to exploration. We're already using a data warehouse (AWS Redshift) to analyze event-level metrics. But more complex queries are much easier, and much faster to do, when the data is stored in ways that are more suited to such calculations. So, that's what this article is hopefully helping people to do! For a related topic, "Entity-Centric Indexing" is the ElasticSearch equivalent of what I'm working on!