4 ms·
(Data engineer here) Nice article! I did something very similar to this for my blog but used Snowplow's javascript tracker (https://github.com/snowplow/snowplo
by mejakethomas 8y ago
(Data engineer here)
Nice article! I did something very similar to this for my blog but used Snowplow's javascript tracker (https://github.com/snowplow/snowplow-javascript-tracker https://github.com/snowplow/snowplow-javascript-tracker), a cloudfront distribution with s3 log forwarding, a couple lambda functions (with s3 "put" triggers), S3 as the post-processed storage layer, and AWS athena as the query layer. The system costs under $1 per month, is very scalable, and is producing amazingly good/structured data with mid-level latency. I've written about it here:
https://bostata.com/post/client-side-instrumentation-for-under-one-dollar/ https://bostata.com/post/client-side-instrumentation-for-und...
By using the snowplow javascript tracker, you get a ton of functionality out of the box when it comes to respecting "do not track", structured event formatting, additional browser contexts, etc. If you want to see how the blog site is functionally instrumented, filter network requests by "stm" (sent time) and you'll see what's being collected.
I've found (after setting similar systems for 15+ companies of varying scale) that where a system like this breaks down is when you want to warehouse event data and tie it to other critical business metrics (stripe, salesforce, database tables that underpin the application, etc). Another point it starts to break down is when you need low-latency data access. At that point it makes more and more sense to run data into a stream (kinesis/kafka/etc) and have "low latency" (couple hundred ms or less) and "high latency" (minutes/hours/etc) points of centralization.
Using multi-az/replicated stream-based infrastructure (like snowplow's scala stuff) has been completely transformational to numerous companies I've set it up at. A single source of truth when it comes to both low-latency and med/high-latency client side event data is absolutely massive. Secondly, being able to tie many sources of data together (via warehousing into redshift or snowflake) is eye-opening every single time. I've recently been running ~300k+ requests/minute through snowplow's stream-based infrastructure and it's rock-solid.
Again, nice post! It's awesome to see people doing similar things. :)
- mejakethomas 8y agoAnd also, huge props for the creativity! You hit a nerve here, and it's a topic being discussed at pretty much any/all companies.
- pcmaffey 8y agoThanks! Not a data engineer, but I used to work at a data engineering company, and can attest to the complexity and rawness of the industry. Your setup looks solid! As a frontend engineer having seen under the hood of data pipelines at scale, I wanted to reverse engineer the parts of it that I care about (product analytics via event logging), and package it up for my little side projects. It's awesome that this is inspiring to people. If people get anything from what I wrote, it'd be this: While large companies all roll their own data pipelines, it's _not that difficult_ for a startup / smaller co / individual to do product analytics on a level that makes sense for them, without just automatically reaching for GA or whatever.
- mejakethomas 8y agoI totally agree
- samat 8y agoIs there a way to guild a comment here?
- kfarr 8y agoThis is a great idea using cloudfront logs as the "raw data store". Well done.
- simonw 8y agoYour post is fantastic. Is there any chance you could share the lambda function code you use for annotating the log entries?
- dpeck 8y agoCould you share the lambda function? It'd be great to be able to see this end-to-end without getting into implementation.