4 ms·

I remember evaluating Heap (granted, a long time ago, so I'm sure it's evolved a lot since then) and the auto-tracking capabilities and retroactive analysis sou

I remember evaluating Heap (granted, a long time ago, so I'm sure it's evolved a lot since then) and the auto-tracking capabilities and retroactive analysis sounded awesome, but in practice it wasn't as valuable or easy as I thought. I found that in general, knowing the questions you want to answer and structuring your tracking around that to be far more useful, and that actually being able to take mountains of generic behavioral data and making sense of it wasn't that easy and a ton of work in and of itself. E.g. retroactively finding clicks to a single 'signup button' is easy, but a lot harder when that button is on many different pages and you want to group them or segment them in certain ways, or the implementation changes in some places, etc. In the end, it seemed like it was doing engineering work just to parse the data back out so it seemed more sane to structure in a better way ahead of time. I'm curious what steps have been taken in the past several years to improve on this and how Freshpaint is approaching this. I think it's a really interesting problem and if done well, could be massively valuable. Good luck!

soumyadeb 7y ago
We have been thinking about this a lot in my company (RudderLabs - where we have built an open-source product around auto-tracking). Totally agreed that managing/using all this auto-track data can be really overwhelming and sometimes it is just easier to just generate the events via code. Your maintainability point is also very important - with events generated via code, you have a single owner for updating the website and updating the events as opposed to auto-track where the responsibilities are split between website developer and marketer. However, there are interesting use cases that are only possible via auto-tracking. Let's say a particular link on your website is broken (e.g. takes to an old sign-up page which is broken). You will be able to identify that that broken click "automatically" because the ratio of successfully signed up vs not-signed up users would be very low for people who click on that vs those who don't. You can automatically highlight many such interesting insights by doing some basic stats counting. If you throw more advanced MLish stuff, you can find really interesting (but sometimes hard to explain) insights. Happy to chat more if you want. Email in profile.
malisper 7y ago
There's two main reasons you would want to use auto-track. You should use auto-track if: - You don't have the resources to setup tracking code. - You want to look at data that you forgot to track previously. We've heard a lot of stories of people launching a new feature and forgetting to setup tracking on it. If you have the resources to figure out and instrument everything you want to track ahead of time, then you should go ahead and do so. A manually tracked implementation is more likely to be correct than a autotracked one. Even so, auto-track is still useful as a tool for when you have new questions you want to ask or you realize that you need data that you aren't currently tracking. You should not use autotrack to perform fishing expeditions where you just look at all the data and try to make sense of it. You should still have some guiding questions and autotrack is a tool that makes it easy to get the data you need to answer those questions.

soumyadeb 7y ago

We have been thinking about this a lot in my company (RudderLabs - where we have built an open-source product around auto-tracking). Totally agreed that managing/using all this auto-track data can be really overwhelming and sometimes it is just easier to just generate the events via code. Your maintainability point is also very important - with events generated via code, you have a single owner for updating the website and updating the events as opposed to auto-track where the responsibilities are split between website developer and marketer. However, there are interesting use cases that are only possible via auto-tracking. Let's say a particular link on your website is broken (e.g. takes to an old sign-up page which is broken). You will be able to identify that that broken click "automatically" because the ratio of successfully signed up vs not-signed up users would be very low for people who click on that vs those who don't. You can automatically highlight many such interesting insights by doing some basic stats counting. If you throw more advanced MLish stuff, you can find really interesting (but sometimes hard to explain) insights. Happy to chat more if you want. Email in profile.

malisper 7y ago

There's two main reasons you would want to use auto-track. You should use auto-track if: - You don't have the resources to setup tracking code. - You want to look at data that you forgot to track previously. We've heard a lot of stories of people launching a new feature and forgetting to setup tracking on it. If you have the resources to figure out and instrument everything you want to track ahead of time, then you should go ahead and do so. A manually tracked implementation is more likely to be correct than a autotracked one. Even so, auto-track is still useful as a tool for when you have new questions you want to ask or you realize that you need data that you aren't currently tracking. You should not use autotrack to perform fishing expeditions where you just look at all the data and try to make sense of it. You should still have some guiding questions and autotrack is a tool that makes it easy to get the data you need to answer those questions.