3 ms·
Is there value in doing this? Absolutely. Figure out the core metrics for your business, measure them, pay attention to those metrics, and figure out how to i
by btilly 16y ago
Is there value in doing this? Absolutely. Figure out the core metrics for your business, measure them, pay attention to those metrics, and figure out how to improve them. Going one step further, I recommend creating an A/B test platform for yourself where you can easily measure how A/B tests impact the metrics you care about. There is a lot of value to be derived here, and in the end it is going to be an open-ended commitment.
Is it worthwhile for you to do this right now? Only you can answer that. These things are certainly not part of a minimum viable product. But if you find yourself looking at certain figures over an over again, then by all means build a dashboard.
Pro tip. If you're still small, you can probably query your whole database interactively without any noticeable impact on your site. Throw together a prototype based on doing that that before you start to get fancy with things like regular cron jobs. It is faster to put together, and it is easier to make tweaks. Only after you know you've got the right stuff should you migrate the queries to cron jobs and refactor your dashboard to hit cached data.
- joshdotsmith 16y agoYeah, I'm almost done with what I'd call our MVP (and that's after spending a long time working on a very full-featured site that just wasn't as compelling as the MVP itself), so these metrics are going to help me figure out where to head next. I'm not sure yet how I'd go about building an A/B platform, but its on the docket of things I want to do. Also, the reason I'd be running a cron job is to have hourly data for all of these metrics (to look for those changes over time).
- btilly 16y agoSee http://elem.com/~btilly/effective-ab-testing/ http://elem.com/~btilly/effective-ab-testing/ for my advice on how to put together an A/B platform. Sections 2-4 are the most relevant. The most important detail is the warning in slide 34 that you should not use the person_id to set the A/B test version. The reason is that if you make that mistake, then you can only run one test at a time, and can't easily drop versions of tests later. On the other hand if you make assignment random, then you are free to run tests in parallel, which is very useful for trying out lots of ideas. (Particularly when you have less traffic.)
- joshdotsmith 16y agoThanks for that! I actually just signed up with Optimizely today and really love the way they run A/B tests so far. I need to think a little deeper on how to integrate these tests with the rest of my metrics, but it's a really good (and easy) start. While I haven't reached a level of real statistical confidence yet with my first test, I've already been able to increase engagement with a single change on my landing page by 35%. Wow. clickable: http://www.optimizely.com/ http://www.optimizely.com/