3 ms·

The criticism of sampling in this blogpost is way overblown. (1) there's virtually no difference between the accuracy of a sample size of 10k and one of 100k, a

The criticism of sampling in this blogpost is way overblown. (1) there's virtually no difference between the accuracy of a sample size of 10k and one of 100k, and those are the sample sizes you're usually working with (2) in particular, when working with custom reports you get from the API, you can actually specify the accuracy you want (high, normal, low) as part of your API request. To boot, 9 times out of 10 what you're interested in as an analyst is not the absolute numbers anyway, but ratios and trends. So whether or not e.g. every precious little pageview gets counted is irrelevant, as long as the way it is counted is stable over time. This is endemic to so many discussions about big data: "if we don't have every individual data point ever, all hell will break loose and we lose the ability to make any sense of the data." Take an intro to basic probability and statistics, will ya. There's legitimate problems with using Google Analytics for a startup, but they're mostly related to the fact that it doesn't provide good tooling around A/B, customer lifecycle management and custom metrics – they're possible but you're not making it easy on yourself. These things are the bread and butter of SaaS and app analytics (as opposed to ecommerce or media) so it makes sense to invest in something like Mixpanel/Heap/Keen/KISSMetrics. But those are issues with Google Analytics as a product, not with the quality of its data.

fndrplayer13 12y ago
I agree with this in principle, but honestly the sampled numbers can be really misleading. It all depends on the magnitude of your visits. If you ask for a year's worth of data for a company with 10 million visits for the year, you're going to get into trouble. This makes creating tools and automated workflows that use the GA API pretty hard. Sampling itself makes a ton of sense though, and its why Google can offer this service at the level of quality they do.
dredmorbius 12y ago
Generally yes, sampling works well and is robust even for surprisingly small samples. For stratified samples -- particularly if you want to stratify traffic by income potential, sampling over all visits may mask data of interest. As for your example, a 10k and 100k sample will provide a 3.16x difference in the standard deviation (that is: accuracy of estimates of central tendency). That is, 10x more datapoints result in only 3x greater accuracy.
crdb 12y ago
Yes and no. Exporting visit trends over 5 landing pages and a month? Sure. Exporting page views for 100,000 products each of whom got 5-100 views? Then that 5% sample is going to exclude most products. The latter approach is however necessary if you're trying to determine how each product category is really performing. Two alternatives I prefer to Google Analytics Premium (once you get to that size): Webtrekk, a small but competent German company whose product costs around 1/10th as much per year, has a fraction of the bugs, and does reliable unsampled daily dumps (moving to hourly, I believe), although the UI is a little less intuitive; and a self-hosted Piwik instance, so you don't need to worry about data exports. The truth is modern relational databases are incredibly powerful and will easily scale even with information like impressions in onsite search. There are multi-TB instances of Postgres out there. I really suggest installing either in parallel to GA or on their own when you set up tracking. I do agree with you that anybody involved in any kind of job that includes "analytics" in the title, or indeed most people in management, should take an intro stats course. I particularly like Introduction to Statistical Learning because of its brevity, relatively high abstraction level, and lack of maths.

fndrplayer13 12y ago

I agree with this in principle, but honestly the sampled numbers can be really misleading. It all depends on the magnitude of your visits. If you ask for a year's worth of data for a company with 10 million visits for the year, you're going to get into trouble. This makes creating tools and automated workflows that use the GA API pretty hard. Sampling itself makes a ton of sense though, and its why Google can offer this service at the level of quality they do.

dredmorbius 12y ago

Generally yes, sampling works well and is robust even for surprisingly small samples. For stratified samples -- particularly if you want to stratify traffic by income potential, sampling over all visits may mask data of interest. As for your example, a 10k and 100k sample will provide a 3.16x difference in the standard deviation (that is: accuracy of estimates of central tendency). That is, 10x more datapoints result in only 3x greater accuracy.

crdb 12y ago

Yes and no. Exporting visit trends over 5 landing pages and a month? Sure. Exporting page views for 100,000 products each of whom got 5-100 views? Then that 5% sample is going to exclude most products. The latter approach is however necessary if you're trying to determine how each product category is really performing. Two alternatives I prefer to Google Analytics Premium (once you get to that size): Webtrekk, a small but competent German company whose product costs around 1/10th as much per year, has a fraction of the bugs, and does reliable unsampled daily dumps (moving to hourly, I believe), although the UI is a little less intuitive; and a self-hosted Piwik instance, so you don't need to worry about data exports. The truth is modern relational databases are incredibly powerful and will easily scale even with information like impressions in onsite search. There are multi-TB instances of Postgres out there. I really suggest installing either in parallel to GA or on their own when you set up tracking. I do agree with you that anybody involved in any kind of job that includes "analytics" in the title, or indeed most people in management, should take an intro stats course. I particularly like Introduction to Statistical Learning because of its brevity, relatively high abstraction level, and lack of maths.