4 ms·

Try analyzing a TB of logs per day when all you really want is aggregated statistics. Or really any non-trivial amount of data, even 1GB is a problem. If you w

by aeyes 6y ago

Try analyzing a TB of logs per day when all you really want is aggregated statistics. Or really any non-trivial amount of data, even 1GB is a problem. If you want to know click paths you'll need additional data in your logs. If you want to know how much time the average user spent reading an article on your blog, you are probably out of luck using logs.

giantrobot 6y ago
Most websites aren't Google or Facebook so they're not going to generate terabytes of server logs. A million DAU will generate at most gigabytes of log activity in a day. It's pretty easy to tune logs to be more compact and they're highly compressible. Most sites don't have anywhere close to a million DAU. If you can't handle processing a gigabyte of data it seems like there's fundamental problems with your development team. I don't know how to help you if you can't look at log time stamps and figure out click paths for users. It was a solved problem twenty years ago. It really doesn't matter how long it took someone to read and article on a blog. They either saw your ads or clicked affiliate links or they didn't. It doesn't really matter if they finished an article or bounced right off the page. It doesn't matter if they took ten minutes or twenty minutes to read an article. You don't know if their kid interrupted them half way through or if they're just a slow reader. That kind of shit is just meaningless made up metrics. It's the Gish gallop of web advertising. Throw a wall of bullshit metrics at site owners or advertisers to justify them paying for the privilege of listening to that bullshit.
chrismorgan 6y ago
> Try analyzing a TB of logs per day when all you really want is aggregated statistics. You’re making a completely unfair comparison here. Client-side analytics is performing aggregation as it goes; server-side analytics can do just the same, and serious packages in that space do do that. As an example that readily springs to mind, you can feed server logs into Matomo and use it just as effectively as when you feed client logs into it via matomo.js. On click paths, they’re largely just an artefact of aggregation, so long as you can track individual users (which you admittedly won’t get out of the box in server logs, and perhaps that’s what you were referring to, but you can definitely make it happen). Multiple tabs thwarts doing the “navigated from page A to page B” form of path tracking correctly purely server side, but that’s not a realistic form anyway and isn’t what you’re likely to use; rather you use “page A was loaded, then page B was loaded” and just guess paths to be adjacent loads, which is perfectly compatible with server side analysis. How much time, yeah, that’s one that you can’t do any sort of good judgement of server-side. Not that client-side tracking is particularly excellent at it. All up though I suspect that so long as your bounce rate isn’t too high you’ll get decent enough figures from reckoning the time until the next page load and eliminating statistical outliers. If bounces are too high the figures could be misleading because bounces are likely to behave differently. But people often put far too much effort into trying to track everything, where it’s commonly quite sufficient to take a smaller sample and extrapolate to the whole.

by aeyes 6y ago

giantrobot 6y ago

Most websites aren't Google or Facebook so they're not going to generate terabytes of server logs. A million DAU will generate at most gigabytes of log activity in a day. It's pretty easy to tune logs to be more compact and they're highly compressible. Most sites don't have anywhere close to a million DAU. If you can't handle processing a gigabyte of data it seems like there's fundamental problems with your development team. I don't know how to help you if you can't look at log time stamps and figure out click paths for users. It was a solved problem twenty years ago. It really doesn't matter how long it took someone to read and article on a blog. They either saw your ads or clicked affiliate links or they didn't. It doesn't really matter if they finished an article or bounced right off the page. It doesn't matter if they took ten minutes or twenty minutes to read an article. You don't know if their kid interrupted them half way through or if they're just a slow reader. That kind of shit is just meaningless made up metrics. It's the Gish gallop of web advertising. Throw a wall of bullshit metrics at site owners or advertisers to justify them paying for the privilege of listening to that bullshit.

chrismorgan 6y ago

> Try analyzing a TB of logs per day when all you really want is aggregated statistics. You’re making a completely unfair comparison here. Client-side analytics is performing aggregation as it goes; server-side analytics can do just the same, and serious packages in that space do do that. As an example that readily springs to mind, you can feed server logs into Matomo and use it just as effectively as when you feed client logs into it via matomo.js. On click paths, they’re largely just an artefact of aggregation, so long as you can track individual users (which you admittedly won’t get out of the box in server logs, and perhaps that’s what you were referring to, but you can definitely make it happen). Multiple tabs thwarts doing the “navigated from page A to page B” form of path tracking correctly purely server side, but that’s not a realistic form anyway and isn’t what you’re likely to use; rather you use “page A was loaded, then page B was loaded” and just guess paths to be adjacent loads, which is perfectly compatible with server side analysis. How much time, yeah, that’s one that you can’t do any sort of good judgement of server-side. Not that client-side tracking is particularly excellent at it. All up though I suspect that so long as your bounce rate isn’t too high you’ll get decent enough figures from reckoning the time until the next page load and eliminating statistical outliers. If bounces are too high the figures could be misleading because bounces are likely to behave differently. But people often put far too much effort into trying to track everything, where it’s commonly quite sufficient to take a smaller sample and extrapolate to the whole.