4 ms·
So, what happened to collecting traffic information yourself? You know, from Apache logs and similar tools. Yes, I know google analytics gives a lot of bells
by raverbashing 12y ago
So, what happened to collecting traffic information yourself?
You know, from Apache logs and similar tools.
Yes, I know google analytics gives a lot of bells and whistles and tools and whatnot.
But it's still YOUR website, YOUR data, YOUR traffic. YOU can get (most of) the info that GA is sampling down.
- jszymborski 12y agoPiwik _gives_ you all those bells and whistles, and is 100% open-source and self-hosted. http://piwik.org/ http://piwik.org/
- taf2 12y agoNo doubt Piwik is great but again it's not as easy to setup or maintain as google analytics... GA is easy and free but there's costs in that you might get sampled data if you're doing high volume of traffic... okay but there is also cost in self hosting when you're doing high volume of traffic... no free lunch applies..
- fiatjaf 12y agoI just wrote in another comment[1] about Microanalytics[2]. It is an easy Javascript option that gives you a lot of cool stuff. If you're not a powerful corporation it is definitely worth a try. [1]: https://news.ycombinator.com/item?id=9090721 https://news.ycombinator.com/item?id=9090721 [2]: https://github.com/fiatjaf/microanalytics https://github.com/fiatjaf/microanalytics
- taf2 12y agoCollecting data from YOUR website - costs time/money. Google Analytics is FREE and low cost time/money to install. It's true the data is there and we can get to it... but this takes some foresight that having installed analytics does not require. e.g. a few months after the original apache logs have rotated - you realize it would be nice to know how many mobile safari users are coming to your site vs chrome android users... because you're trying to determine the impact of releasing that next feature that requires webrtc. You can add the analysis to future traffic and give it a few extra weeks but wouldn't it be nice to know right now? GA is nice because it lets you avoid the investment in time/money and you can still go back and look at historical data points you might not have imagined being useful before... I think in this way sampling is really not a big deal... if it is you're right - absolute right. collect the data points that require this level of accuracy yourself... but I don't think if you need that level of accuracy early on you're focusing on the right things...
- gaius 12y agoCollecting data from YOUR website - costs time/money. Google Analytics is FREE and low cost time/money to install It would appear to be just free enough to get you hooked, but to make it accurately give useful results, then it's very, very expensive, is the thrust of the article.
- jlarocco 12y agoNot sure I agree with that. There are, or used to be, plenty of tools for parsing the web logs, crunching the numbers, and spitting out pretty graphs. Again, maybe not as pretty as GA, and not as many bells and whistles, but good enough.
- taf2 12y agobut that can't be as easy as copy pasting a snippet of javascript code into your website?
- raverbashing 12y agoOh wow, yeah, reading files in /var/log/httpd/ is soooo hard. I agree that not all people have the capability to work with this (read, front-end developers, Wordpress customizers, etc) still, I mean, even CPanel lets you download it from past months.
- richardbrevig 12y agocPanel has multiple log analyzers built in. So the user doesn't have to go through that work. If your site is on a shared host and it uses cPanel, just log into cPanel and you'll find them.
- jlarocco 12y agoActually, copy/pasting javascript would be MORE work. The hosting company I use, all I have to do is go to "http://mydomain.com/stats" http://mydomain.com/stats" and enter in the username and password I setup. No javascript changes necessary, because everything is in apache or nginx access logs, which have been around for just about ever. Kinda sad/funny that younger web devs don't know this stuff.
- lmkg 12y agoThree big reasons. 1. If you use a CDN or other sort of caching service, a significant fraction of traffic never hits your server, and your logs are incomplete. 2. Logfile parsing does not measure on-page events that do not involve a server call (such as certain UI interactions, and clicking exit links out to other websites). 3. Parsing logfiles gives crap data. There's a ton of filtering and heuristics that are built into modern web analytics tools. It's a back-breaking task to re-implement them, and your data is literally less-than-useless until you're at leas part-way through the list (as in, provides negative value because it drives you to incorrect conclusions). For example: Web crawlers. Google, Bing, Ask, Baidu, Yandex, and a dozen smaller or region-specific search engines all generating automated traffic to your site. Some of their IP's are public, but Google intentionally runs anonymous web crawlers to make sure people aren't sending Google different pages than what they send humans. And those are just the legimate robots. By some estimates, as much as 50% of modern web traffic is robots. The majority of these are spam bots of one form or another. You know, those one that troll forums with ad links or malware? Those guys aren't negligible. Another example: Associating multiple visit sessions from the same person. This is a fuzzy statistical decision based on cookies, IP address, user agent string, device type, and other data. It's complicated, with a lot of edge cases. Modern tools aren't 100% accurate, but they've already seen and accounted for a decade of exceptions. Another example: Cache-busting. Networks and browsers will cache server calls (including beacon-based tracking), no matter how much you explicitly tell them not to. Unless you specifically write scripting to add random tokens to each of your server calls, you will actually be working with data that has been sampled to an unknown extent. And unlike GA, the sampling is page-based rather than visitor-based which means that visit sessions will have holes in them. In my work as a web analyst, I worked with one company was moving off their own home-rolled system based on Apache log files. They were even putting information from this into their quarterly earnings report. Once we set them up with a "real" system, we found that their homebrew system had over-stated traffic by a factor of ten. Some combination of bot traffic, and users on consumer broadband having variable IP addresses, and cross-domain cookie tracking meant that the web aspect of their business had only 10% of the reach they previously thought. THAT SAID... rolling your own home-brew system is sometimes the correct answer. I'm working with two such clients right now, and they are making the decision that is right for them. It's not the right decision for the majority of companies--it's a large engineering effort, and it needs to have a significant expect payout.